Retrieval Augmented Diffusion Model for Structure-informed Antibody Design and Optimization
Authors: Zichen Wang
Yaokun Ji
Jianing Tian
Shuangjia Zheng
Venue: ICLR 2025 Poster (2025)
Keywords: Generative model
Retrieval augmented geneartion
Protein design
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We designed a retrieval-augmented conditional diffusion model for antibody design and optimization

Abstract: Antibodies are essential proteins responsible for immune responses in organisms, capable of specifically recognizing antigen molecules of pathogens. Recent advances in generative models have significantly enhanced rational antibody design. However, existing methods mainly create antibodies from scratch without template constraints, leading to model optimization challenges and unnatural sequences. To address these issues, we propose a retrieval-augmented diffusion framework, termed RADAb, for efficient antibody design. Our method leverages a set of structural homologous motifs that align with query structural constraints to guide the generative model in inversely optimizing antibodies according to desired design criteria. Specifically, we introduce a structure-informed retrieval mechanism that integrates these exemplar motifs with the input backbone through a novel dual-branch denoising module, utilizing both structural and evolutionary information. Additionally, we develop a conditional diffusion model that iteratively refines the optimization process by incorporating both global context and local evolutionary conditions. Our approach is agnostic to the choice of generative models. Empirical experiments demonstrate that our method achieves state-of-the-art performance in multiple antibody inverse folding and optimization tasks, offering a new perspective on biomolecular generative models.

Published as a conference paper at ICLR 2025
RETRIEVAL AUGMENTED DIFFUSION MODEL FOR STRUCTURE-INFORMED ANTIBODY DESIGN AND OP- TIMIZATION
Zichen Wang Yaokun Ji
Jianing Tian
Shuangjia Zheng-!
Global Institute of Future technology, Shanghai Jiao Tong University;
2 School of Software & Microelectronics, Peking University
cichen.w8s jtu eduu cn
shuangjia z edu
ABSTRACT
Antibodies are essential proteins responsible for immune responses in organisms, capable of specifically recognizing antigen molecules of pathogens. Recent ad- vances in generative models have significantly enhanced rational antibody de- sign. However, existing methods mainly create antibodies from scratch without template constraints, leading to model optimization challenges and unnatural se- quences. To address these issues, we propose a framework, termed RADAb, for efficient antibody design. Our method leverages a set of structural homologous motifs that align with query structural constraints guide the generative model in inversely optimizing antibodies according to desired design criteria. Specifically, we introduce a structure-informer retrieval mech- anism that integrates these exemplar motifs with the input backbone through a novel dual-branch denoising module, utilizing both structural and evolutionary in- formation. Additionally, we develop conditional diffusion model that iteratively refines the optimization process by incorporating both global context and local evolutionary conditions. Our approach is agnostic to the choice of generative models. Empirical experiments demonstrate that our method achieves state-of- the-art performance in multiple antibody inverse folding and optimization tasks, offering a new perspective on biomolecular generative models.
diffusion
INTRODUCTION
Antibodies, essential Y-shaped proteins in the immune system, are pivotal for recognizing and neu- tralizing specific pathogens known as antigens. This specificity primarily arises from the Comple- mentarity Determining Regions (CDRs), which are crucial for binding affinity to antigens (Jones) et al., 1986 Ewert et al. 2004 Xu & Davis 2000: Akbar et al., 2021). The design of effective CDRs is therefore central to developing potent therapeutic antibodies, dominant class of protein therapeutics. However, the development of these antibodies typically relies labor-intensive perimental methods such as animal immunization or screening extensive antibody libraries, often failing to produce antibodies that target therapeutically relevant epitopes effectively. Thus, the abil- ity to generate new antibodies with pre-defined biochemical properties in silico carries the promise of speeding up the drug design process.
Computational efforts in antibody design have traditionally involved grafting residues onto exist- ing structures (Sormanni al.] 2015), sampling alternative native CDR loops enhance affini- ties(Aguilar Rangel et al. (2022). and using tools like Rosetta for sequence design improvements in interacting regions (AdolI-Brylogl et al. 2018). Many recent studies have focused on applying
deep generative models to design antibodies (Luo et al. 2022 Martinkus et al.) 2024 Zhu et al., 2024). They take advantage of geometric learning and generative models to capture the higher-
"Equal Contribution Corresponding author

---

Published as a conference paper at ICLR 2025
order interactions among residues directly from the data. These innovations provide more efficient methods to search sequence and structure spaces.
Albeit powerful, current generative models struggle to design antibodies that adhere to structural constraints and exhibit desired biological properties. This challenge primarily arises from a lack of diversity in the available training data. Predominantly, research efforts have relied on the SAbDab database (Dunbar et al. 2014). which comprises fewer than ten thousand antigen-antibodd complex structures. The limited scope of this dataset restricts the models' ability capture comprehensive high-order interaction information between antigen-antibody residues, thereby increasing the risk of overfitting, Moreover, most existing methodologies attempt to design antibody sequences de novo, without the benefit of template-base guidance. This approach inherently demands a greater volume of data and extensive training or fine-tuning on specific datasets to achieve efficacy in practical applications.
In this work, we draw inspiration from template-based and fragment-based antibody design to develop a model that fully utilizes protein structural database, effective motif re- trieval, and semi-parametrie generative neu- ral networks. Our goals are to: (a) leverage template-aware local and global protein geo- metric information to enhance model genera- tive capability, (b) integrate motif evolutionary signals to prevent overfitting, and (c) necessi- tate minimal training or fine-tuning for effective use in real-world applications.
Template based Ab Design
Cancaree
Sequente
OCHE
Grafting
Retrieval
Backkone Algerent IMBU Tos
Ment Struturdd
Generator
Einlatioary Information
RAG based Ab Design
Corstramt
Diffusion
Figure Illustration of the ettrivallaugmment framework.
To this end. we introduce the Retrieval- Augmented Diffusion Antibody design model (RADAb), novel semi-parametric antibody design framework. To fully exploit the protein structure space, we first compiled database of CDR-like fragments from the non-redundant Protein Data Bank (PDB) (Berman et al. (0000) These CDR-lik fragments are linear functional motifs structurally compatible with antibody CDR loop, found in any protein geometry within the PDB. For a given antibody to be improved, we perform a structural retrieval to obtain motifs with structures similar to the desired CDR framework. As protein quences capable of folding into similar structures often share homology and consensus, we hypoth- esize that these retrieved motifs, enriched with evolutionary information, can enhance the model's generalization.
Unlike traditional rational design methods that optimize by grafting single CDR fragment, pro- pose to use a set of structural homologous CDR-like motifs together with the desired backbone for iterative sequence optimization (Figure [.. Our major contributions follow: (1) We propose a first- of-its-kind retrieval-augmenttd generative framework for rational antibody design. It uses a set of functional CDR-like fragments that satisfy the desired backbone structures and properties to guide generation toward satisfying all the required properties. (2) A novel retrieval mechanism is intro- duced for integrating these exemplar motifs with the input backbone through a novel dual-branch denoising module, utilizing both structural and evolutionary information. Additionally, we present coupled conditional diffusion module that iteratively refines the evolution process by incorporating global and local conditions. This allows the model to incorporate more functional information than traditional antibody inverse folding models. (3) Empirical experiments demonstrate that our method improves the tfate-ofthhe-a art methods in multiple antibody inverse folding tasks, e.g., 8.08% AAR gains in long CDRH3 inverse folding task and an average of cal/mol absolute AAG improvements in functionality optimization task, offering fresh perspective on biomolecular generative models.
2 RELATED WORK
Antibody Design Computational antibody design primarily follows two paths: conventional energy function optimization methods and machine learning approaches. Early antibody design methods were often limited to sequence similarity and energy function optimization (Lapidoth et al. 2015) AAdolf- Bryfogle et al., (2018). Recent success of machine learning approaches mainly falls into two directions: antibody sequence design and antigen-specific antibody sequence-structuure co-design.
2

---

Published as a conference paper at ICLR 2025
The methods used for antibody sequence design mainly include language- based models (Ruffolo
iet al. 202I Olsen et al. 2002 Wang et al., (023a) and inverse folding models (Dreyer et al. 2023]
Hoie et al.; 2024). The other line focuses on antibody requence-structur co-design mainly taking
antibody-antigen complex as a graph, then using graph networks to extract features and predict
the coordinates and residue type of antibody CDR (Jin et al.; 2021: Kong et al. 2022 2023 Lin
et al., 2024; Luo et al., 20222] Zhu et al.) 2024, Martinkus et al. 2024). While these works are undoubtedly powerful, they often generate antibodies from scratch without incorporating explicit structure constraints, which can introduce challenges in designing functional antibodies (Zhou et al., 2024). Instead, our method leverages the power of templates from a structure-informee perspective.
Diffusion generative models Diffusion models (Sohl-Dickstein et al. 2015; Song et al., 2020 Ho et al., 2020) are class of generative models that have achieved impressive progress on a lot of generation tasks. Denoising diffusion probabilistic models (DDPMs) are a branch of diffusion mod- els, which contain two Markov processes. The forward process perturbs the data into pure noise, and then learns to generate data by reversing the forward Markov process. Because of the diffu- sion model's flexibility and controllability, numerous works are focusing on employing retrieval- augmented methods complement the diffusion framework for text-to-image generation (Sheynin let al. 2023). image generation (Blattmann et al.) 2022). human motion generation (Zhang et al. 2023a and small molecule generation (Huang et al. 2024
Retrieval augmented generative models Retrieval augmented generation technique was first pro- posed in the field of natural language processing to enhance the language models by introducing an additional database (Lewis et al. [020 Guu et al., 2020). prompting the language models to gen- erate more realistic and diverse results. Subsequently, retrieval augmented generation (RAG) has conducted diverse explorations in large fields, including natural language processing (Zhang et al.)
2023b Gao et al.
2023
Xu et al., 2024
Yoran al., 2024 Caffagni et al. (024) and computer
vision Long et al.
2022:
Blattmann et al. 2022 Hu et al. [2023] Rao et al. (2023).
Recently, several studies have been proposed using retrieval techniques to enhance molecular gen- eration. RetMol (Wang et al.] (023b) generates new molecules based on existing small molecules by retrieving a set of exemplar moleculars. IRDiff (Huang et al., (024) enhances protein-spes specific molecular generation by using protein pockets to retrieve moleculars which interct with the pocket. Although there has been some retrieval-based work in the field of protein design and discovery (Zhou & Grigoryan 2015 Aguilar Rangel et al. 2022), to the best of our knowledge, this is the ofst-offits-kinn kind retrieval- based generative framework for antibody design.
PRELIMINARIES AND NOTATIONS
3.1 NOTATIONS
Antibody consists of two heavy chains and two light chains. Each chain's 's tip has a complementary site that specifically binds unique epitope on the antigen. This site includes six complementary- determining regions (CDRs): CDR-HI, CDR-H2, and CDR-H3 on the heavy chain, and CDR-L1, CDR-L2, and CDR-L3 on the light chain (Presta 1992 Al-Lazikani et al. 1997).
Our work represents each single protein residue in terms of the residue type 8; e MACDEGGHIKLMNPQORSTT
the coordinate 2, R33 and the orientation O e SO (3), where i = 1, ..., N and N is the number of residues in the complex. Concretely, assuming the CDR sequence to be generated includes m amino acids and starts from position a, can be denoted as R {s, lj E 11 I...... + m.}. Let M be the length of the antíbody, the antibody frame- work is defined as C.. = [(.. K3, 0,) 1EE {l, M3 Va + I, ..., a + m).j E {l,..., M).. The antibody framework sequence is defined as S.b = {s:
{1, ...., M) Ma 1, 1..., + m. The antigen is defined as Cag = {(s O.) E (M 1,...... 1)). The retrieved Cik-like fragments defined A- (A, E (1. k)}. The goal of framework is to extract the antibody CDR structure from the antibody framework complex C.b. then input it into the retrieval module to retrieve A, and ultimately predict the distribution of R through Cag. Cab and A.
(1.
3

---

Published as a conference paper at ICLR 2025
Filter
Top Augmented Sequencas
Sequence
HARDIO
Context Encoder
0
Tops Rendwoed Engrress
PDB data
..GDPYEG..
Structure informed Natwork
CoR-facusing Asial Attention
Augmentee space
ESMP
Frokutionary
ple Ri C C
Inverse Folding
Recommended
H1
HB
space
ABKSOFIIING...
Relax
Fold
Fitness landscape space
R²
R'
72
Cutput
Antibody
The overall architecture of the proposed RADAb framework. (A) Structural retrieval process, the CDR backbone is imput into MASTER and the output is a set of ranked CDR-like fragments. (B) Diffusion process and denoising network which takes antibody-antigen context and retrieved evolutionary information as conditions. The structure fixed during diffusion process. (C) Our method restricts the antibody small region through fixed structural constraints and retrieval- augmented constraints (functional constraints) to achieve higher fitness.
Figure 2:
3.2 DIFFUSION MODEL FOR ANTIBODY DESIGN
Due to difftusion models' excellent performance and controllability, there are now many diffusion- based works that have achieved notable results (Luo al.) 2022 Filleaa-Moreile
al., 2023] Kulyté et al.) 2024). To be specific, they are denoising probabilistic diffusion models that transform the amino acid type s, the backbone Co: atom coordinates and the amino acid orientation O during the diffusion process. We focus on sequence, of which the forward process perturbs the data in the
following ways (Hoogeboom et al. 2021):
q ( 1s Multinomial (1- onchot (s)- + gr.
(1)
20
where 8e is the noise schedule for the diffusion process, approaches T, 8 will approach 1, and the probability distribution will become closer to pure noise. corresponds to 20-dimensional all-one vector.
To reverse the aforementioned forward process and denoise to generate CDR sequence, predictions need be made by neural network FOli), which takes the antibody-antigen context as condition:
(s RCCCus, Coy) = Multinomial (F (R', Cub. Cog) [i])
(2)
As an example, this work uses Diffab (Luo al.) 2022) as the backbone for the generative model to conduct retrieval augmented generation. Note that the proposed retrieval system generative model
agnostic, and the developed modules can be integrated with any diffusion generative model.
4 METHODS
We propose RADAb (as demonstrated in Figure 2) novel structure-informed eetrieval-augmente diffusion framework for antibody sequence design and optimization. The model uses structural retrieval algorithm to search for antibody homologous structures and take their sequences as condi- tional inputs for the diffusion model to provide homologous patterns and evolutionary information.
4.1 STRUCTURAL RETRIEVAL OF CDR FRAGMENTS
The structure of protein is determined by its sequence, and protein sequences that can fold into similar structures exhibit similar properties. These structurally similar protein sequences contain rich evolutionary information. Based on this, perform retrieval in the PDB database using CDR structures, aiming to obtain fragments that are similar to the real CDR and have homologous se- quences, with the expectation that they possess similar functions.
4

---

return A. Published as a conference paper at ICLR 2025
To balance the quality of results and the retrieval speed, use MASTER (Zhou & Grigoryan 2013) for the search. MASTER uses the similarity measure. It queries structural fragments composed of one more non-contiguous seg- ments and can find all matching fragments from the database within given RMSD threshold. This allows for fast and accurate searches in the PDB database for protein motifs. Note that MASTER can utilize only the backbone information without any leakage of sequence data during the search process. The retrieval procedure is described in Algorithm [ and detailed Appendix A.3
deviation (RMSD) of backbone atoms as
For the retrieved results, we use the RMSD with the real backbone structure as a score to rank them and filter out the input CDR fragment. For ease of use, we further constructed a CDR-like fragments database (detailed in Appendix A.4). Additionally, to enable the model to learn richer evolutionary information, we filter out identical CDR-like sequences during the training phase. However, to improve the quality of the model's generation, we do not perform similar filters during generation.
Algorithm Structural Retrieval Algorithm Overview
1: Input: Coordinates set X' {2x k {l.....m) m]]
2: Input: Structure database with P structures
in lie
3: Initialize CDR-like fragments set: A /
4: Initialize structure residues set: C + 0
5: Initialize threshold maxA(), maxB(), maxC(
6: for pni== to P do
7: C + all residues in TÚ
8: for each residue j in C do
9:
r + R.MDD(X. (XX.))
10:
if r maxA(XX) then
eliminate j from the list C continue
12:
13:
end
14:
if (r maxB(x)) OR (cRMSD(X)> > maxC(())) then
15:
continue
16:
end if
17:
A -
18:
insert match A into A.
19:
end for
end for
21:
4.2 MODEL. ARCHITECTURES
The model takes the antigen-antibody complex's structure and sequence context, along with the sequences of the CDR-like fragments, as conditional inputs to iteratively denoise. The first branch of the model learns the global context information of the complex, while another branch takes the local homologous information of CDR-like fragments input, aiming learn the functional similarity and evolutionary information of residues with similar structure. The two branches are combined to generate the antibody CDR sequence jointly.
4.2. GLOBAL GEOMETRY CONTEXT INFORMATION BRANCH
Context encoder protein formed by the connection of multiple residues. The features of single residue mainly include the residue type, backbone atom coordinates, and backbone dihedral angles. The features each pair of residues mainly include the types of both residues, sequential relative position, spatial distance, and pairwise backbone dihedrals. These features are concatenated and then input into two separate MLPs. The output is denoted as 2; and Yij-
Evolutionary encoder Recent advances have shown the structure structure-informel protein language model (PLM) is an excellent tool for creating protein sequence embeddings and providing evolutionary information (Zheng et al. 2023 Shanker et al. 2024). Thus, take ESM2 (Lin et al. 2023) as

---

Published as a conference paper at ICLR 2025
antibody sequence encoder, aiming capture the evolutionary relations of antibody residues. The state antibody sequence with CDR at timestep is fed into and output is defined as e'.
Structure-informedd network The above encoding is used conditional input to the Structure- informed network. They, along with the CDR sequence and structural state at the current time step, will be input into stack of Invariant Point Attention (IPA) (Jumper et al. 2021) layers, and jointly transform into a hidden representation h4- Subsequently, the hidden representation h2 is transformed by an MLP obtain the probability representation Tglohal of the amino acid type at each CDR site. This probability representation is then input to the local CDR-focused branch.
4.2.2 LOCAL CDR-FOCUSED INFORMATION BRANCH
Pust-processing of CDR-like fragments We first remove the CDR portions from the antibody se- quences to obtain the antibody framework sequences. Then, we fill these fragments' short sequences into the antibody framework sequences, thereby constructing CDR-like sequence matrix E.
CDR-focused Axial Attention The local CDR-like branch is constituted of a stack of axial atten- tion layers, referred to as CDR-focused Axial Attention. Given that the CDR-like fragments exhibit structures similar to actual CDRs, we employed tied row attention mechanism used in MS ATrans- former (Rao et al. 2021) to leverage these retrieval results. In the standard axial attention (Ho et al.) and column considered independently. However, in MSA (Multi- ple Sequence Alignment), each sequence exhibits relatively similar structural features. Our matrix format is well-suited for adopting a tied row attention mechanism fully utilize the structural simi- larity. When calculating the attention scores for each row, this mechanism simultaneously considers the scores other rows. This approach not only leverages the structural similarity but also reduces memory usage.
2019) mechanism, each
The input CDR-focused Axial Attention is a pseudo-MSA matrix P in equation The first row of this matrix is initially filled with the antigen-antiboddy framework sequence, with the CDR region populated by the noisy sequence R (sampled by Tgloba) at the current time step From the second row to the k-th row (where k is chosen to be 16, meaning the top 15 retrieved CDR-like sequences are used as conditional input), the rows are filled with the CDR-like sequence matrix E. The constructed matrix P is then input to CDR-focused Axial Attention to create the homologous embedding and calculate the probability representation Mocal (equation 4
P concat ((Sob U R)), E)
Flocal [ = Geot (P.j.t) t) for all E col,
Theaal [. Grisdoow (P,,,,) t) for all row
The row self-attention is computed to capture the internal relationships within the antibody-antigen sequences, while the column self-attention is computed to capture the relationships between the CDR residues and the CDR-like residues.
Skip connection for information fusion Although the probability distribution of the CDR region created by the antigen-antibody context features has already been fed into the network, to prevent the loss of
context information during forward propagation, the embedding Plocal and Fglobal are added by skip connection module (He et al. 2016). then execute softmax to obtain the final probability distribution.
4.3 MODEL TRAINING AND INFERENCE
The overall training objective The training objective to minimize the probability distributions predicted by the network under conditions each time step and the true posterior distribution at the same time step. Therefore, we choose KL divergence between the two distributions at each residue in the CDR region as the training loss function,
1 Σ DDDL. DKL. (a (55-1 $5, 11 (5/-1 1 R². Cab, Cag, A))
Type = ER
(5)
The training objective of the whole diffusion process is:
L Fe Unifomm (1...T) 1 1ype

---

Published as a conference paper at ICLR 2025
Conditional reverse diffusion process We employ DDPM generate sequences. The model starts from time step T, initializing each site of the antibody CDR region as a uniform distribution. Then, through the frozen ESM encoder E(-), learned global context network F(OGj] and the local CDR- focused network GOLi), they predict the noise distribution at each time step jointly and denoise step-by-step:
e' = E(aab UR)
P (d) RR', Cab, Cag; A.) Multinomial [F (R', Car, e.)
(8)
+G (F' (R', Cob. Cab.Cagi AA)
During sampling, we remove the CDR region sequences from the antibody structures and fill them with noisy sequence sampled from the uniform distribution. The retrieval process uses the structure of the CDR region as input and outputs a set of CDR-like fragments. Subsequently, this set of CDR- diffusion model, serving as condition along with
like fragments is fed into the retrieval-augment framework context guide the model in step-by-step denoising and generating the CDR sequences.
the
5 EXPERIMENTS
To evaluate the performance of our model's generation, we utilize two tasks: antibody CDR se- quence inverse folding (Section 5. and antibody optimization based on sequence design (Section 5.2). to compare with the baselines. Additionally, we conducted ablation experiments and further analysis to demonstrate the effectiveness of the augmenedl- method (Section (.3)
The dataset for training the model is obtained from the SAbDab and our established CDR-like fragments dataset. Following the previous work (Luo et al. 2022), we first eliminated structures with a resolution lower than 4A and removed antibodies that target non-protein antigens. Chothia (Chothia & Lesk 1987) in ANARCI (Dunbar& & Deane, 2016) is used for renumbering antibody residues. We clustered the SADab datasets based on 50% sequence similarity in the CDR-H3 region, and chose 50 PDB files comprising 63 antibody-antige complex structures as the test set. To ensure distinct training and test sets, removed structures from the training set that were part of the same clusters as those in the test set.
5.1 ANTIBODY CDR SEQUENCE INVERSE FOLDING
Baselines For traditional methods, we simulated a method of grafting using CDR-like data in the process of rational antibody design. Specifically, we directly graft the retrieved top-1 CDR-like fragment onto the antibody framework, termed Grafting. For deep learning methods, we selected a series of state-of-the-art protein inverse folding models for comparison with our work, including ProteinMPPNN (Dauparas et al. 2022). a model that utilizes message passing neural network to design sequences with fixed protein backbone; ESM-IF (Hsu et al., 2022), protein inverse folding model that trained on millions of predicted structures; Diffab-fix (Luo et al. 2022). which can fix the backbone structure and iteratively generate candidate sequences from pure noise in sequence space using diffusion; AbMPNN (Dreyer et al. 2023), model fine-tuned ProteinMPNN on antibody sequence and structure. Because it is not open-sourced, we evaluate on its own test set. For more baseline details, please refer to Appendix AA.5]
Metries To evaluate the accuracy and rationality of the sequences generated by the model, we se- lected the following three popular evaluation metrics: (1) Amino Acid Recovery (AAR,%): AAR refers to the ratio of positions where the designed sequence and the true CDR sequence have the same amino acid; (2) Self-consistency RMSD (scRMSD, Á ): To calculate scR MSD, we refold the antibody sequences generated by the model using ABodyBuilder (Abanades et al.. 2023). Then, we align the refolded antibody framework with the original antibody and compute the RMSD of the C. atoms in the CDR region. (3) Plausibility: We use -ooudo--o--liclihoooo in an antibody language model, AntiBERTy (Ruffolo et al. 2021 to calculate plausibility of the generated sequence.
Results As shown in Table T RADAb outperforms state-of of-the-art methods in each metric and at each CDR region. In particular, in the highly variable and specific CDR-H3 region (Shirai et al. 1999 Raybould et al.) (201)), our method achieved a great improvement in AAR compared to best- performing methods Diffab-fix and AbMPNN. The evaluation results indicate that the retrieval-
7

---

Published as a conference paper at ICLR 2025
Table I: Results sequence design on SAbDab dataset
CDR.HI
Method
AAR(2) 1
scRMSD + Plausibility 1
AAR(S) 1
scRMSD
plausibility 1 AAR(S)
scRMSD
plausibility 1
Grafting
58.05
0.83
-0.597
31.46
0.79
-0.619
19.63
3.20
-0.591
PrecciaMPNN
$8.58
0.64
-0.603
53.18
0.61
-0.568
41.77
2.27
-0.605
ESM-IF1
$3.80
0.66
-0.610
46,06
0.63
-0.589
29.82
2,59
-0.607
Diffab-fix
74.93
0.66
-0.512
65,41
0.59
-0.532
49.17
2.24
-0.541
4
72.83
1.09
-0.664
65.33
0,93
-0.677
52.99
2.80
-0.675
RADAb
76.57
57.02
CDR-L.I
CDR.L2
CDR.L3
Method
AAR(%) 1
seRMSD
Plausibility t AAR(S)
scRMSD
plausibility t AAR(S))
scRMSD
plausibility t
Grafting
68.53
-0.506
43.19
0.52
-0.573
43.61
1.08
-0.395
PriteiMMNN
45.60
-06122
46.78
47.21
ESM-IFI
40.97
0.61
0,650
43,40
0.43
-0.542
38.93
0.92
-0.569
Diffab-fix
79.78
0.56
0.386
81.19
0.44
-0.390
67.97
0.88
-0.414
A
75.06
0.73
-0.543
71.63
0.56
-0.528
64.51
-0.544
RADAb
83.72
0.54
40.379
84.58
0.44
+0.384
73.11
0.87
-0.384
Table 2: Results of long CDRH3 sequence de-
Table 3: Results of binding energy optimization
sign performance.
based on antibody sequence design.
Method
AAR(%) (%)
scRMSD
plausibility
AAG-
IMP.
Method
AAG4
Grafting
7.79
4.05
-0.785
seq
seq (%)
Grafting
135.17
40.22
32.69
ProteinMPNN
46.63
2.71
-0.820
ESM-IF1
30.01
2,86
-0,845
ProteinMPNN
127.14
24.72
35,51
Diffab-fix
42.26
3.02
-0.740
ESM-IFI
162.09
42.28
33.33
4bMPNN
43.27
4.39
-1.012
Diffab-fix
116.36
14.05
34.52
RADAb
51.35
2.52
-0.747
RADAb
109.16
7.06
37.30
augmented method, by introducing structurally similar homologous sequences, has improved the accuracy, consistency, and rationality the model's generation.
In addition, CDR-H3 exhibits significant variability in length, sequence, and structure. Typically, deep learning models show decreased performance when generating longer CDR-H3 sequences (Loo et al. 2022: Høie al. 2024). Therefore, selected subset of the test set with CDR-H3 lengths longer than 14 to evaluate the generation performance. As shown in Table while the gen- eration performance of all methods declines to some extent, our method demonstrates consistency and significantly outperforms the others, with larger margin of improvement.
5.2 ANTIBODY FUNCTIONALITY OPTIMIZATION
In this section, focus the evolution of antibody sequences and evaluate whether the structure of the evolved sequence has greater functionality compared to the structure of the folded original sequence. To this end, first fold the designed CDR-H3 sequences with framework sequences and the original real antibody sequences into complete protein structures using ABodyBuilder. Then, use FastRelax and InterfaceAnalyzer in PyRosetta (Alford al. (017)) to relax the structure and calculate the binding energy AG of the antibody-antigen antigen complex.
Metries We use various metrics to evaluate the efficacy and functionality of our designed antibodies: (1) AAG: This metric represents the difference in binding energy between the complex with the designed CDR folded into the structure and the original complex binding energy. (2) AAG-seq: This metric measures the difference in binding energy between the complex with the designed CDR sequence folded into the structure and the binding energy of the original antibody sequence folded into the structure. It aims eliminate errors introduced by the folding tool, allowing for direct comparison sequence functionality, (3) IMP-seq: This metric indicates the percentage of designed CDR sequences folded into the structure with lower (better) binding energy than the original antibody sequences folded into the structure.
Results As shown in Table 3 after folding and relaxing, the antibody sequences designed show a significant decrease in binding energy compared to other methods, and 37.3% of them have better binding energy than those folded from the original antibody sequences.
8

---

Published as a conference paper at ICLR 2025
Binding energy(AG)
Sample1
Sample2
Sample2
REF
Sample1
TAG:556.05
-GG:11.771 QGWLSGSFDV
AGWVRGAFD!
VGWLSGSFDV
Figure 3: Left: Distribution of the samples' interface energy. Right: Generated CDR-H3 samples and the original structure of PDB: 7d6i antigen-antibody complex. The gray part represents the antibody framework, the red part represents the CDR, and the blue part represents the antigen (with the darker shade indicating the antigen epitope).
Table 4: Ablation Study. G represents using Ground truth as retrieval results, represents the Retrieval-augment mechanism, and E represents Evolutionary embedding mechanism.
60
RADAb
56
Abmpnn
Diffab
52.97
Ablation
G
AAR(%)
scRMSD
Plausibility
48
48.57
49.49
70.56
2.13
-0,534
44.54
X
V
51.36
2.23
-0,538
X
V
X
52.15
2.39
-0,529
3
Value of
15
X
X
X
49.17
2.24
-0,541
Figure 4: The effect of the value of CDR-like fragments k on model's CDRH3 performance.
X
V
57.02
2.23
-0.530
To further demonstrate the optimization of antibody sequence functionality by RADAb, we select a specific antigen-antibody antibody complex from the test set (A neutralizing MAb targeting the receptor- binding domain of SARS-CoV-2, PDB: 7d6i). We generate sequences for CDR-H3 and calculate the binding energy AG of the folded structures. Among these, 68% of the samples exhibited lower AG compared to the original complex. As shown in Figure 3 we select two samples as examples. Although they do not achieve the highest AAR, they demonstrate better binding affinity compared to the native structure.
5.3 ANALYSIS
Ablation We conducted series of ablation experiments for CDR-H3 following the settings de- scribed in Section to validate the effectiveness and relative contributions of the additional con- ditions and data introduced. The specific objectives are: (1) to verify the effectiveness of the retrieval augment module; (2) to assess the validity of the retrieved data; and (3) to evaluate the effectiveness of the evolutionary embedding.
We demonstrated the retrieval augment module's effectiveness by inputting the CDR sequence's ground truth into this module. We also removed the retrieval augmentation mechanism and the evolutionary embedding mechanism respectively to validate their effectiveness. The experimental results show that both the retrieval augmentation module and the evolutionary em- bedding module individually improve performance, and using them together maximizes the model's performance.
As shown in Table 4
Effect of retrieval dataset To further analyze the benefits brought by the retrieval mechanism and retrieved motifs, we conducted series of comparative experiments on the value & CDR-like fragments selected conditions in the diffusion network, as shown in Figure 4 We found that when the value of & low, brings negative benefits to the model, which may be due to overfittiig. As the value of increases, the model performance also gradually improves. When & equals 15, the model achieves the best performance. But when & exceeds 15, the additional information instead introduces noise the model, leading performance degradation.

---

Published as a conference paper at ICLR 2025
6 CONCLUSION AND FUTURE WORKS
In this work, we propose vetrieval-augmente diffusion generative model RADAb for antibody se- quence design. This model leverages global geometric information and local template information, incorporating these conditions into the diffusion process to enhance antibody sequence design and optimization. Experimental results demonstrate that RADAb achieves state-of-the-an performance across multiple tasks. The main limitation of this work is that has not yet been fully validated in wet lab experiments, which will be one of the major tasks in the future. Since have proposed comparatively general retrieval method and retrieval-augee augment framework, another major future task extend the model to the design of various protein motifs.
REPRODUCIBILITY STATEMENT
The code is avalibale at http::/ggthha.commGGEEEELLLLALDD
ACKNOWLEDOMENTS
This study has been supported by the National Natural Science Foundation China [62041209], Natural Science Foundation of Shanghai [24ZR1440600), the Science and Technology Commission of Shanghai Municipality [24510714300).
REFERENCES
Brennan Abanades. Wing Ki Wong, Fergus Boyles, Guy Georges, Alexander Bujotzek, and Char- lotte M Deane. Immunebuilder: Deep-learning models for predicting the structures of immune proteins. Communications Biology, 6(1 (1:55, 2023.
Jared Adolf-Bryfogle, Oleks Kalyuzhniy, Michael Kubitz, Brian D Weitzner, Xiaozhen Hu, Yumiko Adachi, William R Schief, and Roland L Dunbrack Jr. Rosettaantibolydssign (rabd): A general framework for computational antibody design. PLoS computational biology, 14(4)=e100611) 12, 2018.
Mauricio Aguilar Rangel, Alice Bedwell, Elisa Costanzi, Ross J Taylor, Rosaria Russo, Gonçalo JL Bernardes, Stefano Ricagno, Judith Frydman, Michele Vendruscolo, and Pietro Sormanni. Fragment-basee computational design of antibodies targeting structured epitopes. Science Ad- vances, 8(45):ebb99540, 2022.
Rahmad Akbar, Philippe A Robert, Milena Pavlović, Jeliazko R Jeliazkov, Igor Snapkov, Andrei Slabodkin, Cédric Weber, Lonneke Scheffer, Enkelejda Miho, Ingrid Hobæk Haff, al. A compact vocabulary of paratope-epitope interactions enables predictability of antibody-antigen binding, Cell Reports, 34(11), 2021.
Bissan Al-Lazikani, Arthur M Lesk, and Cyrus Chothia. Standard conformations for the canonical structures of immunoglobulins. Journal of molecular biology, 273(4):927-948 1997.
Rebecca F Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, Matthew J O' Meara, Frank P DiMaio, Hahnbeom Park, Maxim V Shapovalov, P Douglas Renfrew, Vikram K. Mulligan, Kalli Kappel, et al. The rosetta all- all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, 13(6):30313045 2017.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip Bourne. The protein data bank. Nucleic acids research, 28(1): 235-242, 2000.
Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Bjöm Ommer. Retrieval- augmented diffusion models. Advances in Neural Information Processing Systems, 35:15309- 15324, 2022.
Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Comia,, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hierarchical retrieval-augmentte generation for mul- timodal 1lms. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1818-1826, 2024.
10

---

Published as a conference paper at ICLR 2025
Cyrus Chothia and Arthur M Lesk. Canonical structures for the hypervariable regions of im- munoglobulins. Journal of molecular biology, 196(4):901-917, 1987.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpnn. Science, 378(6615):49-555) 2022.
Frédéric A Dreyer, Daniel Cutting, Constantin Schneider, Henry Kenlay, and Charlotte M Deane. In- verse folding for antibody sequence design using deep learning. arXiv preprint 2rX1v:2310.119555: 2023.
James Dunbar and Charlotte M Deane. Anarci: antigen receptor numbering and receptor classifica- tion. Bioinformatics, (2(2):2883000 2016.
James Dunbar, Konrad Krawczyk, Jinwoo Leem, Tenry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M Deane. Sabdab: the structural antibody database. Nucleic acids research, 42(D1):D1140: D1 146, 2014.
Stefan Ewert, Annemarie Honegger, and Andreas Plückthun. Stability improvement of antibodies for extracellular and intracellular applications: Cdr grafting to stable frameworks and structure- based framework engineering. Methods, (442):844110 199, 2004.
Nathan C Frey, Dan Berenberg, Karina Zadorozhny, Joseph Kleinhenz, Julien Lafrance- Lafrance-Vanassee Isidro Hotzel, Yan Wu, Stephen Ra, Richard Bonneau, Kyunghyun Cho, al. Protein discovery with discrete walk-jump sampling. In The Twelfth International Conference on Learning Repre- sentations, 2023.
Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, Davide Boscaini, Michael M Bronstein, and Bruno E Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):184-192, 2020.
Pablo Gainza, Sarah Wehrle, Alexandra Van Hall-Beauvais, Anthony Marchand, Andreas Scheck, Zander Harteveld, Stephen Buckley, Dongchun Ni, Shuguang Tan, Freyr Sverrisson, al. De novo design of protein interactions with learned surface fingerprints. Nature, 617(7959):176- 184, 2023.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. aegmentel generation for large language models: A survey. arXiv preprint arXiv:2312.10990 10997, 2023.
Kelvin Guu, Kenton Lee. Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3929-3938. PMLR, 13-18 Jul 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenbom, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv preprint 191iv: 1800 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 18511, 2020.
Magnus Haraldson Høie, Alissa Hummer, Tobias Н Olsen, Broncio Aguilar-Sanjuan, Morten Nielsen, and Charlotte M Deane. Antifold: Improved antibody structure- design using inverse folding. arXiv preprint arXiv:2405.03377 2024.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.). Advances in Neural Information Pro- cessing Systems, volume 34, pp. 12454-12465. Curran Associates, Inc., 2021.

---

Published as a conference paper at ICLR 2025
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexan- der Rives. Learning inverse folding from millions of predicted structures. International con- ference machine learning, pp. 8946-8970 PMLR, 2022.
Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai- Wei Chang, Yizhou Sun, Cordelia Schmid, visual-language pre-training with multi-source multimodal knowledge memory. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
David A Ross, and Alireza Fathi. Reveal:
2023.
Zhilin Huang. Ling Yang, Xiangxin Zhou, Chujun Qin, Yijie Yu, Xiawu Zheng, Zikun Zhou, Wentao Zhang, Yu Wang, and Wenming Yang. Interaction-based retrieval-augment diffusion models for protein-speciee 3d molecule generation. Forty-first International Conference on Machine Learning. 2024.
Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi S Jaakkola. Iterative refinement graph neural network for antibody sequence-structure co-design. In International Conference on Learning Representations, 2021.
Peter T' Jones, Paul H Dear, Jefferson Foote, Michael S Neuberger, and Greg Winter. Replacing the haamllemmentrttyydeeemminn regions in a human antibody with those from mouse. Nature, 321 (6069):522-525 1986.
John Jumper, Richard Evans, Alexander Pritzel. Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool. Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):833588) 2021.
Xiangzhe Kong, Wenbing Huang, and Yang Liu. Conditional antibody design 3d equivariant graph translation. In The Eleventh International Conference on Learning Representations, 2022.
Xiangzhe Kong, Wenbing Huang, and Yang Liu, End-to-end full-atom antibody design. In Proceed- ings of the 40th International Conference on Machine Learning, pp. 17409-17429, 2023.
Paulina Kulyté, Francisco Vargas, Simon Valentin Mathis, Yu Guang Wang, José Miguel Hernández- Lobato, and Pietro Liò. Improving antibody design with force- els, 2024.
force-guided guided sampling in diffusion mod-
Gideon D Lapidoth, Dror Baran, Gabriele M Pszolla, Christoffer Norn, Assaf Alon, Michael D Tyka, and Sarel J Fleishman. Abdesign: A algorithm for combinatorial backbone design guided by natural conformations and sequences. Proteins: Structure, Function, and Bioinformatics, 83 (8):1385-1400 1406, 2015.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrievallaugmennee genera- tion for knowledge-ing www.legg-intensive nlp tasks. Advances Neural Information Processing Systems, 33: 9459-9474 9474, 2020.
Haitao Lin, Lirong Wu, Huang Yufei, Yunfan Liu, Odin Zhang, Yuanqing Zhou, Rui Sun, and Stan Li. Geoab: Towards realistic antibody design and reliable affinity maturation. bioRxiv, pp. 2024- 05, 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 37966637:11123-111 2023.
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6959-6969, 2022.
Shitong Luo, Yufeng Su, Xingang Peng. Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen-specine antibody design and optimization with diffusion-basee generative models for protein structures. Advances in Neural Information Processing Systems, 35:9754-9761 2022.
12

---

Published as a conference paper at ICLR 2025
Karolis Martinkus, Jan Ludwiczak, Wei- Wei-Ching Liang, Julien Lafrance-Vanasse. Isidro Hotzel, Arvind Rajpal, Yan Wu, Kyunghyun Cho, Richard Bonneau, Vladimir Gligorijevic, et al. Ab- diffuser: full-atom generation of in-vitro functioning antibodies. Advances in Neural Information Processing Systems, 36, 2024.
Tobias H Olsen, lain H Moal, and Charlotte M Deane. Ablang: an antibody language model for completing antibody sequences. Bioinformatics Advances, 2(1):vbac046, 2022.
Leonard G Presta. Antibody engineering. Current Opinion in Structural Biology, 2(4):593-596, 1992.
Jiahua Rao, Zifei Shan, Longpo Liu, Yao Zhou, and Yuedong Yang. Retrieval based knowledge aug- mented vision language pre-training. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 5399-5409, 2023.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. International Conference on Machine Learning, pp- 0844-8856. PMLR, 2021.
Matthew IJ Raybould, Claire Marks, Konrad Krawezyk, Bruck Taddese. Jaroslaw Nowak, Alan P Lewis, Alexander Bujotzek, Jiye Shi, and Charlotte M Deane. Five computational developability guidelines for therapeutic antibody profiling. Proceedings of the National Academy of Sciences, 116(10):4025-43300 2019.
Jeffrey A Ruffolo, Jeffrey J Gray, and Jeremias Sulam. Deciphering antibody affinity maturation with language models and weakly supervised learning. arXiv preprint 1rXiv:211..07782, 2021.
Amir Shanehsazzadeh, Julian Alverio, George Kasun, Simon Levine, Jibran A Khan, Chelsea Chung. Nicolas Diaz, Breanna K Luton, Ysis Tarter, Cailen McCloskey, et al. In vitro validated antibody design against multiple therapeutic antigens using generative inverse folding. bioRxiv, pp. 2023-12, 12, 2023a.
Amir Shanehsazzadeh, Sharrol Bachas, Matt McPartion, George Kasun, John M Sutton, Andrea K Steiger, Richard Shuai, Christa Kohnert, Goran Rakocevic, Jahir M Gutierrez, et al. Unlocking de novo antibody design with generative artificial intelligence. bioRxiv, pp. 2023-01, 2023b.
Varun R Shanker, Theodora UJ Bruun, Brian L Hie, and Peter S Kim. Unsupervised evolution of protein and antibody complexes with a atructure-informe language model. Science, 385(6704): 46-53, 2024.
Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. knn-diffusion: Image generation via large-scale retrieval, In The Eleventh International Conference Learning Representations, 2023.
Hiroki Shirai, Akinori Kidera, and Haruki Nakamura. H3-rules: identification of cdr-h3 structures in antibodies. FEBS letters, (55(1-2):188-19))) 1999.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Pietro Sormanni, Francesco Aprile, and Michele Vendruscolo. Rational design of antibodies targeting specific epitopes within intrinsically disordered proteins. Proceedings of the National Academy of Sciences, 112662);9022-9907, 2015.
Amelia Villegas- Morcillo, Jana Weber, and Marcel Reinders. sequence and structure co-design with developability properties. In NeurIPS 2023 Generative Al and Biology (GenBio) Workshop, 2023.
Guiding diffusion models for antibody
Danqing Wang. YE Fei, and Hao Zhou. On pre- pre-training language model for antibody. In The eleventh international conference on learning representations, 2023a.
13

---

Published as a conference paper at ICLR 2025
Zichao Wang, Weili Nie, Zhuoran Qiao, Chaowei Xiao, Richard Baraniuk, and Anima Anandkumar. Retrieval-basee controllable molecule generation. In The Eleventh International Conference on Learning Representations, 2023b.
Fang Wu and Stan Z. Li. A hierarchical training paradigm for antibody structure-sequence co-design. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 31140-31157 Curran Associates, Inc., 2023.
John L Xu and Mark M Davis. Diversity in the cdr3 region of vh is sufficient for most antibody specificities. Immunity, 13(1):37-45, 2000.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In The Twelfth International Conference on Learning Representations, 2024.
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmentte language models robust to irrelevant context. In The Twelfth International Conference on Learning Repre- sentations, 2024.
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmeene motion diffusion model. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 364-373, 2023a.
Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian Yun Nie. Retrieve anything to augment large language models. arXiv preprint N.i::2210000554. 2023b.
Zaixiang Zheng, Yifan Deng, Dongyu Xue, Zhou, Fei Ye, and Quanquan Gu. Structure-informed
language models are protein designers. In International conference machine learning, pp- 42317-42338. PMLR, 2023.
Jianfu Zhou and Gevorg Grigoryan. Rapid search for tertiary fragments reveals protein sequence- structure relationships. Protein Science, 24(4):508-52 524, 2015.
Xiangxin Zhou, Dongyu Xue, Ruizhe Chen, Zaixiang Zheng, Liang Wang, and Quanquan Gu. Antigen-specific antibody design via direct energy- based preference optimization. arXiv preprint arXiv:2403.16576. 2024.
Tian Zhu, Milong Ren, and Haicang Zhang. Antibody design using a score-based diffusion model guided by evolutionary, physical and geometric constraints. In Forty-firs International Confer- ence on Machine Learning, 2024.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL DETAILS
A.1 MODEL DETAILS
For feature dimensions, we set the single residue feature dimension 128 and the pair feature dimension to 64. We leverage 6 IPA layers to capture geometry information. ESM2 650M is utilized in our model to create the embedding of antihody sequences, and the embedding dimension is 1280. In the local and column self-attention). The embedding dimension is 384, the hidden dimension is 1536, and number of attention heads is 6.
network, two layers of axial attention were used (two tied row self-attention
A.2 IMPLEMENTATION DETAILS
Our model was developed and executed within the PyTorch framework. For training, We chose the Adam optimizer with learning rate of 0.0001, weight decay of 0.0, and momentum parameters betal and beta2 set to 0.9 and 0.999, respectively. To dynamically adjust the learning rate, we employed plateau learning rate scheduler. When the validation loss plateaued, the learning rate was reduced by a factor 0.8, with a minimum learning rate set to 5e-6. The scheduler's 's patience was set to 10 epochs. The batch size is 8 during training. We design 8 samples for each CDR in the test set. All experiments are run on a single RTX4090 GPU, with a memory storage of 24GB.
Due to the high variability and specificity of the CDRH3 region, and it is considered the most criti- cal part in determining antigen-antibody binding. We conducted separate training for the sequence design of this region, adding and removing noise only for the CDRH3 region in each training itera- tion, with a total of 100,000 iterations. The other five regions, being more conserved, were trained together for a total of 250,000 iterations (approximately equivalent to 50,000 iterations per region). The reverse generation process time step is set to 100.
A.3 IMPLEMENTATION OF STRUCTURAL RETRIEVAL
The input consists of the backbone atom coordinates of each amino acid in the CDR region, forming set of coordinate points X. T represents the structures of all proteins in the PDB database. A represents a set of protein fragments representing CDR-like fragments corresponding to the input CDR structure. C represents the set of fragments of each structure with length of J represents a linear motif centered on residue j in the structure, with a length equal to the query fragment.
Assume that the coordinates of residue j are aligned with the central residue of X, and then com- pute the RMSD of X' when aligned onto т. If the input contains discontinuous multiple structures, cRMSD will be the cumulative RMSD of these structures. MaxA, maxB, and maxC are three dif- ferent upper-bound thresholds. These thresholds are selected to improve the speed and accuracy of the retrieval algorithm (For detailed proof, please refer MASTER (Zhou & Grigoryan, 2015)).
A.4 IMPLEMENTATION OF CDR-LIKE DATABASE CONSTRUCTING
To eliminate the computational overhead caused by structural retrieval during the model's training and inferencing, we followed previous work (Aguilar Rangel et al., 2022 and initially executed the retrieval algorithm on all CDR structures of all antibodies to construct a CDR-like database.
Each of the CDR structures used as a query to search for structurally similar motifs in the PDB database. The MASTER algorithm is used to match all CDRs against the entire PDB database to find CDR-like structures. This structural search based on the Kabsch algorithm, using the RMSD of the Ca coordinates. For CDR fragments of length 4, the RMSD threshold is 0.4, and the threshold is increased by 0.05 for each additional residue (with a maximum threshold set to 1.0 Å). In this way, obtain CDR-like fragments database corresponding to all CDR structures. Except for strictly filtering out results identical to real CDR sequences, no CDR sequence information was leaked in this process.
15

---

Published as a conference paper at ICLR 2025
A.5
BASELINE DETAILS
A.5.1
TRADITIONAL METHODS
Rosetta-Fixbl AAdoff Bryfogle et al. 2018) Rosetta-Fixee can use energy functions for antibody
CDR sequence design. Since DiffAb has already been proven outperform (Luo et al. 2022 Wu
& Li 2023) on sequence design task, we did not conduct additional comparisons.
Grafting To simulate the rational design commonly used in traditional antibody design methods, which often involves grafting CDR loops. For each CDR region, we directly selected the top-1 frag- ment(best) from the retrieval database for the structures in the test and replaced the corresponding original CDR loop sequences with it.
A.5.2 DEEP LEARNING METHODS
ProteinMPPN Dauparas et al. 2022) ProteinMPNN is a deep learning framework for protein se- quence inverse folding. It leverages message passing neural network model the complex rela- tionships between amino acids in protein structure. We use the antibody's backbone structure as input and keep the sequences outside the CDR regions to be designed fixed. We design sequences for each CDR region separately. The sampling temperature is set to the default value of 0.1.
Esm-IF1 (Hsu et al. 2022) Esm-IFI is a protein sequence inverse folding model trained on millions of AlphaFold2 predicted structures. We use the antibody's s backbone structure input and keep the sequences outside the CDR regions to be designed fixed. We design sequences for each CDR region separately. The sampling temperature is set to the value of 0.2.
Diffab-fix (Luo et al., 2022) Diffab diffusion model that can design sequences of CDR re- gion with fixed CDR backbone. It takes antigen-antibody framework context condition to design CDR sequence. For fair comparison, we retrained it with the default training configuration fixb
ABMPNN (rreyer et al. 2023 AbMPNN is fine-tuned by antibody structure data and predicted OAS (Observed Antibody Space structure data. Its model architecture consistent with Protein- MPNN but achieves better performance in antibody inverse folding. We use the antibody's backbone structure as input and keep the sequences outside the CDR regions to be designed fixed. However, it is not open-sourced yet, so we evaluate it on its own test set. We design sequences for each CDR region separately. The sampling temperature set the default value of 0.
A.6 EXPERIMENT ON CDR-H3's LENGTH
90
80
80.00
Diffab
70.52
AbMPNN
RADAb
Amino adi recovery
60.00
61.44
60
54.32
56.85
50.07
48.55
52.30
50
42.02
30
20
0
0-5
6-9
10-14
15-30
CDRH3 length
Figure S1: AAR distribution of different CDRH3 length
16

---

Published as a conference paper at ICLR 2025
Method
AAGI
AAG-seq
IMP-seq(3))
F-top1
F-top2 +
F-top3 +
Grafting
135.17
40.22
32.69
ProteinMPNN
127.14
24.72
35.51
-54.63
-45.58
-35.70
ESM-IFI
162.09
42.28
33.33
-62.65
-48.81
-33.86
Diffab-fix
116.36
14.05
34.52
-62.51
-54.80
-46.13
RADAb
109.16
7.06
37.30
-69.30
-55.95
-45.96
Table 5: Detailed results of antibody functionality optimization, F means functionality, which refers A AAG-seq.
We further evaluated each model's AAR across different CDR-H3 lengths. As shown Figure ST. although the performance of all models decreases with increasing H3 length, our method still outperforms the others.
A.7 DETAILED RESULTS OF ANTIBODY FUNCTIONALITY OPTIMIZATION
To further evaluate the model's performance in optimizing antibody functionalityy we additionally assessed the A&G of the top-1 to top-3 structures generated by the model. The results are shown in Table which further demonstrate that the antibodies optimized by our model exhibit improved functionality.
B STRUCTURE RECONSTRUCTION
To reconstruct the antibody structure, we use ABodyBuilder2 (Abanades et al., 2023). deep learn- ing model capable of predicting antibody light chain-heavy chain complexes. It significantly faster than AlphaFold2 and offers higher prediction accuracy. We insert the designed CDR sequences into the antibody framework sequence, input into ABodyBuilder2 to fold, and use OpenMM relax to obtain the structure corresponding to the new CDR sequences. Subsequently, we align the structure the real antibody framework. Finally, we use the fastrelax function in PyRosetta (Alford et al., 2017), with the score function set to ref2015 and max iteration set to 1000, to relax the structure.
C TRAINING AND INFERENCE ALGORITHM
In this section, provide detailed algorithm for the training (Algorithm 2 and inferencing (Al- gorithm 3) processes.
Algorithm Training Procedure of RADAb
1: Coordinates set X'
{xy 1 E (1...., m]}
2: A = Retrieval(
3: while not convergence do
4: t ~ Uniform( (I,
9( 53l6, ) 8) "199) o( s5119)
5:
(s) s9)
6:
S = Su Us}
7:
e' F(S)
8:
Context conditions C + (R², Cab, Cag
9:
( (C, Multinomial
10:
T = ERIPP 1 Σ3 DRa. (q (s)-1 ss, 1 |
11: F (C,ee)) "eet),G G (F(C,
(C, e²) (ee),A) A) + Adam (Lyge)
end while
13:
return G(),G(.)
17

---

Published as a conference paper at ICLR 2025
Algorithm 3 Sampling Procedure of RADAb
1: sy ~ Uniform (20)
2: Backbone coordinates set X' = {ax k E m]]
A Retrieval(X))
4: for = 7 to do
5: S = Str U sj
6:
e' = -(S)
7:
Context conditions C C Cob, Cags
8:
p p()) Multinomial +
9: sample 83 from (1)-
10: R
{8) 1Je (a a m\)
11: end for
12: return R
REF REF-8gnn
Samplel
Sample2
AG: 14.9 EGNYDGGSVRRYFDY
AG: -124 EGYSYYGGHYYFEY
41:315.3
KGYYIGGHMYYFDY
REF-3vg9
Sample1
Sample2
AG: 37.9 EGNYYDGGSVRYFDY
AG: -16.2 KGYYYNGGHYYFDY
AG: -17.6
KGYSYYGGHYYVFDY
REF-1hOd
iSamplel
Sample2
4G 8855 LGDYGYAYTMDY
AG: -8.7
AG: -17.6
WGPFGNYYAMDY
WGFYGNYYAMDY
Figure S2: Optimized antibodies with lower binding energy. The gray parts represent the antibody framework, the red parts indicate the designed CDR regions, and the blue parts represent the antigen.
D CASE STUDY
We select portion of the optimized antibodies in Figure S2 They achieved lower binding energy compared to the original antibody structures.
E THE MOTIVATION FOR ONLY CONSIDERING SEQUENCE DESIGN
Based on our observations, inverse folding represents more practical scenario. Current structure- sequence co-design methods typically involve masking the CDR while retaining the presence of an antibody framework backbone, which represents relatively uncommon use case. In most practi- cal scenarios, we either have access to the full complex structure the template antibody and the antigen, allowing to perform inverse folding, tating full atom and de novo design. This also why researches from pharmaceutical companies and efforts involving in vitro experiments antibody loop regions tend to focus more designing
lack template molecule entirely, necessi-
18

---

Published as a conference paper at ICLR 2025
antibodies through inverse folding, evidenced by several recent studies (Shanchsazzudeh al, 2023a by Frey et al. 2023 Hoie et al.) 2024] Shanker et al. 2024). This rationale underpins our decision to concentrate solely on this aspect in work.
Another reason is that performing eequence-structuue co-design while adhering to our retrieval- based approach would risk data leakage. This is because retrieval process relies the known structure of the CDR region, which is only possible when the CDR backbone is already known. At the same time, we recognize that epitope-specific full antibody de novo design, including full-atom design, is highly valuable. We are actively developing retrieval systems for PPI (Protein-Protein Interaction) retrieval to avoid potential data leakage and enhance our model.
F LIMITATIONS
Due to the inherent limitations of the MASTER algorithms, the retrieved linear motifs may have structural issues. Despite careful screening and filtering, tiny portion of the data might have lengths that differ from the original CDR loops or may even become discontinuous due to missing residues. These exceptional cases may have negative impact on our model. We hope that advances in structural retrieval and improvements alignment will jointly address this issue.
When constructing the CDR-like fragments database, searching the entire PDB database using all CDR structures from SabDab database takes approximately 100 hours. Additionally, the ESM2 encoding used to capture antibody sequence evolution information and the axial attention focused on local CDR in the denoising network require more computational resources than typical diffusion methods.
is important note that applying current retrieval mechanism to structure and sequence co-design poses significant risk of data leakage. Specifically, the retrieval process inherently relies using the CDR structure information as query, which practically transforms the task into one resembling inverse folding. However, one potential approach is incorporate PPI (Protein-Proteinn Interac-
tion) retrieval, and we are currently experimenting with MASIF (Gainza et al. 2020 2023) for this purpose. This will be explored further part of our future work.
19


Reviews:
[{"review_id": "mDfGlJjmKa", "reviewer": "Reviewer_vhAX", "summary": "This paper combines the popular technique RAG with antibody design by extracting templates from PDB to enrich evolutionary information, leading to better generalization ability. To be specific, the extracted fragments are based on backbone structures and some properties using a novel retrieval mechanism to utilize both structural and evolutionary information. Experiments demonstrate the effectiveness of the proposed RADAb framework.", "questions": "1. Is the provided experimental results correct? According to previous works, can $\\Delta \\Delta G$ be optimized on such a huge scale?  (~100kcal/mol?) Even for the selected examples in 5.2, $\\Delta \\Delta G$ only changed like ~-40, far away form claimed 109.16.\n2. Also, $\\Delta \\Delta G$ is usually expressed using negative numbers to demonstrate lower energy is achieved. Why $\\Delta \\Delta G$ is positive in Tab. 3? Suppose you are using the absolute value of $\\Delta \\Delta G$ , since lower $\\Delta G$ is better, why lower $|\\Delta \\Delta G|$ indicates better performance? Please explain this.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. I really like the idea of incorporating RAG into antibody design in such an elegant manner, which not only fulfills the relatively blank space of known data but also utilizes evolutionary information. This is intuitive and turned out to be effective.\n2. Nice illustrations, which clearly convey the architecture and designs of authors.", "weaknesses": "1. The experiment part is relatively weak. For example, for antibody functionality optimization task, only one simple experiment is provided.\n2. Baselines are inadequate. Large efforts have been devoted to studying antibody design, such as [1][2][3]:. They utilize the same experimental settings and metrics, which should be taken into consideration.\n3. If I comprehend correctly, although RADab is capable of capturing structural and sequential information, it is still a sequence design work, rather than sequence-structure co-design or even full-atom generation work, which is a major weakness compared to other methods.\n\n[1]Kong X, Huang W, Liu Y. End-to-end full-atom antibody design[J]. arXiv preprint arXiv:2302.00203, 2023.\n\n[2]Xiangzhe Kong, Wenbing Huang, and Yang Liu. Conditional antibody design as 3d equivariant\ngraph translation. arXiv preprint arXiv:2208.06073, 2022.\n\n[3]Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen-specific\nantibody design and optimization with diffusion-based generative models for protein structures.\nAdvances in Neural Information Processing Systems, 35:9754–9767, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pBAVCoPc4P", "reviewer": "Reviewer_Fxue", "summary": "This paper introduces a retrieval-augmented diffusion framework for anti-body design and optimization. Its key contribution is that it retrieves structurally-similar protein segments to the CDR, and use them as conditions in the diffusion generation. Experiments demonstrates impressive results, showing improved performances in antibody CDR sequence inverse folding and antibody functionality optimization. Ablation experiments show that both the retrieval augmentation and evolutionary embedding (ESM2) contribute to the improved performance.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: See the weaknesses below\n\nQuality: The paper overall has good quality. The experiment is sound.\n\nClarity: The paper is written clearly.\n\nSignificance: The paper significantly improves upon the baseline models in terms of the metrics. It has good significance.", "weaknesses": "Originality: There has been retrieval-augmented diffusion model proposed in other applications, such as in [1] and [2] (protein-specific 3D molecule generation). Although the paper is the first to introduce it in the antibody design, the method novelty is fair due to the prior works. The paper needs to also cite previous retrieval-augmented generative models, and clearly state the contribution in terms of the method.\n\n\n[1] Blattmann, Andreas, et al. \"Retrieval-augmented diffusion models.\" Advances in Neural Information Processing Systems 35 (2022): 15309-15324.\n\n[2] Huang, Zhilin, et al. \"Interaction-based Retrieval-augmented Diffusion Models for Protein-specific 3D Molecule Generation.\" Forty-first International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i96yK71ln3", "reviewer": "Reviewer_6nkE", "summary": "In this paper authors propose a method to improve upon inverse folding methods by mining PDB for similar structural motifs and use an MSATransformer to take them into account when predicting the loop sequences, along side the usual structure-conditioned diffusion model as used in DiffAb. Inclusion of the mined structures improved the performance of the model in the chosen experiments inverse folding and energy optimization experiments.", "questions": "Did you investigate using a similar approach for structure and sequence re-design (as done by DiffAb)? This would expand the scope of the paper.\n\nWhen you ablate the numberk of reference loop sequences fed to the model (Figure 4), you show that the optimal number of sequences to use is 15 \"But when k exceeds 15, the additional information instead introduces noise to the model, leading to performance degradation.\". Do you have any motivation why this would be the case? I'd imagine that as long as the mined loops satisfy some RMSD/quality threshold, they should always be beneficial to the model.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Inclusion of the data-mined motifs from PDB with similar loop structures is a quite nice and intuitive improvement. That helps the performance of the model. The paper is clearly written and easy to follow. Ablation is performed on the key changes introduced, showing that they are all beneficial.", "weaknesses": "Authors state that other models \"typically create antibodies from scratch without template constraints, leading to model optimization challenges and poor generative capability\". This is a rather harsh critique and I would like such sentences to be backed either by citations or experiments directly showing that. It is true that the proposed approach did generally better than baselines in the chosen experiments, but the difference to DiffAb was marginal and there can be a multitude of reasons for that difference.\n\nIn the energy optimization experiment authors say that authors use structures \"folded from the original antibody sequences\" to evaluate the energy. Folding can be arbitrarilly bad in determining the loop strucutres (esp. CDR H3). So this might very much end up measuring just the agreement between the folding method and the method being tested (how common the folding method finds the sequence to be) instead of how good the energy trully would be. If sequence recovery is reasonably high, one could use grafting and MD to get higher quality/more plausable complexes. Actually, what is the RMSD to original crystal structure, of the reference structure your approach produces (RMSD of original co-crystal vs folding original Ab and runing Rosetta fast relax)?\n\nWhile in general I like the idea of data mining the PDB for similar loops and using them as reference. I find that the paper is a marginal improvement on existing literature and is of somewhat limited scope for a ICLR paper (only doing inverse folding, while say DiffAb can also do complete loop sequence + structure redesign, and having only two experiments, one of which is not fully convincing to me).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)