A Hierarchical Training Paradigm for Antibody Structure-sequence Co-design
Authors: Fang Wu
Stan Z. Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Antibody Design

Abstract: Therapeutic antibodies are an essential and rapidly flourishing drug modality. The binding specificity between antibodies and antigens is decided by complementarity-determining regions (CDRs) at the tips of these Y-shaped proteins. In this paper, we propose a \textbf{h}ierarchical \textbf{t}raining \textbf{p}aradigm (HTP) for the antibody sequence-structure co-design. HTP consists of four levels of training stages, each corresponding to a specific protein modality within a particular protein domain. Through carefully crafted tasks in different stages, HTP seamlessly and effectively integrates geometric graph neural networks (GNNs) with large-scale protein language models to excavate evolutionary information from not only geometric structures but also vast antibody and non-antibody sequence databases, which determines ligand binding pose and strength. Empirical experiments show HTP sets the new state-of-the-art performance in the co-design problem as well as the fix-backbone design. Our research offers a hopeful path to unleash the potential of deep generative architectures and seeks to illuminate the way forward for the antibody sequence and structure co-design challenge.

A Hierarchical Training Paradigm for Antibody Structure-sequence Co-design
Fang Wu
Stan Z. Li Westlake University Hangzhou, China
Tsinghua University
Beijing, China
Abstract
Therapeutic antibodies are essential and rapidly expanding drug modality. The binding specificity between antibodics and antigens is decided by complementarity- determining regions (CDRs) the tips of these Y-shaped proteins. In this paper, we propose a hierarchical training paradigm (HTP) for the antibody sequence-structure co-design. HTP consists of four levels of training stages, each corresponding a specific protein modality within a particular protein domain. Through carefully crafted tasks in different stages, HTP seamlessly and effectively integrates geo- metric graph neural networks (GNNs) with large-scale protein language models to excavate evolutionary information from not only geometric structures but also vast antibody and non-antibody sequence databases, which determines ligand binding pose and strength. Empirical experiments show that HTP sets the new state-of- the-art performance in the co-design problem as well as the fix-backbone design. Our research offers hopeful path to unleash the potential of deep generative architectures and seeks to illuminate the way forward for the antibody sequence and structure co-design challenge.
Introduction
Antibodies, known immunoglobulins (lg), are large Y-shaped proteins that the immune system uses to identify and neutralize foreign objects such as pathogenic bacteria and viruses [] 2) They
recognize a unique molecule of the pathogen, called an antigen. As illustrated in Figure (a),
each tip of the "Y" of an antibody contains a paratope that is specific for one particular epitope on an antigen, allowing these two structures to bind together with precision. Notably, the binding specificity of antibodies is largely determined by their complementarity- -tettaatty-dtttemiinng regions (CDRs). Consequently, unremitting efforts have been made automate the creation of CDR subsequences with desired constraints of binding affinity, stability, and synthesizability space vast, with up to 20t possible combinations for a I--length CDR sequence. This makes infeasible to solve the protein structures and then examine their corresponding binding properties via experimental approaches.
[3. However, the search
remedy, a group of computational antibody design mechanisms has
been introduced to accelerate this filtering process.
Some prior studies because it lacks valuable geometric information. Meanwhile, since the target structure for antibodies rarely given as a prerequisite (6), more attention has been paid co-designing the sequence and structure. One conventional line of research [7]]] resorts to sampling protein sequences and structures on the complex energy landscape constructed by physical and chemical principles. But it is found to be time-exhausted and vulnerable to being trapped in local energy optima. Another line [IO [2 relies on deep generative models to simultaneously design antibodies' sequences and structures. They take advantage of the most advanced geometric deep learning (DL. techniques and can seize
prefer generate only ID sequences, which has been considered suboptimal
Corresponding Authors, emails; fv23598columba..com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Asligen
Binding Site
Antibody annigen Complex Data (3D)
SabDab -10K
Task Specificity
Training Order
Proteta-pootela Complex Data (3D)
DIPS-4UK
Data Size
Domain
Antiooy Sequance Data (1D)
OAS -172K
Domain
1 Large scale Protein Sequetce Data (1D)
UniProt -8
(a)
(b)
Figure 1: (a) Schematic structure of antibody bonded with an antigen (figure modified from
Wikipedia). (b) The workflow overview of our hierarchical training paradigm (HTP).
higher-order interactions among residues directly from the data [3] Their divergence mainly lies in their generative manner. For instance, early works [10 [
adopt an iterative fashion, while successors [14 15] employ full-shot generation. Most utilize the traditional translation framework,
while some 12 leverage the diffusion denoise probabilistic model.
Despite this fruitful progress, the efficacy of existing co-design methods is predominantly limited by the small number antibody structures. The Structural Antibody Database (SAbDab) [6] and RAbD are two widely used datasets in the field. After eliminating structures without antigens and removing duplicates, SAbDab comprises only a few thousand complex structures, whereas RAbD consists of 60 complex structures. These numbers that can inspire major breakthroughs in DL areas [17 18]. Consequently, deep-generative models fail benefit from large amounts of 3D antibody-antigen complex structures and are of limited sizes, or otherwise, overfitting may occur.
orders of magnitude lower than the data sizes
To address this issue, in this paper, propose hierarchical training paradigm (HTP), novel unified prototype to exploit multiple biological data resources, and aim at fully releasing the potential of geometric graph neural networks (GGNNs) [19] 20] for the sequence and structure co-design problem. Explicitly, HTP consists of four distinct training ranks: single-protein sequence level, antibody sequence level, protein- pentein-protein complex structure level, and antibody-antige complex structure level. These steps are itemized as the data abundance declines, but the task specificity increases, depicted in Figure (b). Alongside them, present various pretraining objectives for the first three levels to mine correlated evolutionary patterns, which benefit the co-design task in the final stage. Specifically, we first pretrain the protein language models (PLMs) tremendous single-protein sequences to obtain general representations and fine-tune them on antibody sequences capture more condensed semantics. For 3D geometry, we invented pocket-painting task exploit the protein-protein complex structures and simulate the CDR generation process. After that, we combine the marvelously pretrained PLMs and the well-pretrained GGNNs co-design the expected antibodies. Our study provides a promising road to excavate the power of existing deep generative architectures and hopes to shed light on the future development of antibody sequence and structure co-design. The contributions of our work can be summarized as follows.
First, we equip GGNNs with large-scale PLMMs to bridge the gap between protein databases of different modalities (i.e., 1D sequence, and 3I) structure) for the antibody co-design problem.
Second, we design four distinct levels of training tasks to hierarchically incorporate protein data of different domains (i.e., antibodies, and ant-aniibodies) for the antibody co-design challenge. HTP breaks the traditional co-design routine that separates proteins of different domains and extends antibody-antigen complex structures to broader databases.
Comprehensive experiments have been conducted to indicate that each stage of HTP significantly contributes to the improvements in the capacity of the DI. model predict more accurate antibody sequences and restore its corresponding 3D structures. To be explicit, HTP brings a rise of 78.56% in the amino acid recovery rate (AAR) and a decline of 41.97% in structure prediction error for the sequence-structure co-design. It also leads to an average increase of 26.92% in AAR for the fixed-backbone design problem.

---

Level-1
Level-11
CDR,
CDR
CDR₂
FC-layers
Protein Language Models MPL (eg. ESM-2, ESM-MSA, ProfTrans)
Transforerr based Models
FW;
<MASK>
FW,
<MA
FW,
<MASK>
FW,
<MASK>
Single Protein Sequences (eg, UniRef50, UniParc)
Antibody Sequences (og. OAS,ABCD)
Level-III
Protein-proteie Complexes Structures (eg, DIPS, DB5.5)
Chain
Chain A
Chain
Pseudo-CDR
R
Goometric Graph Neural Networks (eg, EGNN, SE(3)- Trunsforrrr, GVP-GNN)
Chain
Figure 2: The illustration of the first three stages of our hierarchical training mechanism for antibody sequence-structure co-design. In level I, Transformer- language model is trained by masked language modeling on large number of single protein sequences to extract general-purposed representations. In level II, the Transformer-bant antibody sequences from databases like OAS or ABCD. CDRs are all masked and require recovery, with other framework regions reserved. In level III, GGNNs are asked predict both the sequence and structure of the pseudo-CDR on poteiin-poteent complex structure databases. The residue features are based on PL.Ms gained the previous step.
PLM is then further fine-tuned
specific
2 Methods
This section is organized as follows. Subsection 2.1 describes the background knowledge and mathe- matical formulation the co-design problem. Subsection intruuuus the backbone architecture to encode the geometric structure of protein-protein complexes as well as antibody-antig complexes the 3D space. Subsection 2.2 to 2.6 concentate on explaining the four different levels our HTP.
2.1 Preliminary
Background An antibody is Y-shaped protein with two symmetric sets of chains, each consisting of heavy chain and light chain. Each chain has one variable domain (VH/VL) and some constant domains. The variable domain can be further divided into a framework region (FW) and three CDRs. Notably, CDRs in heavy chains contribute the most to the antigen-binding affinity and are the most challenging characterize.
Notations and Task Formulation We represent each antibody-antiger complex as a heterogeneous graph Gr.-. It is made up of spatially aggregated components, i.e., the antibody and antigen denoted as Gr = (Vz, E and GR = (VR- &R), respectively. 9L and GR use residues as nodes with numbers of Nr, and NR separately. The node locations X WN x3 and XR E Nax3 are defined as their corresponding a-carbon coordinate, and are associated with the initial vh-- dimension roto tio-trrnnslatinn invariant features ha Xvs and h E & (e,g. residue types, electronegativity) CDRs are subgraphs of GL and can be divided into GHC = (VHC. Ehcl and ILC {VLc, ELC), which belong to the heavy chain and the light chain, respectively. We assume that Ghe and GLC have Nис and NLC residues. Besides, it is worth noting that the distance scales between internal and external interactions are very different. Based on this fact, we strictly distinguish the interaction within and across two graphs GR and Gr. as and E,, U En and Ei,R. individually. This implementation avoids the underutilization of cross-graph edges' information due to implicit positional relationships between the antibody and the antigen (21].

---

In this work, we hypothesize that the antigen structure and the antibody framework are known, aiming to design CDR with more efficacy. Formally, our goal jointly model the distribution of CDR in the heavy chain given the structure of the remaining antibody-antigen complex as p(Gnc (9 Gnc) or in the light chain as p(9cc|91. 1 ILC). Since the heavy chain plays a more critical role in determining antigen binding affinity, we take 9нC as the target example in the following content, but design both heavy and light chains in the experiment section 3.1
2.2 Single-protein Sequence Level
The idea that biological function and structures are recorded in the statistics of protein sequences selected through evolution has long history [22]. Unobserved variables that determine the fitmess of protein, such as structure, function, and stability, leave a mark on the distribution of the natural sequence observed (23 To uncover that information, a group of PL has been developed at the scale of evolution, including the series of ESM [22] and ProtTraas 24 They are capable of capturing information about secondary and tertiary structures and can be generalized across a broad range of downstream applications. Recent studies (25) also demonstrate that equipping GGNNs with pretrained language models can end up with a stronger capacity. Accordingly, we adopt an ESM-2 with 150M parameters to extract per-residue representations, denoted as h', E R R.Pra and h/2 R-NXXWLLM, and use them as input node features. Here ULLM = 640 and we use + to denote the trainable parameter set of the language model.
Noteworthily, ESM-2 supplies plenty of options with different model sizes, ranging from 8M to 15B, and a persistent improvement has been found as the model scale increases. However, the trajectory of improvement becomes relatively smooth after the scale reaches 10 Therefore, for the sake of computational efficiency, we select the 150M version of ESM-2, which performs comparably with the 650M parameters of the ESM-1b model [22). As declared by Wu et al. (26), incompatibility exists between the experimental structure and its original amino acid sequence (i.e., FASTA sequence). For simplicity, we obey Wu et al. [26]'s 's mode, which uses the fragmentary sequence directly as the substitute for the integral amino acid sequence and forwards it to the PLMMs
2.3 Antibody Sequence Level
Though PLMs have achieved great progress within protein informatics, their protein representations are generally purposed. Noticeably, the residue distribution of antibodies is significantly different from non non-antibodies. Over the past decades, billions of antibodies have been sequenced (27). which enables the training of language model specifically for the antibody domain (28) Several researchers have recognized this problem and present models such as AntiBERTa 29 and AbLang [30] to decipher the biology of disease and the discovery of novel therapeutic antibodies. Nonetheless, those pretrained antibody language models have some intrinsic flaws and may not be suitable to apply in our sequence-structure structure co-design problem immediately.
First and foremost, they are directly pretrained on antibody sequence datasets and fail to exploit the vast amounts of all protein sequences. Second, the existing pretrained antibody language models are all on a small scale. For example, AntiBERTa consists of 12 layers with 86M parameters. Last but not least, both AntiBERTa and AbLang regard the heavy and light chains individually, and AbLang even trains two different models for them. This approach is precluded from encoding comprehensive and integral representation of the whole antibody. More importantly, they are pretrained on antibodies and cannot be perfectly generalized to extract representations of antigens, which is necessary for analyzing the interactions between antibodics and antigens.
To avoid their braks we chose to fine-tune ESM on available antibody sequence datasets and considered both antibody and antigen sequences. Following AbLang (30, leverage the Observed Antibody Space database (OAS) 31 and subsequent update [32] OAS is project that collects and annotates immune repertoires for use in large-scale analysis. contains over one billion sequences, from over 80 different studies. These repertoires cover diverse immune states, organisms, and individuals. Additionally, Olsen et al. (30 has observed OAS that approximately 80% sequences lack more than one residue at the N-terminus, nearly 43% of them are missing the first 15 positions, and about 1% contain at least one ambiguous residue for each sequence. Remarkably, OAS contains both unpaired and paired antibody sequences and we utilize only paired ones. To better align with our

---

co-design target, we implement masked language modeling (MLM) and mask residues in all CDRs (i.e., VH, and VL) simultaneously increase the task difficulty.
2.4 Geometric Graph Neural Networks
To capture 3D interactions of residues in different chains, we adopt variant of equivariant graph neural network (EGNN) B3] act on this heterogeneous 3D antibody-antigen graph. The architecture has several key improvements. First, it consists of both the intra- and inter- message- message-passing schemes to distinguish interactions within the same graph and interactions between different counterparts. Second, it only updates the coordinates of residues in CDR, ie., the part that is to be designed, while the positions of other parts are maintained as unchangeable. Last, it uses features from the pretrained PLMSs as the initial node state rather than a randomized one. Here, all modules are E(3)-equivariant.
The (-th layer of our backbone formally defined as the following:
(1) mj-si = de (h.² hj d xi (2xx5)))) Veri E Et. U
Mj-s = ag ay (1)
Ød (d (x .x(",xx()) Vei ELR
(2)
h (1+1) ob
Σ my-i. Emy-n h (1
(3)
5'
where d(...) the Euclidean distance function. de is the edge operation and d, denotes the node operation that aggregates the intra-gray intra-graph messages m, -E, my- and the cross-graph message
14 = Ey -y as well as the node embeddings h,") to acquire the updated node embedding h ((+1), da operates on the inter-atomic distances. Φe, Φh. and da are all multi-layer perceptrons (MLPs). Besides that, aj-zs an attention weight with trainable MLPs 6² and Φ², and takes the following form as:
exp ((00) (h!"))
6 (n,)))
0j-t
Ey exp ((05 h ok
As for coordinate iterations, we note that residues located in CDRs (i.e. GHC) are the sole con- stituent that needs spatial transformation. On the contrary, the position of the remaining piece (i.e., GLR GHс) is ascertainable. If we change the coordinate of GLR - GHc, its conformation can be disorganized and irrational from physical biochemical perspectives. Therefore, it is reasonable to stabilize GLR - Gnc in each layer and simply alter Gnc. Mathematically,
x + WT EIEN. (x9- xx") (z(1,)), if E 9HC (
(41))
(5)
otherwise
where N₁ denotes the neighbors of node i and take the mean aggregation update the coordinate for each movable node. Φz varies according to whether the edge Gij represent intra-graph connectivity or cross-graph connectivity. In particular, dx Qm (m,+)) eij E E/4 u e( Otherwise, (t)) do = d (Km-)) when eij E &in where om.. and o, are two different functions to deal with different types of messages. Then, du is left multiplied with x(" -x,) to keep the direction information. Equation (5 takes as input the edge embedding m,++ or μ4ーク as a weight sum all
relative coordinate xX") (1) and output the renewed coordinates x! (1+1)
To summarize, the 1-th layer of architecture (1 (L.) takes input the set of atom embeddings (1)
h. (1) h }. and 3D coordinates {x19 xx }- Then it outputs a transformation on (1) (1)
R
as well as coordinates of residues on the CDRs, that is, x XHC. Concisely, h ((+1) ((+1) (1+1)
.xiC h
(G (1) x AR R
(1)
while the coordinates of other non-CDR parts remain the same as
in the last layer as x set of the whole GGNN architecture.
((+1) Ux M
(1+1), (1+1) = x Ux(( x xHC We assign θ the trainable parameter
5

---

Level-IV
Antigen
Antigen
K
Sequence
P
Neural Network
Completed A Antiboyy
FW
L
CDR
?
Antibody
FW,
M
Sequence
FW,
..LLNRE...YGDSVV
FW,
Framework Region (FR)
Antibody-antiges Complex Structures (eg, SAbDab)
nighhthroughhh.. Evaluation
The final stage our hierarchical training mechanism for antibody sequence-structure co-design. In level IV, both the pretrained language model and geometric graph neural networks employed to implement the co-design task with the parameters of the pretrained language model fixed. At last, the in silico antibody is experimentally validated via seg-tthooughput binding quantification.
Figure 3:
2.5 Protein-protein Complex Structure Level
Apart from empowering PLMs with tremendous protein sequences, plenty of protein-protein complex structures stay uncultivated, which can be harnessed to promote the geometric backhone architecture. Despite the emergence of several structure-based pretraining methods the biological domain [2]] 34), it is not trivial to apply them to our antibody design problem because all previous studies focus on single protein structures. order enable GGNNs encode the general docking pattern between multiple proteins, we propose the pocket inpainting task.
Specifically, used the Database of Interacting Protein Structures (DIPS) [35. DIPS is much larger protein complex structure dataset than existing antibody-antige complex structure datasets and is derived from the Protein Data Bank (PDB) [36] In DIPS, each complex GL. has two sub-units Or and GR. We calculate the distance between each amino acid of these two substructures and select residues whose minimum distance the counterpart substructure is less than the threshold cp SÁ as the pocket Gp. Mathematically, the pocket nodes Vp is written as follows:
Ni UL,i min d(x xn.) < <(p) eni min d(xns, XL.) < <6p
NR
Vr
(6)
then our target is to retrieve Gr given the leftover GLR Gp. Here, We follow the official split based on PDB sequence clustering 30% sequence identity level to ensure little contamination between ,050/15,268 complex samples.
sets. It results in train/val/test of
2.6 Antibedy-antigen Complex Structures Level
After the preceding levels of preparation, is time to co-design the antibody. Given an antigen G2 and fractional antibody Gr. GHC, first employ the well-trained PLM + to attain per-residue node features (0) and h where the nodes in the unknown part ДиС C are tagged as masked token. Then both features and coordinates fed into the well-trained geometric encoder to unravel the CDR se-
quences and structures in concurrent way, i.e., h xHC R EGNNe (o) XL h x
(L)
(0)
(0)
Eventually, we use an MLP d and a Softmax operator as the classifier to output the probability distribution of residue types p4 Softmar (oo h (n(()) E R220 for U; E VиC-
CDR Coordinates Initialization. How to initialize the positions of residues in CDRs is of great im- portance to the co-design problem, and there is no consensus among different approaches. HERN [I]] tries two kinds of strategies. One strategy is to randomly initialize all coordinates by adding a small
Ejeg, X, +, No 1). The other is predict the pairwise distance instantly D (N (N2
Gaussian noise around the center of the epitope as x (0)
NR.
between paratope and epitope atoms

---

Table 1: Results of sequence and structure co-deign SAbDab.
Model
CDR-H1
CDR-HZ RMSD
CDR-H3
AAR
RMSD ]
TM-Score T
AAR (
TM-Skore
AAR (R)
RMSD
TM-Sccre
1.505
0.9206
2.85
0.9253 0.010
0.13
1.012
C-RGNN
30.39
1,98 0.00
0.380 2: 0.003
33.36 1:
1.32 0.05
0.9507 1: 0.005
21.89 1.5
3.59
0.16
09187
0:011
MEAN
4:.50
L.4 0.04
0.9411
a: 0.008
37.18 1.5
1.27
0,9522 4: 0.007
22.56 1.7
3.44
0.118
0.9248
0.009
HERN
2:27
1.99 0.004
0,9472
a 0.005
4..53
1.26.10.03
0.9531
4: 0.006
25.73 1.4
0.11
0.9340
0.004
DifAA
5..82+0.9
1.51
0,9688
0.001
45,95
1.24++000
0.9588
0.002
27.04
2
0.4417
00008
HTP
81,33 1.6
0.49 119::000 0.02
0,9829 = 0.006
67.77 :: 1.3
0.53 0.02
0,9860 :: 0.004
40.98 1.5
2.06 = 0.03
0.9621 0,005
Moddl
(%)
RMSD 1
CDR-LI
TM-Soore 1
AAR (%)
RMSD
CDR-L22
TM-Skore
AAR (5)
RMSD
CDR-L3
TM-Score
1.
188
0,9588
2..62:10..
1.35
0 0.009
23.73
2.14
0.9247
0.010
C-RGNN
41.44:22.55
2.06+00.02
0.9326 = 0.008
36.71 4.33
1.26-#:000 0:01
0.9682 # 0.006
33.80=
1.96=0.06
0.9008 +.0008
MEAN
47.69 2.3
L.7 0.00
0,9461 = 0.006
3.5
1.24::0.01
0.9647 :: 0.008
35.18 2.6
L.84 184.2:000
0.9370 4: 0.005
HERN
55.24-42.7
1,63 0.002
0.9502
+-1
1.18 1.18++0022
0.9712
37.28
1.77--0.00
0.9399
0.002
DifAAb
62.71
0.9637
1,11
43.62
HTP
91.13 1.0
0.67 1: 0.04
0,9866 =: 0.005
89.80 :: 0.8
1,03 :: 0.05
0,9846 :: 0.008
73.82 1.1
0.78 = 0.04
0.9704 1: 0,003
and reconstruct atom coordinates from this distance matrix. The results show that the former performs
better. DiffAB [2] initializes them from the standard normal distribution as x!" (N Ia). MEAN [4 leverage the even distribution between the residue right before CDRs and the one right after CDRs as the initial positions.
All of the above initialization mechanisms have corresponding drawbacks. To be specific, HERN insufficiently considers the context information since only characterizes the shape of the epitope and ignores the incomplete antibody structure Gr.R Gиc- The initialization of normal distribution only suitable diffusion- -based models. Moreover, our empirical experiments observe the least instability during training if an even distribution of MEAN is adopted, but residues right before or right after CDRs can be missing. Here, we propose another way to initialize the residue positions in CDRs. First, we follow MEAN and select the residue right before and right after CDRs. If both residues exist, we take the mean coordinates. Otherwise, we use only the existing one. After that, we add some little noise like HERN to separate nodes and introduce some randomization to prevent overfitting, which is proven to bring slight improvements in performance.
Loss Function. Two sorts of losses are used for supervision, First, cross-entropy (CE) loss is employed for sequence prediction as Lang = Vec EneVee CE(p;; G)), where e; the ground truth residue type for each node, Apart from that, a common RMSD loss is utilized for structure prediction
and we leverage the Huber loss (37 to avoid numerical instability as Lstract = Huber
X [HC xIe)
(L)
where the latter is the ground truth coordinates of the target CDR. The total loss is weighted sum of the above two as L = Loeg + XL where À >> 0 the balance hyperparameter.
Experiments
We assess our HTP via two mainstream challenging tasks: sequence-structure co-design in Section 3.1 and antibody sequence design based antibody backbones in Section [3.2] Our evaluation is conducted in the standard SAbDab database [16]. More experimental setting details and data
descriptions are elucidated in Appendix A
3.1 Sequence-structure Co-design
Task and Metrics. In this task, we remove the original CDR from the antibody-antige complex in the test set and try co-design both sequence and structure of the removed region. Here we set the length of the CDR to be identical to the length of the original CDR. But in practice, the lengths of CDRs can be variable. For quantitative evaluation, we adopt amino acid recovery (AAR) and coo--menn-squeee
error (RMSD) regarding the 3D predicted structure of CDRs as the metric. AAR defined as the overlapping rate between the predicted 1D sequences and the ground truths. We also take advantage of T'M score (38 to calculate the global similarity between the predicted and ground truth antibody structures. It ranges from and evaluates how well the CDRs fit into the frameworks.
7

---

Table 2: Results of the fix-backbone design task SAbDab.
CDR-HI
CDR-H2
CDR-H3
Model
AAR (%)
Perplexity
AAR (%) 1
Perplexity 4
AAR (S)
Perplexity
RosettaFix
36,29
14.78 0.01
12.30 0,3 0.02
28.13
24.05 0.14
Structured-11 IT:
53.24
8.61 1:008
49.87 1.2
10.27 = 0.06
30.29 0:4
19.65
0.11
DiffAb
59,91 +: 1.2
0.44
1..14
0.08
33.
16.84=0..
GVP-GNN
62.72 1.5
0.00
62.48
4.77
3..59
15.79
HTP
86.01 + 1.1
1.71 0.04
64.46
3.29 + 0.06
43.25 1.2
9.01 + 0.10
CDR-L1
CDR-L,2
CDR-L3
Model
AAR (%)
Perplexity
AAR (%) 1
Perplexity 4
AAR (%)
Perplexity
RosettalFix
0..
0.00
0.2
14.67 0.01
32.17
18.01 0.00
Structured-IF IF
36.73
0.10
1.8
8.93
43.48
13.88 0.02
DiffAb
58.82.1:
6.89
5.,40
0.06
47.31
10.60 0.02
GVP-GNN
60.18 1.4
5.48
1.5
0.006
1..3
8.27 0.00
HTP
93.62 1.5
1,09 + 0.08
91.46 1.7
1.58
80.71
2.66 + 0.10
Baselines. We pick up a broad range of existing mechanisms for comparison. Rosetta Antibody De- sign (RAbD) 9 is antibody design software based on Rosetta energy functions. RefineGNN [IO] is an regr-segees model that first consider 3D geometry for antibody design and is B(3)-invariant However, its original version is merely conditioned on the framework region rather than the antigen and the remaining part of the antibody. We follow Jin et al. [1 and replace its encoder with a message-passing neural network (MPNN) encoder and use the attention layer extract informa- tion from the antibody-antigen representation. We name this modified model C-RGNN to make a distinction from its original architecture. Hierarchical Equivariant Refinement (HERN) (II is the abbreviation of Hierarchical Equivariant Refinement Network, which employs a hierarchical MPNN to predict atomic forces and refine binding complex in an iterative and equivariant man- ner. Its autoregressive decoder progressively docks generated antibodies and builds geometric representation of the binding interface guide the next residue choice. Multichannel Equivariant Attention Network (MEAN) [1] adopts multi-round progressive full-shot schemee instead of an autoregressive one to output both 1D sequences and 3D structures. DiffAb [2] a diffusion-basee mechanism that achieves state-of ofte--of-hheart performance on antibody design recently. It consists of three diffusion processes for amino acid types, coordinates, and orientations, respectively.
Results and Analysis. We run each model three times with different random seeds and report the mean and standard deviation of each metric in Table [ where metrics are labeled with 144 if higher/lower is better, respectively. It can be found that our model outperforms all baselines by a significant margin in terms of AAR, RMSD, and TM-Score. Specifically, HTP brings improvement of 53. 97%, 47. 42% and 51. 55% in AAR and 67. 54%, 57. 25% and 29. 75% in RMSD over the state-of-the-aar DiffAb in H1, H2 and H3, respectively. This implies that our HTP might have higher success rate in designing new antibodies targeting the given antigen. Moreover, it can also be expected that the performance of our HTP can be further improved as the number of antibodies and antigens with solved 3D structures keeps increasing. In addition, the light chain generally has higher AAR and a lower RMSD than the heavy chain. For example, our model achieves nearly 90% AAR in CDR-LI and CDR-L2. This phenomenon is consistent with the fact that CDR in the heavy chain is much longer and more variant.
3.2 Fix-backbone Sequence Design
Task and Metrics. This problem is more straightforward than the previous co-design task. The backbone structure of CDRs is given and only requires the CDR sequence's design. The fixed- backbone design is common setting protein design [9 40) with an alias of inverse folding. We rely on the metrics of AAR introduced in Section 3.1 to examine the generated antibodies. The metric RSMD ruled out since the backbone structures are fixed. We also rely on perplexity (PPL. to understand the uncertainty of model predictions, which is common for evaluating language models in natural language processing. short, the perplexity calculates the cross entropy loss and takes its exponent.
8

---

Table 3: Comparison with pretrained antibody-specific language models.
CDR-H3
PLMs
AAR (%)
RMSD
TM-Scure
25.31 = 0.7
2.95 + 0,02
0.9391
0.005
AbLang
28.46 = 1.6
2.88 + 0.17
0.9435 = 0.009
AntiBERTa
34.52
1.2
2.41
0.13
0.9473
0.006
HTP 40.98 +: 1.5
2.06 1 0.03
0.9621 + 0.005
Baselines. Apart from some baselines that have been listed in the sequence-strructure co-design problem, we compare HTP with three additional approaches. RosettaFix 44 is Rosetta-based software for computational protein sequence design. Structured Transformer 42 (Structured- TF) is an auto-regression generative model that is able sample CDR sequence given the backbone structure. GVP-GNN 4 extends standard dense layers to operate on collections of Euclidean vectors and performs geometric and relational reasoning on efficient representations of macromolecules.
Results. Table 2 documents the result. It is clearly found that our model achieves the highest AAR and the lowest PPL compared to all the baseline algorithms. To be specific, HTP leads to an increase of 37.13339 13%. 3.01 %, and 18.47% in AAR over the best GVP-GNN in HI, H2, and H3, separately, and 28.76% 44.28%. and 49.90% in LI, L2, and L3, respectively. This demonstrates that our model is also effective in capturing the conditional probability of sequences given backbone structures. Furthermore, we can also observe that CDR-H3 is the hardest segment in comparison to other regions as its AAR is usually lower than 50%. Meanwhile, the average AAR of the light chain more than 75%, indicating that the light chain maintains less diversity than the heavy chain.
3.3 Discussion
Comparison with Antibody-specifie Language Models. Recently, emerging efforts have been paid to train large antibody--peccfic language models. Studies have also demonstrated that these models can capture biologically relevant information and be generalized to various antibody-related applications, such as paratope position prediction. Here, we make further investigation into the efficacy of these seiibody-specifff language models for sequence-structure structure co-design. To be precise, we abandon the pretraining stages of the single-protein protein and antibody sequence levels and directly leverage external -npebody-specific language models. As shown in Table AntiBERTa and AbLang provide biologically relevant information that is beneficial for the challenge of co-design with an increase of 12.44% and 36.38% in AAR. However, their improvements are much smaller than those of PLMs trained through the first two levels of tasks in HTP.
Up- and Downstream Protein. Recently, Wang et al. proposed a joint
recovery method based RosettaFold to scaffold functional sites of proteins. They perform the inpainting and fix-backbone sequence design tasks without immediate up- and downstream protein visible. However, we discover that our HTP can generate adequate diversity without the need to mask neighboring residues, that is,, up- and downstream proteins. This difference stems from the fact that our HTP considers the entire antigen and available antibody as the context to complete the masked CDR rather than only depending on the tiny context of up- and downstream protein.
Data Leakage of Language Models. It is undisputed that the evaluation of design methods that use PLMSs should be stringent and ensure that the test data were not previously seen by those pretrained models. However, ESM-2 is trained on all protein sequences in the UniRef database (September 2021 version), while our test set includes sequence released after December 2021, well structures with any CDR similar to those released after this date (with sequence identity higher than 50%). is fairly possible that the training set of ESM-2 includes antibody sequences similar to the test set, leading to the intolerable data leakage problem.
Here, we conducted additional experiments aimed at assessing the contribution of ESM-2 in directly recovering CDR sequences. Specifically,, we abandon the structural information and re-generate CDRs entirely based on sequential information. Towards this end, we first extract residue-level representations via (fixed-weight) ESM-2 and feed them to three-layer perceptron to predict the masked CDR-H3, where no antigen sequences given. The results show that this algorithm only

---

achieved an AAR of 14.63% to recover CDR-H3, much lower than all baseline methods such as RAdD (21.73%) and our HTP (40.98%). This compellingly demonstrates that ESM-2 is not the primary driver of our favorable numerical outcomes and the experimental benefits brought forth by HTP are not due to data leakage. This also accords with ATUE'S [5 findings that ESM models perform well in wow-antibody-speciii specffiiciiiii reeated tasks but can even bring negative impacts for high- seabbody-specifiiiyy--eee
tasks. We have also provided adequate evidence Appendix [ to show that the other pretraining resources are free from any possible data leakage concern.
4 Conclusion
Antibodies are crucial immune proteins produced during an immune response to identify and neu- tralize the pathogen. Recently, several machine learning- -baed algorithms have been proposed to simultancously design the sequences and structures of antibodies conditioned on the 3D structure of the antigen. This paper introduces novel approach called the hierarchical training paradigm (HTP) to address the co-design problem. It Leverages both geometric neural networks and large-scale protein language models and proposes four levels of training stages to efficiently exploit the evolutionary information encoded the abundant protein sequences and complex binding structures. Extensive experiments confidently show that each stage of HTP significantly contributes to the improvements of the deep learning model's capacity in predicting more accurate antibody sequences and storing its corresponding 3D structures. Instead of focusing on the architecture side, our study hopes shed light on how to better blend protein data of different modalities (i.e., one- and three-dimension) and domains (i.e., antibodies, and non-antibodies) for tackling the sequence-structure co-design challenge, which has been long ignored by existing works.
Limitations and Future Work
In spite of the promising progress of our HTP, there is still some space left for future explorations. First, more abundant databases can be exploited in our framework. For example, AntiBodies Chemically Defined (ABCD) [46) is large antibody sequence database that can be used to improve the capacity of protein language models at the second level. We do not use it in our work because our request for this database has not been approved by the authors far. Secondly, we fix the language models during the last two levels of training (i.e., levels that need complex structure prediction) for simplicity and use them as the node feature initializer. It might be beneficial if both the PLM and the geometric encoder are tuned.
Acknowledgments and Disclosure of Funding
This work was supported by National Key R&D Program of China (No. 2022ZD0115100 National Natural Science Foundation of China Project (No. U21A20427), and Project (No. WU2022A009) from the Center of Synthetic Biology and Integrated Bioengineering of Westlake University. F.W. and S.L. led the research. F.W. contributed technical ideas and developed the method and performed analyties. S.L. provided evaluation and suggestions. The authors thank Professor Siqi Sun and Tao You from Shanghai Artificial Intelligence Lab (SAIL) for their efforts in processing the OAS data, and Professor Buyong Ma from Shanghai Jiaotong University for his comments on improving the quality of the paper.
References
[1] Gary W Litman, Jonathan Rast, Michael Shamblott, Robert N Haire, Michele Hulst, William Roess, Ronda T Litman, Kristin R Hinds-Frey, Anna Zilch, and Chris T Amemiya. Phylogenetic diversification of immunoglobulin genes and the antibody repertoire. Molecular biology and evolution, 1007):60-72, 1993.
[2] Charles A Janeway, Paul Travers, Mark Walport, and Donald J Capra. Immunobiology. Taylor Francis Group UK: Garland Science, 2001.
[3] Matthew Raybould, Claire Marks, Konrad Krawezyk, Bruck Taddese, Jaroslaw Nowak, Alan P Lewis, Alexander Bujotzek, Jiye Shi, and Charlotte M Deane. Five computational
10

---

developability guidelines for therapeutic antibody profiling. Proceedings of the National Academy of Sciences, 116110):4025-4030, 2019.
[4] Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-based deep representation learning. Nature methods, 16(12)1135--132. 2019.
[5] Jung-Eun Shin, Adam J Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C Kruse, and Debora S Marks. Protein design and variant prediction using autoregressive generative models. Nature communications, 12(1):1-11, 2021.
[6] Sharon Fischman and Yanay Ofran. Computational design of antibodies. Current opinion in structural biology, 51:156-162, 2018.
[7] Tong Li, Robert J Pantazes, and Costas D Maranas. Optmaven-a new framework for the de novo design of antibody variable region models targeting specific antigen epitopes. PloS one, 9 10594, 2014.
[8] Gideon D Lapidoth. Dror Baran, Gabriele M Pszolla, Christoffer Norn, Assaf Alon, Michael D Tyka, and Sarel J Fleishman. Abdesign: A algorithm for combinatorial backbone design guided by natural conformations and sequences. Proteins: Structure, Function, and Bioinformatics, 83
(8):1385-140% 2015.
[9] Jared Adolf-Bryfogle Oleks Kalyuzhniy, Michael Kubitz, Brian D Weitzner, Xiaozhen Hu, Yumiko Adachi, William Schief, and Roland L. Dunbrack Jr. Rosettaantibodddesigg (rabd); A general framework for computational antibody design. PLoS computational biology. 14(4): c10061 12, 2018
[10] Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. Iterative re- finement graph neural network for antibody sequence-stue struccure co-design. arXiv preprint arXiv:2110.04624, 2021.
[11] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Antibody-antigen docking and design via hierarchical equivariant refinement. arXiv preprint vrXiv:2207.06616 2022.
[12] Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen- specific antibody design and optimization with diffusion-basee generative models. bioRxiv, 2022.
[13] Rahmad Akbar, Habib Bashour, Puneet Rawat, Philippe A Robert, Eva Smorodina, Tudor-Stefan Cotet, Karine Seem-Karlsen, Robert Frank, Brij Bhushan Mehta, Mai Ha Vu, et al. Progress and challenges for the machine learning-based design of fit-for-purpe -orpppse monoclonal antibodies. In Mabs, volume 14, page 2008790. Taylor & Francis, 2022.
[14] Xiangzhe Kong, Wenbing Huang, and Yang Liu. Conditional antibody design as 3d equivariant graph translation. arXiv preprint arXiv:2208. 06073, 2022.
[15] Chence Shi, Chuanrui Wang, Jiarui Lu, Bozitao Zhong, and Jian Tang. Protein sequence and structure co-design with equivariant translation. arXiv preprint arXiv:2210008761 2022.
[16] James Dunbar, Konrad Krawezyk, Jinwoo Leem, Terry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M Deane. Sabdab: the structural antibody database. Nucleic acids research, 42(DI):D1 140-D11444 2014.
[17] Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3d protein structures. arXiv preprint arXiv:2205.15675, 2022.
[18] Fang Wu, Huiling Qin, Wenhao Gao, Siyuan Li, Connor W Coley, Stan Z Li, Xianyuan Zhan, and Jinbo Xu. Instructbio: A large-scale sepi-supe learning paradigm for biochemical problems. arXiv preprint arXiv:2304.03900 2023.
[19] Fang Wu, Qiang Zhang, Dragomir Radev, Jiyu Cui, Wen Zhang, Huabin Xing, Ningyu Zhang, and Huajun Chen. 3d-transformer: Molecular representation with transformer in 3d space. 2021.
11

---

[20] Fang Wu, Dragomir Radev, and Stan Li. Molformer: Motif geneous molecular graphs. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 37, pages 5312-5320, 2023.
Motif-based based transformer on 3d hetero-
[21] Fang Wu, Shuting Jin, Yinghui Jiang, Xurui Jin, Bowen Tang, Zhangming Niu, Xiangrong Liu, Qiang Zhang, Xiangxiang Zeng. and Stan Z Li. Pre-training of equivariant graph matching networks with conformation flexibility for drug binding. Advanced Science, 9(33):2037796, 2022.
[22] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, (18(15):20166299)
2021.
[23] Sean R. Eddy. Profile hidden markov models. Bioinformatics (Oxford, England), 14(9):555-763 1998.
[24] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, al. Prottrans: towards cracking the language of life's s code through self supervised deep learning and high performance computing. arXiv preprint arXiv:2007.06225, 2020.
[25] Fang Wu, Lirong Wu, Dragomir Radev, Jinbo Xu, and Stan Li. Integration of pre-trained protein language models into geometric deep learning networks. Communications Biology, 6 (1):876, 2023.
[26] Fang Wu, Yu Tao, Dragomir Radev, and Jinbo Xu. When geometric deep learning meets pretrained protein language models. arXiv preprint arXiv:2212.03441 2022.
[27] Neha Chaudhary and Duane R Wesemann. Analyzing immunoglobulin repertoires. Frontiers in immunology, 9:462, 2018.
[28] Fang Wu, Nicolas Courty, Shuting Jin, and Stan Li. Improving molecular representation learning with metric learning-enhanced optimal transport. Patterns, 4(4), 2023.
[29] Jinwoo Leem, Laura Mitchell, James HR Farmery, Justin Barton, and Jacob Galson, Deciphering the language of antibodies using self-supervised leamning Patterns, page 100513, 2022.
[30] Tobias H Olsen, lain H Moal, and Charlotte Deane. Ablang: An antibody language model for completing antibody sequences. bioRxiv, 2022.
[31] Aleksandr Kovaltsuk, Jinwoo Leem, Sebastian Kelm, James Snowden, Charlotte M Deane, and Konrad Krawezyk. Observed antibody space: a resource for data mining next-generation sequencing of antibody repertoires. The Journal of Immunology, (01(5)2502 2018.
[32] Tobias H Olsen, Fergus Boyles, and Charlotte M Deane. Observed antibody space: A diverse database cleaned, annotated, and translated unpaired and paired antibody sequences. Protein Science, 31(1):141144 2022.
[33] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pages 9323-9332. PMLR, 2021.
[34] Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. arXiv preprint arXiv:2203.061 2022.
[35] Raphael Townshend. Rishi Bedi, Patricia Suriana, and Ron Dror. End-to-end learning on 3d protein structure for interface prediction. Advances in Neural Information Processing Systems, 32, 2019.
[36] Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya Shindyalov, and Philip Bourne. The protein data bank, Nucleic acids research, 28( (8(1):2355242 2000.
12

---

[37] Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pages 492-518. Springer, 1992.
[38] Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 5744))002777 2004.
[39] Doug Tischer, Sidney Lisanza, Jue Wang. Runze Dong, Ivan Anishchenko, Lukas F Milles, Sergey Ovchinnikov, and David Baker. Design of proteins presenting discontinuous functional sites using deep learning. Biorxiv, 2020.
[40] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.
[41] Andrew Leaver-Fay, Michael Tyka, Steven M Lewis, Oliver F Lange, James Thompson, Ron Jacak, Kristian W Kaufman, P Douglas Renfrew, Colin A Smith, Will Sheffler, al. Rosetta3: an object-oriented software suite for the simulation and design of macromolecules. In Methods in enzymology, volume 487, pages 545-574. Elsevier, 2011.
[42] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances in neural information processing systems, 32, 2019.
[43] Bowen Jing. Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learn- ing from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411, 2020.
[44] Jue Wang. Sidney Lisanza, David Juergens, Doug Tischer, Joseph L Watson, Karla M Castro, Robert Ragotte, Amijai Saragovi, Lukas Milles, Minkyung Baek, et al. Scaffolding protein functional sites using deep learning. Science, 377(6604):387-3944 2022.
[45] Danqing Wang, YE Fei, and Hao Zhou. On pre-training language model for antibody. In The Eleventh International Conference on Learning Representations, 2022.
[46] Wanessa C Lima, Elisabeth Gasteiger, Paolo Marcatili, Paula Duck, Amos Bairoch, and Pierre Cosson. The abed database: repository for chemically defined antibodies. Nucleic acids research, 48(D (8(DI))D261-D264 2020.
[47] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
[48] Brian L Hie, Kevin K. Yang, and Peter S Kim. Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins. Cell Systems, 1344):274-285, 2022.
[49] Lei Li, Shuang Chen, Zhichao Miao, Yang Liu, Xu Liu, Zhi-Xiong Xiao, and Yang Cao. Abrsa: robust tool for antibody numbering. Protein Science, 2854 2019. 2019.
[50] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[51] Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32, 2019.
[52] Liang He, Shizhuo Zhang, Lijun Wu, Huanhuan Xia, Fusong Ju, He Zhang, Siyuan Liu, Yingce Xia, Jianwei Zhu, Pan Deng, et al. Pre-training co-evolutionary protein representation via a pairwise masked language model. arXiv preprint arXiv:21 2021.
[53] Amy X Lu, Haoran Zhang, Marzych Ghassemi, and Alan Moses. Self-supervised contrastive learning of protein representations by mutual information maximization. BioRxiv, 2020.
[54] Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint axXiv:2004.03447,, 2020.
13

---

[55] Pascal Sturmfels, Jesse Vig, Ali Madani, and Nazneen Fatema Rajani. tion: An Aligmment-based pre-training task for protein sequence models. arXiv:2012.00195, 2020.
Profile predic- arXiv preprint
[56] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning, pages
PMLR, 2021.
[57] Surojit Biswas, Grigory Khimulya, Ethan C Alley, Kevin M Esvelt, and George M Church. Low-n protein engineering with data-efficient deep learning, Nature methods, 18(4):389-396, 2021.
[58] Tristan Bepler and Bonnie Berger. Learning the protein language: Evolution, structure, and function. Cell systems, 12(6):654-669 2021.
[59] Zichen Wang. Steven A Combs, Ryan Brand, Miguel Romero Calvo, Panpan Xu, George Price, Nataliya Golovach, Emmanuel O Salawu, Colby Wise, Sri Priya Ponnapalli, et Lm-gyp: A generalizable deep learning framework for protein property prediction from sequence and structure. bioRxiv, 2021.
[60] Jesse Vig, Ali Madani, Lav R Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. Bertology meets biology: interpreting attention in protein language models. arXiv preprint arXiv:2006. 15222, 2020.
[61] José Jiménez, Stefan Doerr, Gerard Martinez- Alexander S Rose, and Gianni De Fabritiis. Deepsite: rrinein-binding site predictor using 3d-convolutional neural networks. Bioinformatics. 33(19)23036-304, 2017.
[62] Jiyu Cui, Fang Wu, Wen Zhang, Lifeng Yang, Jianbo Hu, Yin Fang, Peng Ye, Qiang Zhang, Xian Suo, Yiming Mo, et al. Direct prediction of gas adsorption via spatial atom interaction learning. Nature Communications, 14(1):7043, 2023.
[63] Fang Wu and Stan Z Li. Diffmn: A geometric diffusion model for molecular dynamics simulations. In Proceedings the 2023.
the AAAI Conference on Artificial Intelligence, volume 37, pages
[64] Fang Wu, Siyuan Li, Lirong Wu, Stan Z Li, Dragomir Radev, and Qiang Zhang. Discovering the representation bottleneck of graph neural networks from multi-order interactions. arXiv preprint arXiv:2205.00 07266, 2022.
[65] UniProt Consortium. Uniprot: hub for protein information. Nucleic acids research, 43(D1): D202-D212, 2015.
[66] Robert D Finn, Alex Bateman, Jody Clements, Penelope Coggill, Ruth Y Eberhardt, Sean R Eddy, Andreas Heger, Kirstie Hetherington, Lüss Holm, Jaina Mistry, et al. Pfam: the protein families database. Nucleic acids research, (221)):222-D230 2014.
[67] Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self- supervised learning. arXiv preprint arXiv:2204.042 13, 2022.
[68] Yuzhi Guo, Jiaxiang Wu, Hehuan Ma, and Junzhou Huang. Self-supervised pre-training for protein embeddings using tertiary structures. 2022.
14

---

Appendix
A Experimental setting
Data Descriptions for Different Stages of HTP. (1) For the single-protein sequence level, we employ the state-of-the-ar ESM-2 [47], which outperforms all tested single-sequence protein lan- guage models across wide range of structure prediction tasks and enables prediction of the atomic resolution structure. It is trained on 86 billion amino acids across 250 million protein sequences that span evolutionary diversity. Specifically. ESM uses UniRef50, September 2021 version. The training dataset was partitioned by randomly selecting 0.5% (z 250,000) sequences to form the validation set. The training set has sequences removed via the procedure described in Hie et al. [48]. ESM-2 runs the MMseqs search to obtain the query and target databases. All train sequences that match a validation sequence with 50% sequence identity in this search are removed from the train set. The details of the ESM series can be found in htttps://g gi thub.
chaaarchhesse esm
(2) For the antibody sequence level, we use the Observed Antibody Space database (OAS) and its subsequent update [32] as the pretraining data. It currently contains more than one bil- lion sequences, from more than 80 different studies that cover various immune states, organ- isms, and individuals, which can be downloaded from its official website at https://opig stats ox. ac uk/ wwtp://wwwwwww//////ea//e We upload the processed paired data in https: //ap:.////nnnn/. baidu com/ Bttm:///81188119Mfaaama htwp://a.....aaa..oooaaaaaaaaaae in inhttps://pan baidu. com
(password: 1212) as well as the unpaired data (password: 96uf).
(3) For the Structures (DIPS) B5]. It is larger protein complex structure dataset than existing antibody-antigen complex structure datasets and is extracted from the Protein Data Bank [36]. We attain the database from Atom3d in Zendo httpss//zenodo org/record/4911100
protein complex structure level, we use the Database of Interacting Protein
which is a collection of both novel and existing benchmark datasets spanning several key classes of biomolecules. Referring to Atom3d, we split protein complexes by sequence identity at 30%, resulting in train/validattinnnesst sets with 87,303/31,05001555
instances.
(4) For the antibody-antigen complex structure level, we select all available antibody-antigen pro- tein complexes from SAbDab [66] at https://opig stats. ox. ac ww/webapps///////e/kk bataappss//esaadab. sabdab/ leading to dataset containing 9,823 structures. CDRs are identified using the antibody numbering program AbRSA (49. Following the setting in [12), the chosen data points are divided into training and test data based their release date and CDR sequence identity. To be explicit, the test split contains protein structures released after December 24, 2021, as well as structures with any CDR similar to those released after this date with sequence identity higher than 50%. The antibodies in the test set are further clustered with CDR sequence identity of 50% to climinate duplicates, resulting in 20 antibody-antigen structures. The training and validation splits just include complexes not involved during the curation of the test split. After that, we randomly divided the remaining complexes with a ratio of 90% and 10% into training and validation sets.
Dataset Sequence Similarity. In our splitting strategy, the test set of SAbDab includes sequence released after December 2021, as well structures with any CDR similar to those released after this date (with sequence identity higher than 50%). It is fairly possible that the pretraining datasets include antibody sequences similar to the test set. To navigate this issue, we have conducted comprehensive analysis of the sequence similarity between the various pretraining data sources and the test set in SAbDab. This analysis includes general protein sequences from general protein- protein complexes from DIPS and antibodies from OAS. We have plotted the sequence similarity distributions in Figure and present the statistical findings of this analysis in Table (
Table 4: The statistics of the sequence similarity between our splitted SAbDab test set and different pretraining datasets.
Dataset
Mean
Std.
Min.
25%
50%
75%
Max.
DIPS
0.188
0.036
0.000
0.183
0.429 0.243
0.198
0.208
OAS
0.246
0.017
0.200
0.235
0.254
0.401
15

---

OAS Database
DIPS Database
0.0
0.6
Probabiity Dendity
0.5
0.4
0.4
0.3
0.3
0.1
0.1
0.2
0.8
0.4
0.6
1.0
Sequence Similarity
Sequence Similarity
(a)
(b)
Figure 4: The distributional plot of sequence similarity between different pre-training resources and the test set in SAbDab.
The statistical findings unequivocally demonstrate that the highest sequence similarity values across these three datasets are consistently below 0.5. This empirical evidence serves as a strong basis for our resolute assertion that neither DIPS, nor OAS encompasses sequences that exhibit significant similarity to the SAbDab test set. With this robust evidence at hand, we maintain our firm confidence that the hierarchical training paradigm we have employed is free from any potential data leakage concerns.
Implementation Details. HPT is implemented in Py PyTorch and Py PyTorch Geometric packages, For all four training stages, we leverage an Adam optimizer [50] with a weight decay of le-5, All experiments are run multiple A100 GPUs, each with memory storage of 80G.
(1) For ESM-2 the single-protein sequence level training. we adopt a middle-size version, which has a parameter number of 150M, 30 layers, and hidden dimension of 640. Besides, we append the ESM-2 with a three-layer perceptron to forecast the residue type for MLM.
(2) For the antibody sequence level training, we use a batch size of 2 to avoid out-of-memory error and workers load the data. The number of epochs is 100 and starting learning rate is 1e-5. Apart from that, we utilize a ReduceLROnPllteen scheduler with a factor of 0.6, patience of 5 epochs, and a minimum learning rate of lc-7.
(3) For protein-proteie protein complex structure level training, we use a batch size of 32, 1000 epochs, and 4 works to speed up data loading. The starting learning rate is 1e-4, and a scheduler is utilized to automatically adjust the learning rate with factor of 0.6 and patience of epochs. We adopt a distance threshold of 8,0 determine the connection between different graph nodes (i.e., the alpha carbon of each residue). As for the loss weight balance, we set À 1.
(4) For the antibody-antigen complex structure level training, we also adopt the distance threshold of 8.0Ato build the graph connection. For the random initialization of the CDR coordinates, we use noise of e 0.1. As for the other important hyperparameters, we use a grid search mechanism to find the optimal combination. Notably, the geometric neural networks used in the third and fourth levels are matched to each other. If we alter the setting of GGNNs in the antibody- structure level training, we need retrain it in the protein- pootiin-protein complex structure level first. The entire hyperparameter search space is depicted in T
complex
Reproduction of Baselines. Concerning the implementation of several baseline methods, we use the official repositories for conditional RefineGNN (https://gi wttp:thtww.wcccoootttt.teme www/wwngong-aae RefineCNN/), HERN ( https: gi thub. com/ wtwm www.wwgngggnooeoo jitgggjjinnabooaaa http////c/ggggn...//ee/ DiffAb hhtpps //github. com htomm/loost2//difaaa pretrained PI.Ms, we download the code from https: //gi thubg com com/aleeeeeeooocoooeoccccc httpa/acccccccaaa/m//////a www.cheaaaaattteeeeeeeeeee anti for AntiBERTa and
To reproduce the performance of existing antibody-specific
com/ cww/oxpig/..///ccooccmoooocc AbLang for Abl Abl.ang. our comparison, we directly use their pretrained residue features the input for GGNNs without any fine-tuning.
16

---

Table 5: Hyperparameters setup for HTP.
Hyperparameters Search Space
Symbol
Value
Training Setup
Epochs
[100, 500, 1000]
Batch size
[32, 64, 128]
Learning rate
[le-4, 5e-5, le-6, 1e-7]
Warmup
[Yes, No]
Warmup epochs
[10, 20]
Loss Balance weight for Coordinates and Residue Types
[0.1. 0.3, 0.5, 0.7]
GNN Architecture
Dropout rate
[0.1, 0.2]
Number GNN layers
[2, 4, 6]
Tanh activation function
[Yes, No]
Coordinate Normalization
[Yes, No]
The hidden dimension of node representations
[320, 640]
The hidden dimension of edge representations
[16, 32, 64]
Code Availability. All relevant Python code to reproduce the results in our paper stored in the
GitHub repository at https: cww../ccooaeeeeeeeo.ooo.. lms724///
B
Additional Results
B.1
Ablation Study
We investigate the effectiveness and necessity of each component of HTP. As shown in Table the elimination of the level of protein-proter complex structure level level induces performance degradation, where RMSD increases from 2.06 2.49. Moreover, HTP by replacing features obtained by pretrained PLMs with learnable embedding features, whose performance is worse than HTP. To concise, AAR declines from 40.98 25.31, and RMSD increases from 2.06 to 2.65. In summary, our HTP brings significant relative improvements of 78.56% AAR, 41.97% in RMSD, and 2. 2.94% in TM-Score. This phenomenon strongly supports the superiority of our approach over existing naive co-design algorithms that are trained only on antibody-specific structure data.
implement variant of
Table 6: Effects of each module, where SPS stands for the single-protein sequence level, PPCS denotes the protein-protein complex structure level, and AS represents the antibody sequence level. The last row computes the relative improvements HTP over the primitive baseline without any protein data augmentation.
SAbDab (CDR-H3)
SPS
AS
PPCS
AAR (%) f
RMSD |
TM-Score
X
X
X
22.95 + 0.5
3,55 0.01
0.9146 + 0.003
2
/
X
X
33.87 + 0.8
2.77 0.04
0.9450 + 0.006
/
/
X
+
2.49
0.9538 + 0.004
25.31
0.7
2.95
0.9391 + 0.005
/
/
/
40.98 + 1.5
2.06 0.03
0.9621 + 0.005
Imp.
78.56%
41.97%
2.94%
C Related Work
Antibody Design. The majority old-school computational approaches for antibody design are based sampling algorithms over handcrafted and statistical energy functions to iteratively modify protein sequences and structures 9.. These physics-based algorithms are computationally expensive and prone to be stuck in local energy minimum, which triggers the adaptation of deep
17

---

learning in this but disregard the available antigen structures.
The initial researchers 4 5
use pure PLMs to generate protein sequences
To circumvent this, Jin et al. [0] introduce RefineGNN, the first co-design architecture that aims to neutralize SARS-CoV-2. Later, HERN [IT] is proposed more general version for paratope docking and design, which opens the door to produce antibodies given arbitrary antigen structures. Subsequent efforts are spent in either modifying the generative style or utilizing more advanced deep learning architectures such as diffusion denoise probabilistic models (DDPMs). For example, DiffAb [12] achieves niomie-resolution antibody design with -quiiivaai,, while MEAN [4 corrects the autoregressive manner with a full-shot one to prevent low efficiency and accumulated errors during inference.
Protein Sequence Modeling, Sequence-based protein representation learning is mainly inspired by the field of natural language processing. A large body of early work is focused on modeling individual protein families [51], solving problems such as functional nanobody design [5]. The success of this method then motivates the prospective trend to model large-scale databases of protein sequences by means of unsupervised learning. This line of study targets capturing the biochemical and co-evolutionary knowledge that underlies large-scale protein sequence corpus by self-supervised pertaining. Thanks to them, a number of pertaining objects have been explored
such as the next amino acid prediction [ (4), masked language modeling (MLM) [5T] 22), pairwise
MLM [52]. contrastive predictive coding (53), conditional generation [54), and position-specific
scoring matrix prediction (55). In addition, another line [56] [57] is based on multiple sequence alignment (MSA), leveraging sequences within a protein family to seize the conserved and variable regions of homologous sequences. Notably, some schemes for protein sequence modeling also seek to incorporate structural information in either the pretraining stage [58] or the finetuning stage [59]]
The improvements in model scale and architecture are also crucial to the recent achievement of PLMs. Explicitly, Rao et al. [51] evaluate various PLMs in panel of benchmarks and discover that multi-head attention outpaces the Potts model contact prediction, even if using single sequence
for inference. Concurrently, Vig et al. [60) observe that specific attention heads of pretrained Transformers have straight correlations with protein contact. Others [24] investigate a variety of Transformer variants and demonstrate that large Transformers can obtain state-of-the-at th-----rrt features in various tasks. Apart from that, the latest ESM-2 47] trains the largest PLM with 15B parameters and shows that models are scaled, they learn information enabling the protein structure prediction at the resolution of individual atoms.
Protein Structure Learning. With the rapid advance of geometrie deep learning, it has become increasingly attractive and challenging to represent and reason about the structures of macromolccules in 3D space. For the sake of encoding spatial information in protein structures including bond lengths and dihedral angles, numerous 3D geometric neural networks such as 3DCNN [61] or GNNs [4 43
62 63] have been invented. They excel at capturing complex interactions between sets of amino acids [64] and attain pivotal Euclidean geometry, egg., E(3) or SE(())-quivaariance and symmetry.
However, compared to protein sequences in databases like UniProt (65) or Pfam (6). the known structures in the PDB are scarce and hard to obtain. Therefore, it becomes urgent to develop structure-based mechanisms to efficiently learn protein representations with much fewer pretraining data. For instance, Hermosilla and Ropinski (17) use contrastive learning in terms of molecular substructures to help models understand protein structure similarity and functionality. Moreover, Chen et al. [67] propose self-supervised framework that predicts angles and inter-residue distances. Additionally, Guo et al. [68] present a coordinate denoising score matching method. Wu et al. (2] put forward novel prompt-based denoising conformation generative pretraining method based on the trajectories of molecular dynamics simulations. A recent attempt (34] makes a combination of both contrastive learning and self-prediction with more intriguing augmentation functions. Despite this progress, all of them are dealing with single-protein structures. No preceding studies have considered ttrseture-based pretraining in the circumstance of multiple proteins. That is, how pretrain on motein-proteci complex, or more specifically, the antibody-antigen complex, remains unexplored.
18


Reviews:
[{"review_id": "26wNTHpTMf", "reviewer": "Reviewer_KohD", "summary": "Antibody sequence-structure co-design and fix-backbone design is a very appealing task for both industry and academia especially in the context of drug design. The paper introduces a hierarchical training paradigm (HTP) as a potential solution of this problem. Moreover, the offered approach deal with a major issue of small dataset size for training. The experiments demonstrate effectiveness and contribution of HTP.", "questions": "1. Are you going to make HTP open source?", "limitations": "The paper adequately address limitations and future work.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Originality: The problem addressed in this paper is important and up to date. Structural biology in general and structural immunology in particular lacks big amount of data, therefore, it's important to find ways to train models on small datasets. Authors propose a novel way how to overcome this problem. The related work are cited and discussed. \nQuality: The submission is well written and organized. The claims are supported by appropriate experiments. The used methods and equations are explained and sufficient. Authors provide ablation study and discussion of related work.\nClarity: The text is well written and organized. The paper contains all the necessary citations. \nSignificance: The results are important and useful for the cases with limited training data (for example, structural biology). The submission provides a comparison to previous and related work and shows the impact and advantages of the current paper.", "weaknesses": "Please review more thoroughly currently available approaches for antibody design and fix-backbone protein design. For example:\nhttps://arxiv.org/abs/2110.04624\nhttps://arxiv.org/abs/2207.06616\nhttps://www.biorxiv.org/content/10.1101/2022.07.10.499510v5.abstract\nhttps://arxiv.org/abs/2302.00203\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your thoughtful and constructive review of our paper. We greatly appreciate your feedback and are pleased to know that you find the work strong in several aspects.\n\nFirstly, we are glad to hear that you acknowledge the originality and importance of the problem addressed in our paper. We agree that the field of structural biology, especially structural immunology, often suffers from limited data availability, making it challenging to train accurate models. Our proposed novel approach aims to overcome this issue, and we are delighted that you find it valuable. Furthermore, we are grateful for your comments on the quality of the submission. We put significant effort into ensuring that the claims made in the paper are well-supported by appropriate experiments. We are also pleased that you found the ablation study and discussion of related work to be satisfactory.\n\nRegarding the weaknesses you pointed out, we sincerely appreciate your suggestions for reviewing currently available approaches for antibody design and fix-backbone protein design. The provided links [A, B] are very relevant. Though we have already included them as baselines in Section 3 for comparison, there is still room for us to thoroughly assess these approaches to strengthen our paper's discussion sections. We believe that including these references will enhance the paper's completeness and provide a more comprehensive view of the state-of-the-art in the field.\n\n[A] Jin, Wengong, et al. \"Iterative refinement graph neural network for antibody sequence-structure co-design.\" arXiv preprint arXiv:2110.04624 (2021).\n\n[B] Luo, Shitong, et al. \"Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures.\" Advances in Neural Information Processing Systems 35 (2022): 9754-9767.\n\nAs for your question about making our method open source, we are pleased to confirm that we intend to release the code and data associated with our work upon acceptance. \n"}, {"review_id": "vstmofzjGi", "reviewer": "Reviewer_81S2", "summary": "This paper proposes a hierarchical training paradigm for antibody sequence-structure codesign. It incorporates different sources of data, including general protein sequences, antibody sequences, general protein-protein complexes, and antibody-antigen complexes. The motivation is that there are a lot more data on general proteins and pre-training the model on these non-antibody data may provide additional boost to model performance. Specifically, it uses the pre-trained ESM-2 language model to incorporate all the knowledge learned from general protein sequence data. It then fine-tunes it on all antibody sequences in Observed Antibody Space (OAS). The fine-tuned language model is used to calculate features for antibody and antigen sequences. Next, the model is trained on all protein-protein complexes in DIPS. Lastly, the model is fine-tuned on antibody-antigen complexes in SAbDab. The method shows substantial improvement on antibody design benchmarks compared to existing baselines.", "questions": "* Can you plot sequence similarity between all the different training data sources (general protein sequences in UniRef50, general protein-protein complexes in DIPS, antibodies in OAS)?", "limitations": "The authors has addressed the limitations and negative societal impact in the appendix.", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "* This paper proposes to incorporate multiple sources of biological data for model pre-training. If implemented properly, this is a important contribution to the field.\n* The evaluation is comprehensive. It includes all the recent baselines proposed in the field. The improvement over existing methods are substantial (but it can be due to potential data leakage, see discussion below).", "weaknesses": "* The main limitation of this paper is potential data leakage. ESM-2 is trained on all protein sequences in the UniRef database (September 2021 version). The test set includes sequence released after December 2021, as well as structures with any CDR similar to those released after this date (with sequence identity higher than 50%). Therefore, it is fairly possible that the training set of ESM-2 includes antibody sequences similar to the test set. Likewise, OAS may also contain antibody sequences similar to the test set and the author did not perform any filtering to ensure this may not happen. Similarly, DIPS also contains antibody-antigen structures. Even though it only contains structures released before 2019, it may still contain similar antibody sequences or antigen sequences to the test set. Can the authors provide evidence that there is no potential data leakage?\n* The model architecture is an adaptation of existing models (e.g., EGNN). The technical innovation of the model architecture is relatively weak (though this is not the focus of the paper).\n* The antibody-antigen complex test set contains only 21 structures, which is too small for evaluation. Expansion of the test set is necessary (at least over 100 structures is needed).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and comments on our paper. We appreciate your acknowledgment of the potential importance of incorporating multiple sources of biological data for model pre-training. We have carefully considered your concerns and would like to address them as follows:\n\n* Data Leakage Or Not\n\n(1) We acknowledge the concern you raised regarding the potential high sequence similarity between the different datasets used in our proposed approach. In response, we have conducted a comprehensive analysis of the sequence similarity between the various pretraining data sources and the test set in SAbDab. This analysis includes general protein sequences from UniRef50, general protein-protein complexes from DIPS, and antibodies from OAS. We have plotted the sequence similarity distributions and have attached the corresponding figures in the uploaded PDF in the general author rebuttal. Below, we present the statistical findings from this analysis:\n\n|   Dataset           | Mean              | Std               | Min.              | 25%               | 50%               | 75% | Max. |\n|-----------------|-------------------|-------------------|-------------------|-------------------|-------------------|-----|------|\n| UniRef50 |  0.051 | 0.021 | 0.002 | 0.041 |0.052 | 0.059 | 0.260   |\n| DIPS | 0.188 | 0.036 | 0.000 | 0.183 | 0.198 | 0.208 | 0.429 |\n| OAS | 0.246 | 0.017 | 0.200 | 0.235 | 0.243 | 0.254 | 0.401   |\n\nFrom the statistical results, it is evident that the maximum sequence similarity values for these three datasets are all below 0.5. Based on this evidence, we can confidently conclude that neither UniRef50, DIPS, nor OAS contains sequences that are substantially similar to the SAbDab test set. Consequently, we firmly believe that there is no potential data leakage in our hierarchical training paradigm.\n\n\n(2) Furthermore, we would like to address the broader question of whether it is acceptable to utilize additional publicly available unlabeled data for model pretraining, even if there is some distributional overlap with the downstream data distribution. Our viewpoint aligns with common practices in various domains of AI, including computer vision, natural language processing, and AI for scientific research. It is indeed a reasonable practice to leverage self-supervised learning on independently collected databases to enable deep learning models to capture a broader feature space. This broader exposure often leads to enhanced generalization capabilities, especially when dealing with out-of-distribution samples, such as proteins from different families or clusters [A].\n\nIt is noteworthy that several prior studies in computational biology and bioinformatics have successfully employed pretrained language models like ESM or ProtTrans to enhance model performance. These studies did not intentionally exclude samples belonging to distributions closely related to the test set. For instance, DiffDock [B] utilizes ESM-2 to derive residue embeddings for receptors, and RDE [C] pretrains a Graph Transformer on PDB-REDO for side-chain angle recovery, which is then applied to predicting mutation effects (ddG) on Skempi. CLEAN [D], featured in Science, initializes residue features with ESM-2 to achieve superior accuracy, reliability, and sensitivity in assigning Enzyme Commission (EC) numbers compared to existing tools.\n\nBeyond language models, even structurally pretrained algorithms often assess their effectiveness without strictly filtering out overly similar structures. For example, GearNet [E] employs pretraining on the AlphaFold protein structure database (805K structures) and evaluates its model across diverse downstream tasks like Enzyme Commission (EC) number prediction, Gene Ontology (GO) term prediction, Fold classification, and Reaction classification. Similarly, PromptProtein [F] pretrains on UniRef50, PDB (200K structures), and STRING datasets, demonstrating efficacy in tasks involving Gene Ontology and Enzyme Commission numbers.\n\nAll the aforementioned examples strongly support our assertion that pretraining on publicly available sequential or structural databases can indeed empower deep learning models effectively. Crucially, this empowerment is feasible as long as the approach avoids any use of downstream test data and corresponding labels. In our HTP approach, datasets like OAS, UniRef, and DIPS are independently collected and do not reference any data points in SAbDab. This ensures that regardless of the specific downstream problem or test data employed, our pretraining steps in HTP can consistently transfer prior knowledge to various real-world problems.\n\nThank you for your thoughtful considerations and queries. We are confident that our approach adheres to rigorous standards of integrity and robustness, and we appreciate the opportunity to clarify these points.\n\n[A] Erhan, Dumitru, et al. \"Why does unsupervised pre-training help deep learning?.\" Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010.\n\n[B] Corso, Gabriele, et al. \"Diffdock: Diffusion steps, twists, and turns for molecular docking.\" ICLR 2023.\n\n[C] Luo, Shitong, et al. \"Rotamer Density Estimator is an Unsupervised Learner of the Effect of Mutations on Protein-Protein Interaction.\" ICLR 2023.\n\n[D] Yu, Tianhao, et al. \"Enzyme function prediction using contrastive learning.\" Science 379.6639 (2023): 1358-1363.\n\n[E] Zhang, Zuobai, et al. \"Protein representation learning by geometric structure pretraining.\" ICLR 2023.\n\n[F] Wang, Zeyuan, et al. \"Multi-level Protein Structure Pre-training via Prompt Learning.\" The Eleventh International Conference on Learning Representations. 2022.\n"}, {"review_id": "nwRFvzy92a", "reviewer": "Reviewer_H5io", "summary": "In this study, the author points out that the efficacy of existing co-design methods is predominantly limited by the small number of antibody structures. They propose a hierarchical training paradigm (HTP), a novel unified prototype to exploit multiple biological data resources and aim at fully releasing the potential of geometric graph neural networks (GGNNs) for the sequence and structure co-design problem.", "questions": "- In line 590, Appendix, is weight decay 1e-5?\n- Missing References: Wang et al, On Pre-training Language Model for Antibody, ICLR 2023\n- How do you split the epitope from the antigen? Is there any threshold?\n- In this work, you build a graph with a cutoff of 8A, and initialize the antibody with the center of the residues before/after the CDR. Is it possible that the initialized antibody is far from the antigen, i.e., larger than 8A, such that the antibody can not interact with the antigen?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- This study is well-motivated, and sheds light on integrating big data (including sequence data, protein complex data) into antibody design.\n- The writing is clear and the pipeline is easy to follow.", "weaknesses": "- Missing comparison with the SOTA method, dyMEAN, whose AAR of CDR-H3 is 43.65% (Table 4). (End-to-End Full-Atom Antibody Design, ICML 2023)\n- Some results are inconsistent with the reference. In the original paper of MEAN, the AAR of CDR-H3 is reported to be 36.38% or 39.87% under different evaluation settings (Table 1, MEAN), while in this manuscript, MEAN's AAR is 22.56% (Table 1).\n- Some results are confusing. The performance on fix-backbone design (Table 2) is lower than that on co-design (Table 1). For example, the AARs of CDR-L1/L2/L3 on co-design are 91.13%/89.80%/73.82%, while the performance on fix-backbone design is only 77.49%/74.15%/76.96%. The AARs of CDR-H3 are both 40.98% on two tasks.  Fixed-backbone design should be much easier than the co-design task. Can you give an explanation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review of our study. We appreciate your positive feedback on the motivation of our work and the clarity of the writing, as well as your constructive comments that will help us improve the quality and accuracy of the paper. We have carefully considered each of the points you raised and would like to address them accordingly:\n\n* Comparison with dyMEAN    \n\ndyMEAN [A], indeed, is an extraordinary work for antibody sequence-structure co-design and extends the framework of MEAN [B]. However, the problem setting of dyMEAN [A] is completely different from most existing co-design architectures (i.e., MEAN [B], RefineGNN [C], HERN [D], DiffAb [E], and ours). To be explicit, dyMEAN [A] assumes that the 1D incomplete antibody sequences (without CDRs) are given but no antibody structures are provided. To resolve this dilemma, dyMEAN proposes an end-to-end pipeline to replace the previous multi-stage schema: IgFold for structure prediction + HDock for docking on the target epitope + MEAN [B] for binding CDR generation. In contrast, we follow the conventional setting of RefineGNN [C], HERN [D], DiffAb [E], and MEAN [B], and assume that the incomplete antibody structures (without CDRs) are offered. Therefore, it is hard to directly compare the effectiveness of those co-deign approaches with dyMEAN [A]. In Table 1 of the dyMEAN paper [A], the reported results for DiffAb [E], MEAN [B], and HERN [D] all adopt the above-mentioned pipeline (IgFold -> HDock -> CDR generation -> Rosetta) rather than only employing those algorithms. Thus, we are unable to fully compare dyMEAN in our experimental section.  \n\n\n* Inconsistent Results of MEAN\n\nThank you for bringing to our attention the discrepancy in the reported baseline results compared to the original papers. We acknowledge the importance of this observation and would like to clarify the differences in the dataset-splitting mechanism used in our work compared to previous studies. In our study, we followed the dataset-splitting approach of DiffAb [E], which uses a completely different methodology compared to MEAN [B]. Specifically, we utilized the SAbDab dataset and split the data points into training and test sets based on their release date and CDR sequence identity. The test set includes protein structures released after December 24, 2021 and structures with any CDR similar to those released after that date (sequence identity higher than 50%). To eliminate duplicates, we further clustered antibodies in the test set with 50% CDR sequence identity, resulting in a final set of 20 antibody-antigen structures. On the other hand, the training split contains complexes that were not involved in the curation of the test split. In contrast, MEAN [B] employs the split setting of RefineGNN [C], which divides the entire dataset into training, validation, and test sets based on the clustering of CDRs to maintain generalization. Subsequently, all clusters are divided into training, validation, and test sets in an 8:1:1 ratio. \n\nDue to the different data-splitting mechanisms, our approach in DiffAb [E] poses a greater challenge than that of MEAN [B] and RefineGNN [C]. This challenging dataset split is likely responsible for the observed discrepancies in the reported results. For instance, RefineGNN [C] achieves an AAR of 39.40%, 37.06%, and 18.88% for CDR-H1, CDR-H2, and CDR-H3 in the MEAN paper [B]. However, in the DiffAb paper [E], the AAR for CDR-H1, CDR-H2, and CDR-H3 drops to 27.77%, 27.04%, and 8.00%, respectively. This difference indicates that the same algorithm may experience a decline in performance when facing a more challenging data-splitting mechanism. In light of this fact, we believe that our reproduction of several important baselines is well-implemented, and the comparison in our manuscript is fair, considering the more difficult dataset splitting used in our work. However, we are grateful for your concerns, and we will add a few sentences to our paper to clarify the influence of dataset splitting on the model performance.\n\n* Performance Discrepancy in Fix-Backbone Design\n\nWe appreciate your astute observation regarding the contrast in performance between the co-design and fix-backbone design tasks. Indeed, we concur with your insight that fix-backbone design should be relatively less complex since it doesn't entail predicting the precise positions of CDRs.\n\nIn our initial implementation for CDR-L1/L2/L3, we admit that we did not strictly follow a grid search of the entire hyperparameter space. Instead, we opted for early stopping during training for convenience once we observed an acceptable result that outperformed all existing baselines. We acknowledge that this approach might not have fully explored the optimal combination of different hyperparameters. In response to your valuable feedback, we have re-run the experiments and conducted a thorough grid search to find the best combination of hyperparameters for CDR-L1/L2/L3 in fix-backbone design. The updated results are listed below, and as you can see, the overall performance of CDR-L1/L2/L3 in fix-backbone design is indeed better than that of the sequence-structure co-design task:\n\n| Region | CDR-L1 |  | CDR-L2 |  | CDR-L3    |    |   |\n|--|---|-----|---|----|---|--|---|\n|  Metric   | AAR               | Perplexity        | AAR               | Perplexity        | AAR               | Perplexity        |   |\n| HTP | 93.62$\\pm$1.5 | 1.09$\\pm$0.08 | 91.46$\\pm$1.7 | 1.58$\\pm$0.10 | 80.71$\\pm$1.1 | 2.66$\\pm$0.10 |   |\n\nAdditionally, we extend our gratitude for identifying a typographical error in Table 2. To rectify this, we'd like to clarify that the AAR for CDR-H3 in the fix-backbone design task stands at 43.25. We sincerely appreciate your attention to detail and your insistence on rigor in our experimental setup. Your feedback has led us to reevaluate our approach and perform a comprehensive grid search, which has yielded more reliable and accurate results."}, {"review_id": "LOhLNJWI8Q", "reviewer": "Reviewer_eNMD", "summary": "This paper introduces a novel approach called the hierarchical training paradigm (HTP) to address the antibody co-design problem. It leverages both geometric neural networks and large-scale protein language models and proposes four levels of training stages to efficiently exploit the evolutionary information encoded in the abundant protein sequences and complex binding structures.", "questions": "Please see the weakness.", "limitations": "The authors have adequately discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The authors explored using multi-modal data from different domains to enhance the antibody sequence-structure co-design performance, which is a novel perspective.\n3. Extensive experiments show that HTP significantly overperforms previous methods.", "weaknesses": "1.  The code is not provided at the current stage.\n2.  The reported baseline results are inconsistent with the original papers. For example, in table. 1, the CDR-H3 AAR of MEAN is only 22.56%, which is much lower than the 36.38% in the original paper. Therefore, the comparison may be unfair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for your valuable feedback and constructive comments on our paper. We appreciate your recognition of the strengths of our work and acknowledge the weaknesses pointed out. We have carefully reviewed your feedback and would like to address each concern:\n\n1. Code Availability: We apologize for not providing the code at the current stage. We fully understand the importance of code reproducibility for advancing research in the field and will release the code and associated resources upon acceptance.\n\n2. Inconsistent Baseline Results: We appreciate your observation regarding the inconsistency of the reported baseline results with the original papers. It is worth noting that we follow DiffAb [A] and adopt a completely different dataset split from MEAN [B]. To be specific, the selected data points in SAbDab are divided into training and test data based on their release date and CDR sequence identity. The test split includes protein structures released after December 24, 2021, and structures with any CDR similar to those released after the date (sequence identity higher than 50%). Antibodies in the test set are further clustered with 50% CDR sequence identity to remove duplicates, finally resulting in 20 antibody-antigen structures. The training split contains complexes not involved during the curation of the test split. Meanwhile, MEAN [B] uses the split setting of RefineGNN [C], which separates the entire dataset into training, validation, and test sets according to the clustering of CDRs to maintain the generalization test. Then they divide all clusters into training, validation, and test sets with a ratio of 8:1:1.      \nAs a consequence, the split setting of DiffAb [A] poses a greater challenge than that of MEAN [B] and RefineGNN [C]. This statement can be verified via the discrepancy in their reported results. For instance, RefineGNN [C] achieves an AAR of 39.40%, 37.06%, and 18.88% for CDR-H1, CDR-H2, and CDR-H3 in the MEAN paper [B], but only reaches an AAR of 27.77%, 27.04%, and 8.00% for CDR-H1, CDR-H2, and CDR-H3 in the DiffAb paper [A]. This phenomenon indicates that the same algorithm can encounter a decline in its performance when facing a more difficult data-splitting mechanism. Based on this fact, we believe our reproduction of several important baselines is implemented well and the comparison in our manuscript is fair. However, we are still thankful for your concerns and will add a few sentences to clarify the influence of dataset splitting over the model performance.\n\n\n[A] Luo, Shitong, et al. \"Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures.\" Advances in Neural Information Processing Systems 35 (2022): 9754-9767.\n\n[B] Kong, Xiangzhe, Wenbing Huang, and Yang Liu. \"Conditional antibody design as 3d equivariant graph translation.\" arXiv preprint arXiv:2208.06073 (2022).\n\n[C] Jin, Wengong, et al. \"Iterative refinement graph neural network for antibody sequence-structure co-design.\" arXiv preprint arXiv:2110.04624 (2021)."}]

Author rebuttal:
We extend our sincere gratitude to all four reviewers for your insightful and constructive feedback on our proposed hierarchical training paradigm (HTP) for antibody sequence-structure co-design and fix-backbone design. We are encouraged by your positive reception of our work and your recognition of its relevance to both industry and academia, especially in the context of drug design. Your commendation of the soundness, presentation, and contribution of our paper is invaluable to us. 

--------------------

* Potential Data Leakage

As recommended by Reviewer 81S2, we have undertaken an exhaustive examination of the sequence similarity existing between the diverse pretraining data sources and the test set present in SAbDab. This comprehensive analysis encompasses general protein sequences sourced from UniRef50, general protein-protein complexes extracted from DIPS, and antibodies extracted from OAS. The outcomes of this analysis have been visually represented through sequence similarity distributions, and we have included the relevant figures within the uploaded PDF. 

The statistical findings unequivocally demonstrate that the highest sequence similarity values across these three datasets are consistently below 0.5. This empirical evidence serves as a strong basis for our resolute assertion that neither UniRef50, DIPS, nor OAS encompasses sequences that exhibit significant similarity to the SAbDab test set. With this robust evidence at hand, we maintain our firm confidence that the hierarchical training paradigm we have employed is free from any potential data leakage concerns.

* Inconsistent Results

We value the insightful observation of Reviewers eNMD and H5io regarding the disparity between the reported baseline outcomes and the original papers. It's crucial to recognize that we have adhered to the methodology outlined in DiffAb [A], which entails a distinct dataset split approach compared to MEAN [B]. Specifically, our division of data points within SAbDab for training and testing is rooted in release dates and CDR sequence identity. The test partition encompasses protein structures released after December 24, 2021, along with structures that bear any CDR similarity to post-date releases (with a sequence identity surpassing 50%). Antibodies within the test set are further grouped using 50% CDR sequence identity to eliminate duplicates, ultimately culminating in 20 antibody-antigen complexes. Meanwhile, MEAN [B] employs the RefineGNN [C] split configuration, involving segregation of the entire dataset into training, validation, and test subsets, organized as per CDR clustering to ensure generalization. Each cluster is then distributed into training, validation, and test sections, with an 8:1:1 ratio.

This distinctive approach in DiffAb [A] engenders a greater challenge than that of MEAN [B] and RefineGNN [C], a fact substantiated by the variance in their reported results. For example, RefineGNN [C] attains an AAR of 39.40%, 37.06%, and 18.88% for CDR-H1, CDR-H2, and CDR-H3 in the MEAN paper [B], but registers an AAR of 27.77%, 27.04%, and 8.00% for the respective CDRs in the DiffAb paper [A]. This discrepancy underscores that the identical algorithm can witness diminished performance when confronted with a more demanding dataset-splitting framework. Armed with this empirical insight, we are confident in the meticulous replication of crucial baselines in our work, ensuring fairness in our comparisons. Although we stand by the fairness of our approach, we value your concerns and will augment our manuscript with clarifications elucidating the impact of dataset splitting on model performance.

 [A] Luo, Shitong, et al. "Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures." Advances in Neural Information Processing Systems 35 (2022): 9754-9767.

[B] Kong, Xiangzhe, Wenbing Huang, and Yang Liu. "Conditional antibody design as 3d equivariant graph translation." arXiv preprint arXiv:2208.06073 (2022).

[C] Jin, Wengong, et al. "Iterative refinement graph neural network for antibody sequence-structure co-design." arXiv preprint arXiv:2110.04624 (2021).

* Code Availability 

We deeply appreciate the significance of ensuring code reproducibility to facilitate the progress of research within the field. Rest assured, we are committed to releasing the code and its associated resources upon the acceptance of our work.

-----------
We genuinely appreciate the time and effort you've invested in reviewing our work. Your feedback will undoubtedly contribute to the enhancement of our research. We are committed to addressing all your concerns and look forward to sharing our revised manuscript with you.

Meta-review: The paper focuses on the antibody design problem and enhances prior work by proposing to hierarchically pre-train the neural networks using general protein sequence data, general PPIs, and general antibody sequences (OAS). 

Overall, reviewers agree that the proposed approach is sensical and novel. Further, the achieved numerical results are strong. 

The main concern with this paper was potential data leakage. In particular, the evaluation of design methods that utilize protein language models (in this case ESM-2) should be stringent and ensure that the test data were not previously seen by the pre-trained models. Despite the concerns, a careful analysis done by the authors during the rebuttal period provided some evidence that such a leakage did not occur. Though the purported analysis cannot be deemed conclusive, it does corroborate the authors' claims that the experimental benefits brought forth by the approach are not due to data leakage. As such, it was decided to give the paper the benefit of the doubt.
Decision: Accept (poster)