Decoupled Sequence and Structure Generation for Realistic Antibody Design
Authors: Nayoung Kim
Minsu Kim
Sungsoo Ahn
Jinkyoo Park
Venue: Accepted by TMLR ()

Abstract: Recently, deep learning has made rapid progress in antibody design, which plays a key role in the advancement of therapeutics. A dominant paradigm is to train a model to jointly generate the antibody sequence and the structure as a candidate. However, the joint generation requires the model to generate both the discrete amino acid categories and the continuous 3D coordinates; this limits the space of possible architectures and may lead to suboptimal performance. In response, we propose an antibody sequence-structure decoupling (ASSD) framework, which separates sequence generation and structure prediction. Although our approach is simple, our idea allows the use of powerful neural architectures and demonstrates notable performance improvements. We also find that the widely used non-autoregressive generators promote sequences with overly repeating tokens. Such sequences are both out-of-distribution and prone to undesirable developability properties that can trigger harmful immune responses in patients. To resolve this, we introduce a composition-based objective that allows an efficient trade-off between high performance and low token repetition. ASSD shows improved performance in various antibody design experiments, while the composition-based objective successfully mitigates token repetition of non-autoregressive models.

Sungsoo Ahn Published in Transactions on Machine Learning Research (01/2025)
Decoupled Sequence and Structure Generation for Realistic Antibody Design
Nayoung Kim
neyoungkim@khiist.ocok
Korea Advanced Institute of Science and Technology (KAIST)
Minsu Kim Korea Advanced Institute of Science and Technology (KAIST)
min--sukkaist..c....
mangsoo.ahn@pposteek.e
Pohang University of Science and Technology (POSTECH)
Jinkyoo Park
balkyyoo.prrkkkais..coo oc.
Korea Advanced Institute of Science and Technology (KAIST)
Reviewed on OpenReview: https: // operreveew.ew net/ forum? id- CTRABQunkm
Abstract
Recently, deep learning has made rapid progress in antibody design, which plays a key role the advancement of therapeutics. dominant paradigm is to train a model to jointly gen- erate the antibody sequence and the structure as a candidate. However, the joint generation requires the model to generate both the discrete amino acid categorics and the continuous 3D coordinates; this limits the space of possible architectures and may lead to suboptimal performance. In response, we propose an antibody sequence-structure decoupling (ASSD) framework, which separates sequence generation and structure prediction. Although our ap- proach is simple, our idea allows the use powerful neural architectures and demonstrates notable performance improvements. We also find that the widely used non-auloregreesive generators promote sequences with overly repeating tokens. Such sequences are both out- of-distribution and prone to undesirable developability propertics that trigger harmful immune responses in patients. To resolve this, introduce composition- based objective that allows an efficient trade-off between high performance and low token repetition. ASSD shows improved performance in various antibody design experiments, while the composition- based objective successfully mitigates token repetition of non-autoregressive models.
1 Introduction
Antibodies are Y-shaped proteins that detect and neutralize the disease-causing agents. Due to high speci- ficity and binding affinity, they are recognized as one the most promising drug modalities (Mullard 2022) Historically, antibody development relied on experimental and physics-based computational approaches, which are laborious and timo-consuming (Kim et al.] 2023). In response, there growing trend to inte- grate deep learning- approaches for antibody development.
Early deep generative models (Liu et al. 2020 Saka et al. 2021 Akbar et al. 2022 focused only designing
the antibody sequences, neglecting the importance of antibody structure in functionality (Martinkus et al.
2024). To address this, Jin el al. (2021) proposed the sequence-struclure co-design to generate both antibody sequences and structures. Many works made notable progress, inchuding Jin et al. (2021) Verma et al. (2023) Kong et al. (2022): Luo et al. (2022 Wu & Li (2024). Such models are based on the joint prediction of sequence and structure with common architecture for both tasks. However, this restricts task-specific optimization of model architectures, palentially hindering higher sequence-structure modeling performance.

---

Published in Transactions on Machine Learning Research (01/2025)
Additionally, recent works (Melnyk al. 2023 Kong et al. 2022 Verma et al. 2023 Kong et al. 2023
show that sequence generation with non-autoregressive model ac achieves faster inference speed and better performance than the autoregressive counterparts, with its efficacy even extending to protein design Gao ct al. 2022 Zhong et al.) 2023 However, we find that such models generate sequences that are dispro- portionately dominated by frequently occurring amino acid lype (e.g., the natural sequence ARMGSDYDVWFDY versus non-auutoregressive prediction TRYTYYMYY'DY) Such sequences are out-of-distribution and prone to undesirable developability properties that can trigger harmful immune responses in patients (Jin et al. 2021 Wang et al. 2009).
Contribution. In this paper, we propose antibody sequenco-structure docoupling (ASSD), general frame- work for constructing a highly performant antibody sequence-structure co-design model. Our idea is to leverage Aufinsen's dogma, which states that the native conformation of a protein is determined by amino acid soquence (Anfinsen 1973)- To this end, instead of joint prediction, we decouple the sequence-structure design into two stops: a sequence design step followed by a structure prediction step. This docoupling ap- proach is desirable since it allows the suitable choice of model architectures for each task and enables the use of large-scale sequence databases (without structure information) during the sequence design step. Despite the simplicity, such decoupling of sequence-structure co-design has been overlooked in previous works.
We also propose a composition-based objective for antibody sequence generation that resolves the token repetition problem of non-autoreeressive models. Specifically, our key idea is to augment the MLE objective with the maximization of similarity between amino acid compositions of generated and target sequences. To this end, we use REINFORCE trick (Williams) 1992 Fu et al.) 2015) with respect to the composition vector which accounts for the rate of amino acids appearing in the sequences. Our idea is to let the model implicitly learn the dependency between the residues within sequence, effectively preventing excessive repetition of any single amino acid.
Our experiments demonstrate that our sequence-structure decoupling approach improves performance in various antibody design experiments, while our training algorithm effectively prevents excessive token rep- etitions. Notably, our approach establishes a Pareto frontier over other non-auloregressive antibody design models, indicating optimal trade-offs between high sequence modeling capacity and low token repetition. Additionally, we demonstrate that our training algorithm can be generalized to protein design.
2 Related work
Antibody design. Many pioneering works on antibody design predicts only 1D sequences using CNN and
LSTM [Liu et al. 2020 Saka et al. 2021 Akbar et al.] 2022). Recently, Melnyk et al. (2023) improved 1D sequence modeling performance by repurposing a pro-trained Transformer-based English language model for sequence infilling Frey et al. (2023) also demonstrated the potential of sneeryyscooreee based models for antibody sequence design with a walk-jump sampling scheme.
Instead of designing only 1D sequences, Jin et al. (2021) proposes soquenco-strucctue co-design, which involves both 1D sequence and 3D structure infilling of CDRs. Specifically, Jin et al. (2021) represents the antibody- antigen complex with an E(3) ii3)-invariant graph and predicts the sequence in an autoregressive fashion. On the contrary, Kong et al. (2022) constructs an [(3)-eqquvarriant graph to fully capture the 3D geometry and predicts sequence by a full-shot decoding scheme to speed up inference. Verma et al. (2023 formulates a coupled neural ODE system over the antibody nodes and further increases inference speed with single round of full-shot decoding, Wu & Li (2024) introduces a four-level training strategy with various protein /antibody
databases. Unlike deterministic GNN-basod methods, Luo et al. (2022) and Martinkus et al. 2024) adopt
diffusion probabilistic models to generate structures stochastically. Notably, Frey et al. (0223) Martinkus
ct al. 2024) supports their approach with laboratory experiments.
Previous approaches predict the sequence and structure jointly, except Martinkus et al. 2024) which intro- duces a srructure-oo-sqquence decoupling strategy. However, this approach constrains itself to the Structural Antibody Database (SAbDab) (Dunbar et al. 2014), which orders of magnitude smaller than sequence
In this paper, we limit focus one- or few-shot non-auttoregressive models.
2

---

Published in Transactions on Machine Learning Research (01/2025)
VL
Cnrl
C-tl
CL
VL
CDR-L1 CDR-12 CDR-L3
CHo Ck2
Figure 1: Schematic structure of an antibody. An antibody consists of a pair of heavy and light chains, each containing a variable region and constant regions. The variable region consists of three complementarity- determining regions (CDRs) and its complement. called the framework regions. We aim to design heavy chain CDRs, which contribute the most to antibody-antiger interaction.
databases like the Observed Antibody Space (OAS) OOsen et al. 2022). In this work, we demonstrate the superiority requence-to-structin decoupling over joint prediction strategy.
Protein design. Due their relovance, methods in protein design can offer valuable insights into antibody design. Ingraham et al. (2019) represents the protein structure with k-nearest neighbor graph and applies Transformer-basee encoder-decoder model to predict the sequences autoregressively. To fully capture the complex geometry within the graphs. Jing et al. (2020) introduces geometric vector perceptrons that can replace MLPs in GNNs. Tan et al. (2023) introduces an additional global module that captures the global context of the protein structure. To enable an application to a wide range protein design problems, Dauparas et al. (2022) adopts an order-agnostic autoregressive model with random decoding orders. Instead, Gao et al. (2022) adopts onc-shot decoding scheme and drastically improves the inference speed. Recently, Zheng et al. (2023) showed stale-of-the-al performance by incorporating lightweight structural adapter
into protein language model.
Notably, Melnyk et al. )023); Kong et al. 2022 Verma et al. (002); Gao et al. 2022 :: Zheng et al. (2023)
adopt non-autoregrresive factorization trained with MLE objective or its slight variations. In this paper, we showcase that this approach promotes highly repetitive sequences and propose a novel training strategy to mitigate this problem while maintaining a comparable level of performance.
3 Preliminaries and notations
Preliminaries. As shown in Figure ! an antibody consists of pair of light chains and heavy chains. Both the light chain and heavy chain composed of variable region (VL:VH) and constant region(s) (C1, CH,, Cira, Cha). Each variable region could be further decomposed into three complernentarity- determining regions (CDR-L1, CDR-L2, CDR-L3; CDR-HI, CDR-H2, CDR-H3) and framework region,
which is the complement of CDRs. Following previous works (Jin et al. 2021 Kong et al. 2022 Verma
et al. 2023 Luo et al. 2022 Melnyk et al., 2023), we narrow the scope of our task to sequence-structur co-design of heavy chain CDRs as they contribute the most to antibody-antigen interaction.
Notations. We denote the amino acid vocabulary set with A. Let L be the number of residues in CDR. We denote the ground-truth sequence and structure of heavy chain CDR (#1...., and the predicted sequence and structure & and &. We denote all conditional information (e.g., framework region, antigen, initializations) as C.
(s1...., and x
4 Methods
Here, we provide an overview of this section. Our goal is to train a non-antoregreesive generator that can generate the sequence of amino acids and the associated 3D information conditioned on some semantic information. Motivated by Anfinsen's dogma, which states that the protein structure is determined by its amino acid sequence (Anfinsen) 1973), propose antibody sequence-structure decoupling (ASSD)
3

---

Published in Transactions on Machine Learning Research (01/2025)
1. Sequence design
2. Structure prediction
Sequence design model Po(s|c)
Sequence-to-strucie
0 2
Initialize
Initialize
model
P& Px(x(S,c)
3,c
Figure 2: Overview of antibody sequence-structtre dccoupling (ASSD) framework. ASSD first designs CDR sequences with sequence design model and then predicts the corresponding structure with eequence-to-structie model. 8, are the ground-truth, and &, & are the predicted CDR sequence and structure. denotes conditional information, e.g,, framework region, antigen, and sequence/struchun ini- tialization.
framework that decouples sequence-structuure co-design into the sequence design step (Section 4.2 followed by the structure generation step (Section 4.3) Additionally, we resolve excessive token repetition of non- autoregressive models with a composition- basee sequence-level objective (Section 4.2
4.1 Problem formulation & algorithm overview
Problem statement. We first. formulate our problem for designing an antibody. Following previous works (Jin al.) [202]] Kong et al.) 2022; Verma al. 2023 Luo al. 2022 Melnyk et al. 2023), we narrow the scope of our task to sequence-structure co-design of heavy chain complementaaiigxdeeennii regions (CDRs) in the antibody
they contribute the most to antibody-antiges interaction. The heavy chaín CDRs are represented by an amino acid sequence associated with 3D positions. We provide more information on the antibody structure in Figure [
To specific, we let denote the amino acid vocabulary set and the number residues in the CDR. The CDR is represented by the sequence s ($1...., sL) and the positional information = (21....,,z)) where 8; E A corresponds information with backbone atoms. Our goal is to train a generator (s(s, alle) for the amino acid sequence and the positional information æ, given the conditional information c, e.g., the framework region, antigen, and initialization.
particular type of amino acid and H; Rmxx the associated 3D positional
A Algorithm overview. Tn ASSD, we parameterize the generator (s, 2: er)e) as a sequential composition of sequence design module (s(cc and the structure prediction module p& pa)s, c), i.e., we let ( x/c) (s (x c). This docoupling is motivated by a well-known postulate in molecular biology, called Anfinsen's dogma, which states that the native conformation of protein determined solely by its amino acid sequence (Anfinsen 1973 We provide overview of our algorithm in Figure 2
Additionally, consider a non-auutoregressive sequence generative model where the amino acids are predicted independently from each other, i.e., ce(sic) = I (s(s;(c,, as recent works (Kong et al. 2022 2023: Verma et al. 2023 Zheng ct al., 2023) demonstrate both faster speed and better performance compared to the autoregressive counterparts. However, since the non-autoregressive models generate sequences with excessively repetitive tokens, we propose new loss function prevent this.
4.2 Sequence design
In this section, we introduce the training algorithm and implementation details for sequence-only antibody design model po(s|e). Our insight is that the repetitive tokens from the generator stem from the mode- covering behavior of MLEs, i.e, each amino acid prediction aims for an average of different modes, iee,, the most frequent element. To overcome this limitation, we augment MLE training with new mode-seeking training objective based on the compositional similarity of protein sequences that mitigates the repetitive token problem.

---

Published in Transactions on Machine Learning Research (01/2025)
Intuitions for repetitive sequences. We are motivated by the observation that non-autoorerressive sequence generators exhibit the degenerate behavior of generating excessive token repetitions, e.g., 8 = ARGYYYYYYY where A, R, G, Y E A are the possible types of amino acids. However, this is critical evidence that the generator fails to learn the underlying dataset Le., the generaled sequences are out-of-disttribution. Also, overly repetitive sequences should be avoided since they may cause developabil- ity issues such as aggregation that trigger harmful immune responses in patients (Jin et al.
YYYGDR YGDYYY DYYRAG GYAYYY
2021
Wang et al. 2009).
To better illustrate this problem, consider hypothetical example with five possible amino acids (A, R, G, Y, DJ. The training dataset consists of the same conditional in- formation с and sequences s in Figure 3 The MLE objective of a non-autoregressive model (s|ccc II Po(s, (c) is given by
Figure Example to illustrate token repetition problem.
N
L
L N lo c) =
(
Σ
Σ
(ee ps).
(n)
=
where p (Pi..... PL) fo(c) is the head output distribution. we train the model with this objective, it assigns the highest likelihood to the sequence YYYYYY which is overly repetitive (proof in Appendix Aj. Such phenomenon indeed occurs for antibody protein sequences, where in certain regions, some residues have higher frequency than others (Nikula et al. 1995 Shi et al. 2014).
Loss function. We train our sequence-only antibody design model (s(cc on the loss function Leeq(0) (NL + ccompp()) where (N)L(O) is the negative log-likelihood (NLL) objective, Leomp the proposed composition- based loss function that regularize the repetitive tokens, and a some hyperparameter. particular, given sample (s, c) from the dataset, the loss function defined as follows:
(0) log po(c)) foomp (0) ( (d(ss)))
where d(s, 8) is any sequence-level metric between input amino acid sequence 8 and the reconstructed amino acid sequence 3. Following the existing NLP literature (Mathur et al. 2019 Kim et al. 2021 Zhang et al. 2019), we propose to use the cosine similarity defined as follows:
y 9
d(s, 8) = CosineSimillrittyyy.. 9)
l ||l
where y, the composition vector of the sequence 8, 8 that counts the occurrence of each amino acid type in the sequence. For example, yj = - 15ss)) where 1, is binary indicator that has the value of one if the amino acid s; is the j-th type in the vocabulary A and zero otherwise.
Training with REINFORCE. To train our generator, we use the REINFORCE trick Williams 1992 Fu
et al. 2015) with baseline = E(s.e)sold(s, $)] to reduce the variance of the gradients (Appendix Bi This results in the following training objective:
Leog (0, B(a,e)=B
log po (silc) (d(s,
bog logge((8ec)
(1)
We describe the full algorithm in Algorithm
Implementation. Our sequence-structure decoupling approach enables the use a large-scale sequence databasc during the sequence design step. We implicitly exploit this advantage by adopting protein lan-
guage model (pLM) as the sequence generator. particular, we choose ESM2-650X (Lin al. 2023) as our starting point for training and use the LoRA fine-tuning strategy (Hu et al. 2021 (details in Appendix
5

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm Training sequence design model
Require: Antibody sequence datasct {(s, c)), antibody sequence generator po (s(c).
repeat
Sample B {(s(1) ( ( D.
for .,.... M do Sample 8(9) ~ ))) and compute d(s((),)(1) end for
Compute = M Ee d(s(0) (0) and update 0 + arg min (0, as in Equation
until converged
Algorithm Training structure prediction model
Require: trained antibody sequence generator po(s)c), a sequencceto-stmucture model Pa(x c), and antibody sequence-structure dataset D = {(s, x, c)}.
repeat
Sample B {(s(), x c D.
for i 1, M do
Sample a(1) po(s)c()) and predict a(0) po end for Update & - arg min Lstrret (6, 13) as in Equation until converged
4.3 Structure prediction
Here, we describe our sequence-to-structure design model po(æ)s, We train the sequence-to-structuus model based on the following loss function:
L Luber (2t, .ti)
(ot = (
Σ
(2)
where lluber is the Huber loss to avoid mumerical stability (Jin et al.) 2021 Kong al., 2022). (f1,...,$1) po(-18,0 e) is the predicted structure given the predicted sequence 8, and po(-le) is the fixed sequence generative model trained using the loss function proposed in Section 4.2 Note that train the model to prodict the structure from generated antibody sequence & instead of the ground-truth sequence from the dataset. This is to prevent poor generalization performance during inference, caused by the exposure bias (Aroaa et al. 2022
Bengio et al. 2015 Ranzato et al. 2016
Exposure bias may occur since during training, the model conditions on ground-truth sequence, while during inference, conditions on its sequence predictions. This discrepancy leads to error accumulation, where initial mistakes propagate and compound, causing the generated sequence to deviate significantly from realistic outputs. Consequently, using ground-truth sequence for training may severely undermine the model robustness (Arora et al. 2022 Bengio et al.) 2015 Ranzato et al. 2016). We empirically demonstrate the benefit of using the generated sequence as input to the structure prediction model in Appendix D
While one could consider any graph neural network (GNN) to generate the structural information, we adopt the architecture of MEAN (Kong al. 2022) as our structure prediction model. Tn ablations, we demonstrate that our sequence-structun decoupling approach can improve the performance of any GNN- based joint sequenco-structure co-design model
We that antibody structure prediction models such as IgFold RRffolo incomplete sequences like CDR and thus perform poorly under this setting.
2023
predict the structure of

---

Published in Transactions on Machine Learning Research (01/2025)
Table AAR, RMSD, and Prep on CDR-H1 of SAbDab dataset. AR denotes autoregressive models and NAR denotes non-autoregressive models. The best and suboptimal are bolded and underlined, respectively, with Prop restricted to non-autoregressive models. ASSD (MLE) achieves the best performance, while ASSD (MLE + RL) reduces Prep with comparable performance.
CDR-H1
Models
AAR(1) (%)
RMSD(L)
Prep (4) (%)
LSTM AR-GNN RefineGNN
39.47:22.11
0.00=0.00
AR
48.39:14.72
2.90++0.17
0.00==0.00
42.03 88
0.87
0.00=0.00
MEAN
59.32:t4.71
0.91
0.29=0049
LM-Design
64.88:+1.96
0.02+10.06
NAR
ASSD (MLE)
69.41
0. 0.80
0.02=0.066
ASSD (MLE + RL)
68.71+2.25
0.86++0.09
0.00=0000
5 Antibody design experiments
Overview. In this section, evaluate our model four antibody benchmark experiments, including the SAbDAb benchmark (Section 5.1]. RAbD benchmark (Section 5.2 antibody affinity optimization (Sec-
tion 5.3 and CDR-H3 design with docked templates (Section 5.4
Baselines. We select LSTM, AR-GNN, and RefineGNN as the autoregressive baselines with implemen- tations provided in Jiin et al. (2021). For non-autoregressive baselines, we adopt MEAN and LM-Design adapted for antibody design. We have not included /eerma al. (2023) Wu & Li (2024); Martinkus et al. (2024); Frey et al. (2023) as baselines as their training codes are not publicly available. The training details
of ASSD and baselines are provided in Appendix C
Metrics. Our main sequence and structure modeling evaluation metrics are amino acid recovery (AAR) and root mean squared deviation (RMSD), respectively. AAR is defined as AAR(s, $) Σ I(s== = 6,)/ L, where á is the generated CDR sequence and is the ground-truth CDR sequence. RMSD is defined by comparing Co coordinates of generated and ground-truth CDR structures, following Jin et al. (0021 Kong ct al. (202)). Although there is controversy about whether AAR and RMSD are suitable metrics, they are still the most widely used metrics across the antibody protein literature to assess sequence/sstructure modeling abilities. Further justifications and discussions on evaluation metrics are in Section
As none of the antibody sequences in the SAbDab dataset contain more than six token repetitions, we define a sequence with more than six repetitions repetitive. Based this definition, we develop new metric called the percentage of repetitive tokens, Prep Given model's sequence predictions D (g(n) N 1==1: Prep E T(g(n) is repetitive) x 100%/N.
5.1 SAbDab benchmark
In this section, we evaluate the soquence and structure modcling performance for CDR-HI, CDR-H2, and CDR-H3 using the Structural Antibody Database (SAbDab) (Dunbar et al. 2014).
Data. Following Kong et al. (2022), we select 3977 IMGT-mumbeeed (Lffranc et al. 2003) complexes from SAbDab that contain the full heavy chain, light chain, and antigen sequence and structure. We then split the complexes into train, validation, and lest sets according to the CDR clusterings. Specifically, we use MMMsqqs2 (Steinegger Söding 2017) to assign antibodies with CDR sequence identity above 40% to the same cluster, where the sequence identity computed with the BLOSUM62 substitution matrix (Henikoff & Henikoff 1992). Then we conduct 10-fold voss-validation by splitting the clusters into a ratio of 8:1:1 for train/ http:/aall////a hiddd/tts /tstt sets, respectively. Detailed statistics of the 10-fold dataset splits are provided in Appendix F.

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2: AAR, RMSD, and Prep on CDR-H2 of SAbDab dataset. ASSD (MLF)
achieves the best
performance, while ASSD (MLE + RL) roduces Prep with comparable performance.
CDR-H2
Models
AAR(1) (%)
RMSD(4)
Prep(4) (%)
LSTM
30.4112.47
0.00
AR
AR-GNN
38.03++2.81
2.30+
0.00=0.00
RefineGNN
32.05+12.23
0.79+00.06
0.00=10.00
MEAN
48.85+2.32
0.88+0.07
0.03=0.08
LM-Design
55.31+3..2
0.02=+0.07
NAR
ASSD (MLE)
62.10
0.73
0.02-10.07
ASSD (MLE RL)
61.07+0.088
0.78+0.06
0.00=0.00
Table 3: AAR, RMSD, and Prep on CDR-H3 of SAbDab dataset. ASSD (MLE) achieves the best performance, while ASSD (MLE + RL) reduces Prep with comparable performance.
CDR-H3
Models
AAR(f) (%)
RMSD(L)
Prep(d) (%)
LSTM
15.82+1.663
0.00=00.0
AR
AR-GNN
18.72
3.60
0.03
RefineGNN
24.44+1.94
2.24+0.14
0.11+00.13
MEAN
36.50+1.60
2.23
27.40:19.23
LM-Design
37.18
29.40=10.01
NAR
ASSD (MLE)
39.56+1.399
2.21=10.8
9.22-3.76
ASSD (MLE RL)
38.67+1.64
2.19-:0.09
7.02+2.91
Results. Table [ Table 2 and Table presents our results on the SAbDab benchmark. First, we highlight that ASSD trained with the MLE objective outperforms all baselines in AAR and RMSD. However, its Prep is much higher than the autoregressive baselines, implying that some of its sequence designs are invalid. Training with our MLE-RL objective Equation (1 successfully reduces Loken repetitions, while maintaining similar performance. In particular, for CDR-HI and CDR-H2, we reduced token repetition to O; for CDR- H3, reducod Prep to 7.02%, which corresponds to 23.86% reduction in token repetition compared to the MLE-only objective.
We focus on CDR-H3 to showcase that Prep can be further decreased by increasing a in the objective function. Figure ((a) show shows that our approach establishos a Pareto fronticr over other non-autoregressive basclines in terms of Prep and AAR. Note that Prep can be reduced to a level comparable to autoregressive models while maintaining AAR above 30%, a level of performance required for CDR design models (Melnyk et al. 2023). This contrasts MEAN and LM-Design, whose Prep above 25% despite the high AAR.
5.2 RAbD benchmark
In this section, we assoss the ability to model CDR-H3 on the RosettaAntibodyDesia (RAbD) dataset, benchmark containing 60 diverse antibody-antigen complexes curated by Adool-Bryfogle et al. (2018).
Data. We use the SAbDab dataset as the train/validation set and the RAbD dataset as the test set. We climinate antibodics in SAbDab whose CDR-H3 sequences are in the same cluster as the RAbD datasot, then split the remaining clusters into train and validation sets with a ratio of 9:1. This gives 3418 and 382 antibodies for the training and validation sets, respectively.
8

---

Published in Transactions on Machine Learning Research (01/2025)
Litinigge +23
LH +83)
25
LiA LHiDDiig
36-
2
20
20-
10
30
40 46
52
44 46
30
AAR
AAR
(a)
(b)
Figure 4: Effect of a on AAR and prep- The color bar represents the value of a the MLE-RL objective. (a) Result for CDR-H3 of SAbDab benchmark. ASSD approach achieves the Pareto frontier over the non- autoregressive baselines. (b) Result for CATH 4.2 (left) and CATH 4.3 (right) benchmarks. LM-Design trained with our MLE-RL objective maintains performance comparable to ProteinMPNN, while reducing Prop significantly.
Table 4: AAR, Cosim, TM-score, RMSD, and Prep on RAbD dataset. AR denotes autoregressive models and NAR denotes non non- non-autoregreesive models. The best and suboptimal are bolded and underlined, respectively, with Prep restricted to non-autoregresssvv models. ASSD (MLE) and ASSD (MLE + RL) achiove the best performance, with ASSD (MLE + RL) achieving the lowest Prep among NAR models.
Models
AAR (t) (%)
Cosim (t)
TM-score (f) RMSD (4)
Prep (L) (%)
LSTM
16.15
0.5462
0.00
AR-GNN
18.85
0.5963
0.9630
3.56
0.00
RefineGNN
26.33
0.5797
0.9648
1.80
0.00
MEAN
37.15
0.5832
0.9806
1.82
8.33
NAR
LM-Design
38.77
0.5869
15.00
ASSD (MLE)
40.81
0.6015
0.9825
1.78
5.00
ASSD (MLE + RL)
40.53
0.6157
0.9830
1.76
3.33
Metrics. In addition to AAR and RMSD, we include cosine similarity (abbreviated CoSim) and TM- score (Zhang & Skolnick] 22004 Xu & Zhang 2010) evaluate the sequence composition similarity and global structural similarity.
Results. We report our results in Table ASSD with MLE-only training achieves the highest AAR. While maintaining similar level of AAR, we reduce token repelition by 33.4% with our MLE-RL objective function. Additionally, ASSD shows the highest structure modeling capacity, reflected by high TM-score and low RMSD. This indicates the superiority of our sequence-structure docoupling approach over the joint sequence-structure co-design models like MEAN, RefineGNN, and AR-GNN.
5.3 Antibody affinity optimization
The goal of this task is optimize the binding affinity of a given antibody-antigen complex by re-designing the CDR-H3 sequence and structure. Following Kong et al. (2022). pre-train the models on the SAbDab dataset and finc-tune them with the ITA algorithm (Yang et al. 2020) on the SKEMPT V2.0 dataset (Jankauskaité et al. 2019). To evaluate the change in binding affinity AAG, we use the official checkpoint of the oracle f from Shan et al. 2022). Since AAG evaluation requires structure, only sequence- structure co-design models are used as the baselines.
ITA algorithm. Following Kong et al. (2022). we adopt the ITA algorithm adjusted for continuous prop- erties. The amgorithm consists of two parts (1) augmenting the dataset. and (2) training the model the
9

---

Published in Transactions on Machine Learning Research (01/2025)
Table 5: Change in binding affinity AAG after affinity optimization on SKEMPI V2.0 dataset. The best and suboptimal are bolded and underlined, respectively, with Prep restricted to non-autoregressive models. ASSD (MLE) generates re-designs with the highest binding affinities on average. ASSD (MLE +
RL) achieves similar performance, while reducing Prep by 16.67%.
Random*
AR-GNN RefineGNN
MEAN
ASSD (MLE)
ASSD (MLE + RL)
A) Prep(4) (%)
+1.520
-2.279
-5.437
-7.199
-8.454
-8.162
0.000
3.774
50.943
11.321
9.434
Table 6: Average binding affinity of re-designed complexes from docked templates, measured Rosetta Energy Units. ASSD (MLE) designs antibodics with the highest binding affinitics, while ASSD (MLE RL) achieves similar performance with Prep reduced by 66.7%.
RefnnGGNN MEAN
ASSD (MLE) ASSD (MLE + RL)
Affinity (J) Prop (1) (%)
3036.17
815.32
624.65
666.90
0.00
11.67
10.00
3.33
augmented dataset. To augment the dataset D, we select an antibody-antigen complex from D and generate M = 20 candidate urructures with the model. If a candidate is valid and exhibits f < O, we add it to D and form the augmented dataset Q. Then, we fine-tune the model on @. We repeat this two-step process for T 20 iterations. The details of this algorithm are Appendix
Results. As shown in Table 5 ASSD generates re-designs with the lowest binding affinities average. By training ASSD with the MLE-RL objective (Equation (1),, we can reduce Prep by 16.67% while maintaining similar level of performance. Note that this corresponds to 81.48% reduction in token repetition with better performance compared to MEAN, whose approximately half of its re-designed sequences are repetitive and thus out-of-distributtion. For better comparison, we also provide the effect of random mutation from Kong et al. 2022), denoted Random*.
5.4 CDR-H3 design with docked templates
Here, we explore a more challenging sctting where both CDR-H3 and the binding complex are unknown. The goal of this task to generale CDR-H3 designs with higher binding alfinities this challenging setting,
Data. We modify the RAbD test set from Section 5.2 create docked templates. More concretely, we first segregate each antibody-antigen complex into CDR-H3-removed antibody and antigen, which is then used to generate 10,000 docked templates with MEGADOCK (Olue 2023). Among them, we include the top-10 scoring docked templates in the test set.
Evaluation. We prepare the best validation checkpoints from Section 5.2 and generate CDR-H3 re-design for each docked template in the test set. We then evaluate the binding affinities of the re-designed complexes with pyRosetta (Chaudhury et al. 22100) including the side-chain packing. Considering the risk of docking inaccuracy, we report the binding affinities of the top-scoring re-design for each antibody-antigen complex following Kong et al. (2022).
Results. Table i shows our results. Although ASSD trained with regular MLE objectives performs the best, about 10% of its generated sequences are overly repetitive. By training the model with the MLE-RL objective, we reduce Prep to 3.33%, a level comparable to the autoregressive model RefineGNN.
6 Protein design experiments
In this section, we demonstrate that our training algorithm generalizable to single-chain protein design with CATH 4.2 and CATH 4.3 benchmarks. Specifically, we train LM-Design (Zheng et al.] 2023) with our objective varying levels of and report the sequence modeling performance with AAR and sequence validity with Prep- As none of the sequences in CATH 4.2 and CATH 4.3 test sets contain ten repeated
10

---

Published in Transactions on Machine Learning Research (01/2025)
Table Antibody benchmark results of the joint sequence-structure co-design models and their decoupled variants. All variants ESM2-650M the sequence design model and thus share the same results for AAR. The decoupled variants improve both AAR and RMSD across all CDRs.
SAbDab
RAbD
AAR (1) (%)
RMSD (4)
AAR (1) (%)
RMSD (4)
AR-GNN
17.47
3.638
18.81
3.237
AR-GNN*
41.32
2.807
40.45
3.199
RefineGNN
23.49
2.173
22.43
1.716
RefineGNN
41.32
2.000
40.45
1.581
MEAN
35.39
2.188
37.15
1.686
MEAN
41.32
2.180
40.45
1.672
tokens, we define a sequence with more than ten token repetitions as repetitive. We also report the result of ProteinMPNN Dauparas et al. 2022) for comparison.
Results. Figure 4(b) shows our results on CATH 4.2 (left) and CATH 4.3 (right) benchmarks. Both LM-Design and ProteinMPNN, trained with their original objective, achieve high Preps indicating that dut-of-distriuution. By increasing in the MLE-RL
considerable portion of its sequence predictions objective (Equation (]), reduce the Prep while maintaining decent level of sequence recovery. In particular, our objective allows achievement of Prep ~0% for CATH 4.2 and ~5% for CATH 4.3, while maintaining AAR similar to ProteinMPNNN.
7
Ablations
7.1
Effect of sequence structure decoupling
Thus far, we have validated that the sequence-structure decoupling approach with ESM2-650M and MEAN outperforms other models in antibody benchmark experiments. Here, we demonstrate that our decoupling approach can improve the performance of other joint sequence-structure co-design models (i.e., RefineGNN, AR-GNN) by revisiting the benchmarks experiments in Section 5.1 (SAbDab) and Section 5.2 (RAbD) with focus on CDR-H3 region.
Baselines. We select AR-GNN, RefineGNN, and MEAN variants of cach model by using ESM2-650M for the sequence design model and each bascline as structure prediction model. We denote each variant with an asterisk *
baselines. We also create the decoupled
Training. We train the baselines with the objectives proposed in their original papers (Jin et al. 2021 Kong al. 2022). For the decoupled variants, we train ESM2-650M with the cross-cntropy objective and each structure prediction model with the same objective the baseline. To fully reveal the benefit of approach, train all models to their full capacity by setting the training epochs to 9999 and patience to 10.
Results. Table 7 shows that the decoupled variants outperform the original joint prediction models. This demonstrates the advantage of task-specific optimization with the ASSD framework. We note that all variants use ESM2-650M as the sequence design model and thus share the same results for AAR.
7.2 Data leakage of ESMs
In this section, we address the potential concern regarding the influence of data leakage from protein language models on our performance. To address this concern, we repeat the experiment on the SAbDab benchmark (Section 5.1 by training the corresponding Transformer architecture from scratch.
Results. As shown in Table Table 9 and Table 10 our approach still achieves the best results, excluding the baseline that uses FSM. This indicates that the high performance is mainly due to sequence-structure
11

---

Published in Transactions on Machine Learning Research (01/2025)
Table 8: AAR, RMSD, and Prep on CDR-H1 of SAbDab dataset. ASSD still achieves the best performance without using ESM. This suggests that its performance is mainly driven by sequence-structure decoupling rather than data leakage of ESMs.
CDR-H1
Models
AAR(1) (%)
RMSD(4)
Prop(4) (%)
LSTM AR-GNN RefineGNN
39.47--2.11
0.00++0.00
AR
48.39++4.72
2.90+t0.17
0.00=0.00
42.03
0.87+L0.10
0.00=10.00
MEAN NAR ASSD (MLE)
59.32++4.77
0.91++0.10
0.29-
68.9442.90
0.86+t0.19
0.02
ASSD (MLE + RL)
65.23+4..9
0.87+0.12
0.02=#0.05
Table 9: AAR, RMSD, and Prop on CDR-H2 of SAbDab dataset. ASSD still achieves the best performance without using ESM. This suggests that its performance is mainly driven by sequence-structure decoupling rather than data leakage of ESMs.
CDR-H2
Models
AAR(1) (%)
RMSD(L)
Prep (4) (%)
LSTM
30.41+2.47
0.00=0.00
AR
AR-GNN
38.03-42.81
2.30++0.16
0.00++0.00
RefineGNN
32.05+2.233
0.79
0.00=0.00
MEAN
48.85+2.32
0.88+10.07
0.03+00008
NAR
ASSD (MLE)
60.39
0.79+00008
0.02
ASSD (MLE + RL)
60.23
0.83+0.00
0.02+10.07
decoupling rather than data leakage. Our findings are consistent with Wu & Li 2024 and Wang et al. 2022 which show that protein language models alone are insufficient for antibody-relaled lasks.
8 Discussions
Conclusion. In this paper, we proposed antibody sequence-structure docoupling (ASSD), a general frame- work for highly performant sequence-structure co-design models, and a composition- objective for reducing token repetitions for non-autoregrresive models. Experiments on various benchmarks demonstrate that our approach achieves high sequence-structure modeling capacity with limited token repetitions com- pared to previous works.
Structure-based based drug design. A key consideration is whether using pre-trained protein language model for sequence design overlooks essential principles of structure- based drug design by not explicitly conditioning on structural information. However, recent wet lab experiments (Hie et al. 2024] demonstrate that pre- trained protein language models can successfully design functional antibody sequences with high binding affinity, even without explicit structure modcling. Our work builds on this empirical finding, assuming that ESM has implicitly captured structural information, evidenced in Hie al. (2024) and Lin et al.) (2023). Additionally, the proposed ASSD framework is adaptable sequence design models that condition structural data, suggesting a promising direction for future rescarch.
Limitations and future work. A limitation of our work is using AAR and RMSD to assess the antibody design models. Both metrics cannot measure the functional properties of the designed antibodies, which are important in real-life drug discovery pipelines. However, currently, there no standardized metric for assess-
ing the functional properties; thus, AAR and RMSD are the most widely used metries for antibody protein
aatibody/protein
design tasks (Jin et al. 2021 Kong et al.
2022
2023
Verma et al. 2023 Melnyk et al. 2023 Luo al. 2022
12

---

Published in Transactions on Machine Learning Research (01/2025)
Table 10: AAR, RMSD, and Prep on CDR-H3 of SAbDab dataset. ASSD still achieves the best performance without using ESM. This suggests that its performance is mainly driven by sequence-structure decoupling rather than data leakage of ESMs.
CDR-H3
Models
AAR(t) (%)
RMSD(J)
Prop(4) (%)
LSTM
15.82+1.663
0.00++0.00
AR
AR-GNN
18.72=0.82
3.60=0.58
0.03=0.07
RefineGNN
24.44=1.944
2.244:0.14
0.11
MEAN
36.50=1.60
0.07
27.40+9.23
NAR
ASSD (MLE)
37.44=1448
0.27
25.32+12.2
ASSD (MLE + RL)
37.84+1.21
2.24+0.10
21.57
Wu & Li 2024). A related limitation is that the amino acid composition used to construct our composition- based objective is not functional property. Therefore, promising future research would be developing and training antibody design models based on functional property evaluators.
For discussions on why non-autoregressive models may outperform autoregressive models biological tasks, we refer readers to Appendix H
References
Jared Adoll-Bryfolle, Oleks Kalyuzhniy, Michael Kubitz, Brian D Weitzner, Xiaozhen Hu, Yumiko Adachi, William R Schief, and Roland L Dunbrack Jr. Rosettaantbboyydeiii (rabd): general framework for computational antibody design. PLoS computational biology. 14(4)cc100112, 2018
Rahmad Akbar, Philippe A Robert, Cédric R Weber, Michael Widrich, Robert Frank, Milena Pavlović, Lonneke SchelTer, Maria Chernigovskaya, Igor Snapkov, Andrei Slabodkin, el al. In silico proof of principle machine learning-basee based antibody design at unconstrained scale. In MAbs, volume 14, pp. 2031482, Taylor Francis, 2022.
Christian B Anfinsen. Principles that govern the folding of protein chains. Science, 181(409)):233 223-030, 1973.
Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 700-710, 2022.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing syslems, 28, 2015.
Sidhartha Chaudhury, Sergey Lyskov, and Jeffrey J Gray. Pyrosetta: a script-based interface for implement- ing molecular modeling algorithms using rosetta. Bioinformalics, 26(5):686 :689 691, 2010.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et al. Robust deep learning based protein sequence design using proteinmpun. Science, 378(6615):499 56, 2022.
James Dunbar, Konrad Krawezyk, Jinwoo Leem, Terry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M Deane. Sabdab: the structural antibody database. Nucleic acids research, 42(D1)):1110 D1146, 2014.
Nathan C Frey, Dan Berenberg, Karina Zadorozhny, Joseph Kleinhenz, Julien Lafrance- Vanasse, Isidro Hotzel, Yan Wu, Stephen Ra, Richard Bonneau, Kyunghyun Cho, el al. Protein discovery with discrete walk-jump sampling. In The Twelfth International Conference on Learning Representations, 2023.
C Fu, C Fu, and Michael Michael. Handbook of simulation oplimization. Springer, 2015.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Zhangyang Gao, Cheng Tan, and Stan Z Li. Pifold: Toward effective and efficient protein inverse folding. In The Fleventh International Conference on Learning Represenlations, 2022.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zeulemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (BMNLP-IJCNLP), pp. 6112 6121, 2019.
Steven Henikoff and Jorja G Henikoff. Amino acid substitution matrices from protein blocks. Proceedings of the National Academy of Sciences, 89(22):10915 10919, 1992.
Brian L Hie, Varun R Shanker, Duo Xu, Thoodora UJ Bruun, Payton A Weidenbacher, Shaogeng Tang, Wesley Wu, John E Pak, and Peter S Kim. Efficient evolution of human antibodies from general protein language models. Nature Biotechnology, 42(2):275-288 2024.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, el al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Represen- tations, 2021.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances in neural information processing systems, 32, 2019.
Justina Jankauskaité, Brian Sanénez-Garciaa Justas Dapkúnas, Juan Fernández- Reeio, and lain Moal. Skempi 2.0: an updated benehmark of changes in protein protein binding energy, kinetics and thermody- namics upon mutation. Bioinformatics, (5(3):462-463 2019.
Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi S Jaakkola. Iterative refinement graph neural network for antibody sequence-structure co-design. In International Conference Learning Represenla- tions, 2021.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learn- ing from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2020.
Jisun Kim, Matthew McFee, Qiao Fang, Osama Abdin, and Philip M Kim. Computational and artificial intelligence-based methods for antibody development. Trends in Pharmacological Sciences, 44(3):175 189, 2023.
Suyoun Kim, Due Le, Weiyi Zheng, Tarun Singh, Abhinav Arora, Xiaoyu Zhai, Christian Fuegen, Ozlem Kalinli, and Michael L. Seltzer. Evaluating user perception of speech recognition system quality with semantic distance metric. In Interspeech, 2021.
Xiangzhe Kong, Wenbing Huang, and Yang Liu. Conditional antibody design as 3d equivariant graph translation. In The Eleventh International Conference Learning Representations, 2022.
Xiangzhe Kong, Wenbing Huang, and Yang Liu. End-to-end full-atom antibody design. In Proceedings of the 40th International Conference on Machine Learning, pp. 17409-17429, 2023.
Maric-Paule Lefranc, Christolle Pommié, Manuel Ruiz, Véronique Giudicelli, Elodie Foulquier, Lisa Truong, Valérie Choutelin-Contel, and Gérard Lefranc. Imgt unique numbering for immunoglobulin and cell receptor variable domains and ig superfamily v-like domains. Developmental & Comporative Immunology, 27(1):55-777 2003.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita, Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-ccal prediction of atomic-level protein structure with a language model. Science, 379(6637):1123 1130, 2023.
Ge Liu, Haoyang Zeng, Jonas Mueleer, Brandon Carter, Ziheng Wang, Jonas Schilz, Geraldine Horny, Michael Birnbaum, Stefan Ewert, and David К Gifford. Antibody complementarity determining re- gion design using high-capacity machine learning. Bioinformaties, 36(7):2126 2233, 2020.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng, and Jianzhu Ma. Aptigen- antibody design and optimization with diffusion-based generative models for protein structures. Aduances in Neural Information Processing Systems, 35:9754 9767, 2022.
Karolis Martinkus, Jan Ludwiczak, WEI-CHING LIANG, Julien Lafrance- Vanasse, Isidro Hotzel, Arvind Rajpal, Yan Wu, Kyunghyun Cho, Richard Bonncau, Vladimir Gligorijevic, et al. Abdiffuser: full-atom generation of in-vitro functioning antibodies. Advances in Neural Information Processing Systems, 36, 2024.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2799-2808, 2019.
Igor Melnyk, Vijil Chenthamarakshan, Pin-Y Yu Chen, Payel Das, Amit Dhurandhar, Inkit Padhi, and De- vleena Das. Reprogramming pretrained language models for antibody sequence infilling. In International Conference on Machine Learning, pp. 24398-2444 24419. PMI.R, 2023.
Asher Mullard. 2021 fda approvals. Nol. Rec. Drug Discov., (2): 88, 2022.
Tuomo K Nikula, Monica Bocchia, Michael Curcio, George Sgouros, Yan Ma, Ronald D Finn, and David A Scheinberg. Impact of the high tyrosine fraction in complementarity determining regions: measured and predicted effects of radioiodination on igg immunoreactivity. Molecular immunology, 32(12):865 872, 1995.
Masahito Ohue. Megadock-on-colab: an easy-to-use protein protein docking tool on google colaboratory.
BMC Research Notes, 16(1):229, 2023.
Tobias H Olsen, Fergus Boyles, and Charlotte M Deane. Observed antibody space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences. Protein Science, 31)):
141-146, 2022.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeicch Zaremba. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, 2016.
Jeffrey A Ruffolo, Lee-Shin Chu, Sai Pooja Mahajan, and Jeffrey J Gray. Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies. Nature communications,, 14(1):2389, 2023.
Koichiro Saka, Taro Kakuzaki, Shoichi Metsugi, Daiki Kashiwagi, Kenji Yoshida, Manabu Wada, Hiroyuki Tsunoda, and Reiji Teramoto. Autibody design using Istm based deep generative model from phage display library for affinity maturation. Scientific reports, 11(1):5852, 2021.
Sisi Shan, Shitong Luo, Ziqing Yang, Junxian Hong, Yufeng Su, Fan Ding, Lili Fu, Chenyu Li, Peng Chen, Jianzhu Ma, et al. Deep learning guided optimization of human antibody against sars-cov-2 variants with broad ncutralization. Proceedings of the National Academy of Sciences, 119(11))c1122954111, 2022.
Bin Shi, Long Ma, Xiaoyan He, Xiaomei Wang. Peng Wang, Li Zhou, and Xinsheng Yao. Comparative analysis of human and mouse immunoglobblin variable heavy regions from quest. Theoretical Biology and Medical Modelling, 11:1-11, 2014.
with imgt/highv-
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, (5(1)): 1028, 2017.
Cheng Tan, Zhangyang Gao, Jun Xia, Bozhen Hu, and Stan Z Li. Global-context aware generative protein design. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
Yogesh Verma, Markus Heinonen, and Vikas Garg. Abode: Ab initio antibody design using conjoined odes. In International Conference on Machine Learning, pp.
PMT.R 2023.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Danqing Wang, YE Fei, and Hao Zhou. On pro-training language model for antibody. In The Eleventh International Conference on Learning Represenlations, 2022.
Xiaoling Wang, Tapan K Das, Satish K Singh, and Sandeep Kumar. Potential aggregation prone regions in biotherapeutics: survey of commercial monoclonal antibodies. In MAbs, volume 1, pp. 254 267. Taylor & Francis, 2009.
Ronald Williams. Simple statistical gradient-follownng algorithms for connectionist reinforcement learning. Machine learning, 8:229-256 256, 1992.
Fang Wu and Stan Z Li. A hierarchical training paradigm for antibody structure-sequence co-design. Ad- vances in Nearal Information Processing Systems, 36, 2024.
Jinrui Xu and Yang Zhang. How significant is protein structure similarity with ti--scrr== 0.5? Bioinfor-
malics, 26(7):889- 8988,999599-----89 2010.
Kevin Yang, Wengong Jin, Kyle Swanson, Regina Barzilay, and Tommi Jaakkola. Improving molecular design by stochastic iterative target augmentation. In International Conference on Machine Learning, pp. 10716-10726. PMLR, 2020.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019.
Xiang Zhang, Tianze Ling, Zhi Jin, Sheng Xu, Zhiqiang Gao, Boyan Sun, Zijic Qiu, Nanqing Dong, Guang- shuai Wang, Guibin Wang, et al. n-primenovo: An accurate and elficient non-auloregreessve deep learning model for de novo peptide sequencing. bioRriv, pp. 2024-03 05, 2024.
Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 57(4):702-710) 2004.
Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei Ye, and Quanquan Gu. Structure-informed language models are protein designers. In International Conference Machine Learning, pp. 42317-42338. PMLR,
2023.
Chunting Zhou, Jiatao Gu, and Graham Neubig, Understanding knowledge distillation in non-autoregressive machine translation. In International Conference on Learning Representations, 2019.
16

---

Published in Transactions on Machine Learning Research (01/2025)
A Proof for hypothetical example
YYYGDR YGDYYY DYYRAG GYAYYY
Figure 5: Sequence dataset for hypothetical example in Section 4.1
We here provide proof for the hypothetical example in Section 4.1 Following from Section 4.1 objective is given by:
the MLE
(0)
ΣD i=
(n) ( Pp))
where p = (Pi...., PI.) fo(c) is the head output distributions. We can ro-paramcterize the objective for each pl as:
N Σ (s p.)
LMLE(P.)
(n)
n=1
where Σ P: is the constraint. We then compute the optimal p subject to this constraint with Sequential Least Squares Programming (SLSQP):
1
1
1
1
2
1 7
1p
0,
P
0,
7
0,0
P
0,
0
4'
4
4'
4' 4
2
1
1
1
2
1
2
1 4
1 0
pi
0,
4''
4'
0
p
T'
4
0
4
0)
p
0,
4'
where = (PA.PY P...PYPP:PPP PG. PG.PD.PR) respectively. Since pa- (s|c) = -I Cat(sslp?), the sequence with highest likelihood YYYYYY.
17

---

Published in Transactions on Machine Learning Research (01/2025)
B Derivation of objective Commp
We here derive a differentiable objective function of Ccomp = (alo)((lo) [d(s, using the REINFORCE estimator (I u et al. 2015). Consider a sample (8, c) from the dataset.
V V Cooo (0) = V Bg po [d(s.
V Σ (s(
Σ [oop(
Σ polilc/yologpp(ste)ll....8
(ol(e [d( [[(x.,)
E [d(ss 8) V) (8) + bV, (ogpo(((cc)
B [d (o
where the last line follows from Epo(a(c) Vo Po (8 With single-sample Monte Carlo approxima-
tion, we have
VoLcomp(0)) [d(s, 8) b] lologo($$
18

---

Published in Transactions on Machine Learning Research (01/2025)
C Training details
Hyperparamcters. Table 11 and Table 12 shows the hyperparameters of basclines and ESM2-6500M
respectively. For baselines, we follow the training settings in their original papers ІJin et al. 2021 Kong et al. 2022). and for ESM2-650M, we limit the batch size with a maximum token length of 6000 and train for 30 epochs. We set hyperparameter a = 0.2 in our sequence objective and use rank 2 for weights Wa, Wk. W and Wo in the multi-head attention module for LoRA fine-tuning.
Implementation for LM-Design. In paper, adopt LM-Design (Zheng al. 2023 for antibody design. LM-Design is originally pLM-based protein design model (a(s)æ) where is the protein structure and s is the corresponding sequence that folds into this structure. For antibody design tasks, we maintain the original architecture, yet use antigen structure as and initialize with framework/ante sequence information.
Code. Our implementattin built upon com com/Byte CompBByyeePrrocaa/////rrrrrrreee
httpps///gttaab.ccc///eveggggg
and https://
github.. Cww.THHUNP-MTMMEAA..M. tree/main
We deeply appreciate the authors (Lin et al. 2023: Zheng ct al.
2023
Jin et al.
2021 Kong et al. 2022 for their contributions to our project.
Machine specs. All models were trained on a machine with 48 CPU cores and 8 NVIDIA Geforce RTX 3090. We have used 1-3 GPUs for all experiments.
Table 11: Hyperparameters for basclines.
LSTM AR-GNN RefineGNN
MEAN
Vocab size
25
Dropout
0.1
Hidden dim
256
128
Number of layers
neighbors
Block size
Number of RBF kernels
16
16
Embed dim
64
Alpha
0.8
Number of iterations
Optimizer
Adam
Learning rate
0.001
Table 12: Hyperparameters for ESM2-650M used as the sequence cesign model.
csm2 133 650M UR50D
Vocab size Embed dim
1280
Number of layers
33
Optimizer
AdamW
Learning rate
0.001
19

---

Published in Transactions on Machine Learning Research (01/2025)
D Mitigating exposure bias
As stated in Scction 4.3 WC address exposure bias by using the generated sequence the input to the structure prediction model during training. We here demonstrate the benefit of this approach by comparing the performance of the structure prediction model trained with generated sequence (denoted 8) and ground- truth sequence (denoted 8). Specifically, we train the structure prediction model on SAbDab (fold 0) and RAbD benchmark, then compare their RMSD and TM-score.
Table 13: RMSD and TM-score of structure prediction model trained with generated sequence input (8) versus ground-truth sequence input (s). Training the structure prediction model with the generated sequence outperforms its counterpart.
CDR-H1 (SAbDab)
CDR-H2 (SAbDab)
CDR-H3 (SAbDab)
CDR-H3 (RAbD)
RMSD(L)
TM-score(1)
RMSD(L)
TM-score(t)
RMSD(L)
TM-score(t)
RMSD(L)
TM-score
0.7511
0.9901
0.7993
0.9910
2.2094
0.9748
1.78
0.9825
0.7077
0,9892
0.8825
0.9895
2.4743
0.9695
1.86
0.9789
E Data statistics for SAbDab
Table we detail the number of antibodies in each fold for 10-fold crose-validation Section 5.1
Following Kong et al. (2022). Vi 0,...,9, ,, we let fold be the test sct, fold be the validation set, and the remaining folds be the train set.
Table 14: Number of antibodics in each fold for Sequence and Structure Modeling (Section 5.1
CDR-H1
CDR-H2
CDR-H3
fold
539
425
390
fold
529
366
444
fold
269
363
369
fold 3
391
380
389
fold 4
305
392
420
fold 5
523
381
413
fold 6
294
409
403
fold
310
369
327
fold
472
489
355
fold
345
403
467
Total
3977
3977
3977
20

---

Published in Transactions on Machine Learning Research (01/2025)
F ITA algorithm for affinity optimization
Criterion for validity. Following (Jin ct al. 2021 we force all generated sequenees to satisfy the following
constraints: (1) net charge must be in the range
2-200 2.0], (2) sequence must not contain N-X-S/T motif,
(3) a token should not repeat more than five times, (4) perplexity of the sequence must. be below 10.
Algorithm 3 details the ITA algorithm in Section 5.3
where we have followed Kong al. (2022 for the
implementation.
Algorithm 3 ITA algorithm for antibody affinity optimization
Input: SKEMPI V2.0 antibody-antigen complex dataset. D, pre-trained po(s, alc), top-k candidates maintain
1: Initialize Q <- D 2: for 1= do
for (8,0, c) E D do Initialize C 0 for i Io. M do
(,, 20(
if valid((;; a.) and [((s, n, (8, c)) <0 then
8:
C C, c), f((s, c), (8, R, c)))}
9:
end if
10:
end for
11:
(ss, at c)) + [ a, (()) UC
12: Select top-k elements in 2( by f(.-)
13:
end for
14:
for n 1,.... N do
15:
Select batch B from Q Update 0 + 0 (B,,)
16:
17:
end for
18: end for
21

---

Published in Transactions on Machine Learning Research (01/2025)
G Effect of sample recycling on sequence quality
A common NLP approach for reducing token repetition of non-autoregreesive models is recycling tokens for
T iterations Ghazvininejad et al. 2019 Zhou et al. 2019 We here demonstrate that this approach is not effective for antibody design.
Results. As shown in Figure ((a) sample rocycling does not reduce token repctition of non-antoregrcesive models on antibody modeling tasks. It even increases Prep for CDR-H3 from about 15% to 20%. Also,
Figure 6(b) shows that sample recycling decreases AARs for all CDR regions. Contrary to this approach, crapossition-basee objective effectively reduces Prep while maintaining high AAR, as demonstrated in Section i
70
25
65
60
Peo (%)
AAR (%6
15
55
10
CDR-HI
50
CDR-H1
CDR-H2
CDR-H2
CDR-H3
45
CDRHHB
40
35
Number of Recycles (T)
Number of Recycles (T)
(a)
(b)
Figure 6: Effect of sample recycling on Prep and AAR. (a) Sample recycling does not reduce token
repetition for antibody design tasks. (b) Sample recycling even reduces AAR across all CDRs.
H Non-autoregreessive (NAR) vs. Autoregressive (AR) models
We discuss why non-autoregressive (NAR) models often outperform autoregressive (AR) models biological
tasks like protein design (Gao et al. 2022 Zheng et al. 2023), peptide sequencing [Zhang et al., 2024), and antibody design (Kong et al. 2022 Verma et al. 2023 Although the reasons for this performance gap are not fully understood, one possible explanation is that the left-to-right inductive bias of AR models conflicts with the structure of biological sequences. In particular, interactions within biological sequences rely on throo-dimonsional spatial relationships rather than the lincar sequence order imposed by AR models. This intrinsic mismatch likely contributes to the weaker performance of AR models these lasks, aligning with insights discussed in Zhang al. (2024).
22


Reviews:
[{"review_id": "FvGKgbnB0A", "reviewer": "Reviewer_iqzZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\nOverall the paper introduced an interesting attempt to decouple sequence design completely from structure design. Although counter-intuitive, the paper demonstrates improvement from adopting the decoupled approach to existing models. The experiments covered a wide range of ab design tasks and demonstrated strong results compared to the selected baselines.\nThe discussion on mode collapse in MLE training of one-step non-autoregressive training that assumes positional independence makes sense, and the proposed method should be effective in diversifying the aa type composition. \n\nWeakness:\n\nThe analysis of Non-AR model might be incomprehensive, as it mostly talks about a one-step decoding scenario and ignores methods like diffusion and flow matching, which are state-of-the-art methods on a lot of the biological molecule design problems. The observations on the failure case on MLE is mostly caused by one-step decoding, which is not suitable for learning joint distribution. This issue could be largely alleviated in diffusion as it allows every position to cross-attend with the rest to better capture the joint distribution across the sequence. The proposed regularization loss may not lead to the best convergence to joint distribution as well, as it only tries to mimic the overall amino acid type distribution as opposed to the sequence level profile. Why not use sequence level distance between generated and true sequence?\n\nThe biggest concern is the lack of comparisons to recent co-design methods. The baselines selected for the experiments are rather basic and non-sota. The authors claim that they do not compare to AbDiffuser or AbODE because they cannot access their code. However, multiple of the experiments (e.g. on SabDab and RabD) are rather standard, and there is already reported metrics in AbDiffuser/AbODE paper that can be used for benchmarking. Notably, despite the proposed method outperforming their selected baselines, the results are inferior to AbDiffuser/AdODE. E.g., AbDiffuser reported an AAR of 79.09% and 1.12RMSD on SabDab CDR-H1, and AbODE reported 70.5%AAR and 0.65 RMSD, whereas ASSD’s AAR is 69.4%, and RMSD 0.8. Similar data exists for RabD as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents ASSD, an antibody loop design framework that decouples sequence design and structure prediction. Specifically, it is trained to only design sequences first and use a separate model for loop conformation prediction. The structure information is hidden from the entire sequence design process. The authors claim this approach allows the use of the most powerful existing models in each individual design phase, such as ESM.\nThe model demonstrates better performance compared to a selected list of co-design baselines, both AR and non-AR, on metrics like RMSD and Sequence Recovery.  The paper provides ablation study where it modified three previous codesign model into the decoupling framework and showed that it leads to better performance. Furthermore, the paper introduces a composition regularization terms to address the issue of amino acid repetition in one-step non-autoregressive models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The biggest suggestion is to include more discussion and comparison with most recent co-design approach, especially diffusion/flow matching based ones. They should be considered non-AR as well, and will likely to be immune from repeating aa type problem. \nThe authors may consider including the reported metrics from AbDiffuser, AbODE, DiffAb and AbDPO, and discuss whether proposed approach can be used on those methods as well. \nThe authors could also consider Paired-OAS dataset as an additional conditional Ab design task, which is larger than RabD.\n\nThe reported metrics in Table 7 ablation study showed exactly the same AAR for the three modified methods, is this an error?\n\nThere are several typos and mislabels, e.g. there is a missing reference in page 7 paragraph 1, and the legend in Figure 4(b) should say ASSO instead of LM-Design.", "broader_impact_concerns": "No Ethical concerns this reviewer can find.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vW9jurpQLs", "reviewer": "Reviewer_pZoP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is clear and well written.\n- The paper presents convincing experimental results across four benchmarks with multiple metrics. The ablation studies address potential concerns.\n- The proposed two-stage framework and composition-based loss is simple to implement.\n\nWeaknesses:\n- The paper would benefit from a brief explanation of how non-autoregressive models can outperform their autoregressive counterparts. This seems counterintuitive, as predicting each amino acid should ideally be conditioned on other amino acids.\n- Given that the proposed model predicts each amino acid independently, the composition-based loss may not effectively teach the model to avoid repetition. It's possible that the loss simply penalizes large output values in the prediction head rather than encouraging more nuanced sequence learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a two-stage framework, called ASSD, for antibody sequence and structure design. ASSD decouples the joint generation of sequence and structure into a sequence design step and a structure prediction step. The paper also proposes a composition distance loss to penalize the excessive repetition brought by the regular MLE loss. Experiments show ASSD's superior performance over existing AR (autoregressive) and NAR (non-autoregressive) models, and mitigated repetition compared to existing NAR methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- There is a missing reference at the end of the first paragraph at page 7.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HZ5Za45ysC", "reviewer": "Reviewer_LHw5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. This paper proposed a decoupled approach to separate sequence and structure design in antibody sequence-structure codesign task. The benefit of this approach is it allows more model architectures for each task and the use of large sequence database in the sequence design step. \n2. The authors identified the generation of overly repetitive sequences in non-autoregressive protein design models, which is not desired in practical drug discovery. They propose to use a composition-based objective to regularize the generated sequence which maximize the similarity between target and the predicted sequences. \n3. The proposed method is evaluated using multiple benchmarks and experimental settings to verify its performance. \n\nWeakness: \n1. The proposed decoupling approach is overly simple, which firstly uses a sequence generator to generate protein sequence, and then a structure predictor is used to predict structure from the generated sequence. This two models are trained separately. \n2. Due to the decoupling and the lack of loss propagation between the sequence generator and structure predictor, it is not possible to conditionally design an antibody sequence given an antigen structure, which is the most practical task in structure-based antibody design. \n3. In Table 1-3, only CDR-H3 shows significant improvement in $p_{rep}$ using the proposed model. In the case of CDR-H2 and CDR-H1, the baseline models does not show a significant problem of generating repetitive sequences. Why is this? \n3. What does the color bar mean in Figure 4?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed to decouple sequence and structure generation in antibody design task, and introduced a composition-base objective in the training process to avoid overly repetitive tokens. Using these two approaches, this method can achieve SOTA performance in multiple tasks including antibody design, affinity optimization, protein design and etc.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Missing reference. Under Table 2 on page 7, \"Further justifications and discussions on evaluation metrics are in ??\" \n2. Under Equation 2 on page 6, \"$\\hat{x} $.... is the predicted sequence\", $\\hat{x}$ should be the predicted structure given the predicted sequence $\\hat{s}$\n3. Missing legend for ASSD in Figure 4(b)", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a decoupling approach to separate sequence and structure design. The paper also introduces a composition-based objective for antibody sequence generation that resolves the token repetition problem

The key claims are that (i) the experiments demonstrate that the proposed sequence-structure decoupling approach improves performance in a range of antibody design experiments; and (ii) the introduced training algorithm effectively prevents excessive token repetitions. 

While the reviewers expressed some minor concerns in their recommendations regarding the selection of baselines and the adopted sequence design methods, and whether the proposed composition loss was particularly effective, there was consensus that the experiments were sufficient to support the claims. There were requests to include additional baselines during the review process and the authors satisfactorily addressed those requests. Insisting on the inclusion of additional, unspecified baselines (“strengthening of the baseline selection”) after the completion of the review process is unreasonable. The authors provided satisfactory responses to the other criticisms of the reviewers concerning simplicity of the technique and the effectiveness and suitability of the composition loss. 

Audience: A subgroup of TMLR’s audience is interested and conducts research into antibody design. The presented results indicate that a simple method can outperform some recent techniques. This counterintuitive finding is likely to be of interest for the audience, and the introduced method can potentially serve as an effective baseline for future research.

Certifications: The paper does not meet the bar for any certifications.
Decision: Accept (poster)