Bayesian Optimization of Antibodies Informed by a Generative Model of Evolving Sequences
Authors: Alan Nawzad Amin
Nate Gruver
Yilun Kuang
Yucen Lily Li
Hunter Elliott
Calvin McCarter
Aniruddh Raghu
Peyton Greenside
Andrew Gordon Wilson
Venue: ICLR 2025 Spotlight (2025)
Keywords: Bayesian optimization
generative model
antibody
biological sequence
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: To build effective therapeutics, biologists iteratively mutate antibody sequences to improve binding and stability. Proposed mutations can be informed by previous measurements or by learning from large antibody databases to predict only typical antibodies. Unfortunately, the space of typical antibodies is enormous to search, and experiments often fail to find suitable antibodies on a budget. We introduce Clone-informed Bayesian Optimization (CloneBO), a Bayesian optimization procedure that efficiently optimizes antibodies in the lab by teaching a generative model how our immune system optimizes antibodies. Our immune system makes antibodies by iteratively evolving specific portions of their sequences to bind their target strongly and stably, resulting in a set of related, evolving sequences known as a *clonal family*. We train a large language model, CloneLM, on hundreds of thousands of clonal families and use it to design sequences with mutations that are most likely to optimize an antibody within the human immune system. We propose to guide our designs to fit previous measurements with a twisted sequential Monte Carlo procedure. We show that CloneBO optimizes antibodies substantially more efficiently than previous methods in realistic *in silico* experiments and designs stronger and more stable binders in *in vitro* wet lab experiments.

Published as a conference paper at ICLR 2025
BAYESIAN OPTIMIZATION OF ANTIBODIES INFORMED BY A GENERATIVE MODEL OF EVOLVING SEQUENCES
Alan Nawzad Amin New York University
Nate Gruver New York University
Yilun Kuang* New York University
Lily Li'
New York University
Hunter Elliott BigHat Biosciences
Calvin McCarter
Aniruddh Raghu BigHat Biosciences
Peyton Greenside BigHat Biosciences
BigHat Biosciences
Andrew Gordon Wilson New York University
ABSTRACT
To build effective therapeutics, biologists iteratively mutate antibody sequences to improve binding and stability. Proposed mutations can be informed by previous measurements or by learning from large antibody databases to predict only typical antibodies. Unfortunately, the space of typical antibodies is enormous to search, and experiments often fail to find suitable antibodies on a budget. We introduce Clone-informed Bayesian Optimization (CloneBO), Bayesian optimization pro- cedure that efficiently optimizes antibodies in the lab by teaching a generative model how our immune system optimizes antibodies. Our immune system makes antibodies by iteratively evolving specific portions of their sequences to bind their target strongly and stably, resulting as clonal family. We train a large language model, CloneLM, on hundreds of thousands of clonal families and use it design sequences with mutations that are most likely to optimize an antibody within the human immune system. We pro- pose to guide our designs to fit previous measurements with a twisted sequential Monte Carlo procedure. We show that CIoneBO optimizes antibodies substan- tially more efficiently than previous methods in realistic in silico experiments and designs stronger and more stable binders in in vitro wet lab experiments.
set of related, evolving sequences known
INTRODUCTION
Antibody therapeutics are the fastest growing class of drugs, with approved treatments for breadth of disorders ranging from cancer to autoimmune disease to infectious disease (Carter & Lazar, 2018). Biologists wish design antibodies that strongly bind to targets of interest while being stable in the human body. Stable antibodies do not unfold or cause an adverse immune reaction (Jarasch al., 2015). To develop these antibodies, biologists first screen many diverse antibody sequences, or use a lab animal's immune system to find an initial candidate that binds target. This candidate often does not bind strongly or unstable in the human body, so is used as a starting point in an iterative optimization experiment in which biologists predict mutations that result in better more stable binders (Lu et al., 2020).
To make these predictions, we can learn from up to thousands of sequence measurements from many previous iterations (Rapp et al., 2024; Yang et al., 2019; Fannjiang et al., 2022; Brookes et al., 2019) 9)) We can also learn from databases of protein sequences avoid predicting mutations that produce nonfunctional antibodies (Gruver et al., 2023; Stanton et al., 2022; Hie et al., 2023; Prihoda et al., 2022). However, even with this restriction, there are a combinatorial number of mutations we could predict, only handful of which are beneficial. Therefore, optimization experiments regularly fail to find suitable sequences on a budget.
To optimize more efficiently than current methods, we need an informed prior about where and how to mutate to positively affect binding and stability. Ideally we could learn what mutations often lead to better sequences in optimization experiments in the lab. Unfortunately such data is scarce. In principle, we can instead learn from abundant data about what mutations often lead to better
"Equal contribution

---

Published as a conference paper at ICLR 2025
Immune System
Clone- Bayesian Optimization (CloneBO)
Simulate a family evolved to bind an antigen
Measure simulated sequence in lab
Evolution
candidate
Starting
CloneLM Labels
Clonal Family
Figure 1: Our immune system introduces mutations (blue) to evolve weak binders of target into strong binders (green). The result is a set of related sequences that bind the antigen strongly and stably known as a clonal family. We use model trained on these families, CloneLM, to perform Bayesian optimization in a procedure called CloneBO. We use experimental data to generate a clonal family that might have evolved to bind our antigen and suggest sequences to test in the lab.
sequences in our bodies. To make an antibody that binds new target, our immune system evolves sets of related sequences known as clonal families; through selection, sequences in clonal families accumulate mutations that increase binding to a target while maintaining stability (Burnett et al., 2018). Through large-scale sequencing efforts, we can now learn from databases that contain large numbers of these evolving sequences (Olsen et al., 2022a).
In this paper, we introduce Clone-informed Bayesian optimization (CIoneBO), a Bayesian optimiza- tion procedure which efficiently optimizes antibody sequences in the lab by teaching generative model how the human immune system optimizes antibodies (Fig. 1). In Section 2 we review related work. In Section we introduce the problem of iterative Bayesian optimization. In Section 4 describe how in theory we can build a prior for where and how to mutate given observed clonal families. In Section 5 we build such a prior in practice by fitting large language model, CloneLM, to hundreds of thousands of clonal families, We take a martingale posterior approach to sampling in which we generate new clonal families that contain our candidate. In Section 6 we describe how to condition on previous measurements using twisted sequential Monte Carlo procedure so that good mutations are included in our clonal family, and bad mutations are excluded. We use our model to build Bayesian optimization procedure, CloneBO. In Section 7 we show that CloneBO optimizes realistic oracles for stability and binding strength much more efficiently than current methods and also designs strong and stable binders in wet lab experiments. CloneBO outperforms naive and in- formed greedy methods as well as LaMBO, a state of the art method for optimizing sequences. In Section 8 we conclude and describe directions for future work.
Our code and model weights are available at https: / / wtw.:ttu..cooooeeee http:/..//m/aaaaaaaaaiiiii cloneBo.
2 RELATED WORK
To iteratively optimize a protein, can predict sequences using previous measurements (Rapp et al., 2024; Yang et al., 2019; Fannjiang et al., 2022; Brookes et al., 2019). To optimize antibodies for stability in the human body, Hic et al. (2023) and Prihoda al. (2022) suggest introducing mu- tations to select for typicality, make them look more typical, measured by the likelihood of a model trained on large databases of protein sequences. More generally, Gruver et al. (2023) and Stanton et al. (2022) avoid suggesting atypical protein sequences by training a latent space to represent database of sequences and then optimizing in this latent space. However, even the space of typical antibodies is combinatorially large, and therefore challenging to search using only up to thousands of previous measurements. CIoneBO builds an informed prior to efficiently search this space.
CIoneBO builds this prior using clonal families sets of sequences evolving to strongly and stably bind a target (Burnett et al., 2018). Biologists infer evolutionary pressures on antibodies by examin- ing individual clonal families (Mascola & Haynes, 2013) or comparing clonal families (Phad et al., 2022). In the lab, "repertoire mining" optimizes antibodies by suggesting mutations from sequences in clonal family that contains the candidate (Richardson et al., 2021; Olsen et al., 2023). In prac- tice, such a family rarely exists. CIoneBO optimizes a candidate by generating new clonal families that contain the candidate and that match experimental data.
2

---

Published as a conference paper at ICLR 2025
To build prior over measurements in the lab, we assume that sequences in a clonal family distributed with abundance according to their fitness, and that fitness is close to the function we measure in the lab. These are standard assumptions in generative modelling of protein sequences (Weinstein et al., 2022) - one fits a distribution p to a set of protein sequences, then uses log p(X) as estimate of the fitness of sequence X; this fitness then correlates strongly with the function of the protein as measured in the lab (Frazer et al., 2021; Riesselman et al., 2018; Notin et al., 2022; Shin et al., 2021). In our case, each clonal family has its own fitness function which we use to build a prior over fitness functions. Our model, CloneLM, models clonal families as sets of sequences, similar the architecture of Truong & Bepler (2023) who used a language model to model protein families as sets of sequences.
For each clonal family we observe a set of sequences which evolve with respect to a latent fitness function drawn from a prior. Instead of attempting to build an explicit latent variable model, Fong et al. (2024) suggest performing Bayesian inference with a "martingale posterior". Instead of sam- pling and conditioning on the latent variable, they do the same with large number of observations. Lee et al. (2022) suggests using this approach for Bayesian optimization. Falck et al. (2024) suggests that, with some bias, large language models can perform martingale posterior inference. We take this approach when sampling from our prior. We use a large language model to flexibly fit observed sequences and sample sets of sequences, i.c. clonal families, as draws from our prior.
When proposing sequences, we sample a clonal family from an autoregressive language model, CloneLM, but condition its output to fit experimental measurements. To do so, we build twisted sequential Monte Carlo procedure (Whiteley & Lee, 2014) in which we bias the generation of each letter towards the posterior. This technique used to sample from filtering models (Lawson et al., 2023; 2024), large language models (Zhao et al., 2024), or diffusion models (Trippe et al., 2023).
Complementary with work on iterative design are structure-basee de novo design methods which aim to predict antibody sequences that bind particular antigen (Jin et al., 2021; Luo et al., 2022; Kong et al., 2023). These models have the potential to design starting sequences for iterative optimization. These models could in principle also be used for iterative design, but cannot make use of pool of previous measurements, and must have access to structure. We show below empirically that these models are not well suited for this task.
BACKGROUND
We start with an antibody variable domain Xo. sequence of 110 130 letters made of the 20 amino acids, identified to bind a target of interest. Xo often does not bind the target strongly enough or is unstable in the human body, making it unsuitable as a therapeutic. We therefore iteratively propose sequences we expect are stronger or more stable binders, Xy...., XN, and measure their binding or stability in the lab Yi...., YN-
We assume that our measurements are evaluations of a function f that takes sequences to a scalar measurement of binding or stability in the lab: Y, = f(x,) To suggest the next sequence, XN+1, given XINN YEN we can perform Bayesian optimization (Frazier, 2018). First we place a prior on given our known weak or unstable binder Xo. x() Then we infer by building a posterior, x(f] XINN, YI.N). Finally, we suggest X NN+1 given our knowledge of J, for example by Thomp- son sampling: we sample a value of f we believe to be plausible, f ~ p(f]Xo, XI,N, Yi.N), and test the sequence that maximizes this sample, XN+1 = (rgmaxxf(X)
In the lab we often have a limited experimental budget, and therefore want to find a strong or stable binder in as few iterations as possible. To do so, we need an accurate prior on f. The ideal prior could in principle be constructed by performing many optimization experiments in the lab for diverse array of targets and starting candidates and measuring f in each case. Unfortunately, performing large number of these experiments is prohibitively expensive.
4 A PRIOR FROM FITNESS FUNCTIONS OF EVOLVING ANTIBODY SEQUENCES
While we do not have access to a large number of optimization experiments in the lab, we do have access to a large number of similar optimization experiments that occur in our bodies. Our immune system generates antibodies by first identifying candidate sequences that bind a target. then evolves this sequence towards binding its target more strongly while remaining stable in the body: mutations are introduced to sequences and those sequences with higher "fitness" those that bind the target more strongly and stably are selected for reproduction. Each starting candidate
3

---

Published as a conference paper at ICLR 2025
sequence typically produces many diverse sequences that have been evolved to bind target strongly and stably. For each optimization experiment the immune system performs, we therefore observe a set of evolved sequences X,,..... .xM known "clonal family".
The I we measure in the lab measures binding and stability, similar to a function of the fitness of sequences under selection. Therefore build prior over f we start by building a prior over fitness functions F. Then in Section 6 we allow for some discrepancy between f and F which may be caused by a difference between measurements in the lab and selection our bodies.
To get prior over functions F from observed clonal families, we first note that the distribution of sequences we observe, y 1X,, x2....) ). can be written as a Bayesian model. The probability of observing set of sequences in clonal family is exchangable, i.e. it does not depend on their order;
so, by De Finetti's s theorem' (Hewitt & Savage, 1955), sequences in each clonal family are generated iid conditional latent random variable which we call clone:
f II pmXm ((cooe)
M
p(X1:ar)
Next we make the standard assumption that families of evolving proteins are distributed with abun- dance proportional to their fitness (Weinstein et al., 2022), that is,
F(X)= = log p( X (clone).
(1)
Sella & Hirsh (2005) showed that Eqn. holds exactly if a protein evolves under over long time scales. In reality, sequences drawn from p(X clone) can also be correlated by being descendants of the same sequence, but we make the standard assumption that these correlations can be ignored (We- instein et al., 2022). Finally, we can represent that the initial candidate X, binds the target by assum- ing we have observed it in the clonal family, i.e., by looking at p(clon (xoloon(X)))>> x p(clone)p( (Xx /clonc..
We can therefore sample fitness functions from p(F) (Xo) in theory by 1) sampling clonal families that contain Xo. clone" ~ p(clone|Xo)) and then 2) we can set F(X) log p(X|clone")
CLONELM: LEARNING A PRIOR OVER FITNESS FUNCTIONS
In this section fit model to the distribution of clonal families and use it to sample fitness functions in practice. In principle, we could build model with an explicit latent variable meant represent clone. Instead, we take a martingale posterior approach (Fong et al., 2024) simply by building an accurate model of clonal sequences we learn an implicit prior on clone that we can approximately sample from.
In Section 5.1 we fit an autoregressive large language model, CloneLM, to large scale clonal fam- ily data and show can generate realistic clonal families XI:M that contain a candidate sequence Xo- In Section 5.2 we show that given clone, Xo.M, CloneLM implicitly infers the fitness func- Finally in Section 5.3 we
tion when predicting sequences: F(X) RO log PCloneLM M (Xo:M)) ( show CloneLM can therefore sample fitness functions from an implicit prior clone by generating realistic clonal families that contain X, then inferring their fitness functions.
5.1 FITTING LARGE LANGUAGE MODEL TO GENERATE CLONAL FAMILIES
We train a large language model on large scale human clonal family data. Each antibody is made up of amino acid sequences the "light chain" and the "heavy chain".. We train separate models on heavy and light chains of antibody sequences in clonal families.
To build a training set, collect all sets of human heavy and light chain antibody sequences from the database of Observed Antibody Space (OAS) (Olsen et al., 2022a). We annotate clonal families in each set of sequences using FastBCR (Wang et al., 2023) and remove any clonal family with fewer than 25 sequences. Our dataset contains 908 thousand heavy chain clonal families and 34 thousand light chain clonal families.
We then train autoregressive language models with 377 million parameters based on the Mistral-7B architecture on the heavy and light chain datasets (Jiang et al., 2023). We represent each clonal family as sequence of tokens made up of all the amino acid sequences in the clonal family each separated by a special requence-scparatie token. We place spaces between amino acids so that the tokenizer represents each amino acid with its own token. Our model, CloneLM, accurately fits this
We ignore the dependence between the number of sequences we observe, M, and the sequences themselves
4

---

Published as a conference paper at ICLR 2025
Prompt
Clonal
family
SLRGU/E
Sample
Sample
E:E www.LL...... www. Tothoge
Sample 3
Figure 2: CloneLM samples plausible clones. We compare sequences in clonal family to families generated by CloneLM conditional on Xo ("Prompt"). We align sequences to X, and highlight locations where sequences differ from Xo in blue. The sampled clonal families have variants in similar places, are similarly diverse as the real one, and share similar variants within each family.
2.0
A Human-ness i
10
KL / /engthh
0.2
Perplexity
1.8
0.1
1.4
0.0
CF CF
SF
14
M
M
Sequence
(b)
(d)
Figure 3: CloneLM is a prior over fitness functions. (a) For 5 different clonal families, with start- ing sequences Xo. PC (X [XM M+1
gets close PCloneLM Marso+ +1 | in KL. We shade one standard deviation across 10 samples of X1:M X1 X1.M- (b) For different heavy chain clonal families, PCloueLM X (o.M) better predicts sequences in clonal family when conditioned on more sequences from that same clonal family Xo:M- We shade one standard deviation across 10 samples of X1:M- (c) To sample from our prior F' Xo perform martingale posterior procedure. (d) We evolve three antibody therapeutics with three mutations from 25 sampled fitness functions. These sequences evolve to look more like human antibodies.
(X(:ss) |
data it achieves test perplexity of 1.276 on the heavy chain data and 1.267 on the light chain data. We provide details of the data curation and of training the models in App. A.1.
To see if CloneLM generates realistic clonal families, Fig. we compare a heavy chain clonal randomly selected
family from the test set to clonal families generated by CloneLM conditional sequence from the original family Xo- We see the sampled clonal familics are similarly diverse to the real clonal family, include variants in similar locations the true clonal family, and sequences within the same sampled clonal families contain similar variants. We show more examples gen- erated heavy and light chain clonal families in Appendix C.I.
5.2 APPROXIMATELY EXTRACTING A FITNESS LANDSCAPE FROM A CLONAL FAMILY
CloneLM does not explicitly represent the latent variable clone so we cannot exactly query the fitness function F(X)= = log (XX/ccoooc)) However, CloneLM approximates the predictive distribu- tion of Bayesian model p(X M+1 No.M) which implicitly integrates over the latent clone:
( (X = f x No:sa)
As M + oo, p(clone) Xo.M) converges to point mass the latent clone" generating the sequences. Therefore, in theory, as M becomes large, log MM (M (o..M) should converge to F::
Pog.ClloeLMM ( ( = log p( h++1 Xo:M) R log MM = F(XM+1).
We see that CloneLM can infer as such on real data as well - as M becomes large, the predictive of CloneLM, PCIoneLM((MMan (XM+1 approaches convergence and its limit in- creasingly approaches the distribution of sequences in a clonal family, p(X (clone). First in Fig. 3a we take random sequences Xo:Mwae Xb.s from heavy chain clonal families and see if
5

---

Published as a conference paper at ICLR 2025
PCloneLM (X (XM+1 (( converges to PCloneLM (.LLLMMMMMMMsssss Morgo+1 Mi:Misege)) in Kullback-Leibler diver- gence. Setting Misrge = 24, we see that although the divergence does not go to 0, the distributions become very similar M becomes large. Next in Fig. 3b we take random sequences Xo:ar from heavy chain clonal families and see PCloneLMM X (X approaches p( Xclone). We see indeed that when decreasing in M.
(X to predict sequences in the clonal family, its perplexity is
5.3
SAMPLED FITNESS LANDSCAPES AND EVOLVING SEQUENCES
Given model samples realistic clonal familiee and recover F(X) log p(X]clnne) approximately sample from PCloneLM (o))) using martingale posterior procedure (Fong et al., 2024) we sample from the prior of clonal families that contain Xo. XI.M PCloneLM (X XI:M), and then approximate the fitness function as F(X)= ~ PCloueLM (X( (Fig. 3c).
The fitness functions we sample reflect how our immune systems evolve antibodies. In Fig. 3d we take the heavy chains of three antibody therapeutics. bococizumab, trastuzumab, and ramibizimab, sample fitness functions setting from CloneI.M conditional on these sequences with M 10, and iteratively evolve these sequences by adding the most likely mutation under each sampled fitness functions three times. These therapeutics are not originally human sequences and therefore can be unstable in the human body in particular, bococizumab was discontinued due to harsh side effects. they were to evolve our bodies, we would expect them to become more human-lite like, and therefore likely more stable. Indeed like human antibodies, where human-ness is measured by the likelihood of IgLM (Shuai et al., 2023), model trained on a large set of human antibodies.
that as we evolve these sequences they look more
CLONEBO: INFERENCE WITH EXPERIMENTAL MEASUREMENTS
We now describe how to use our prior over fitness functions F to optimize sequences in the lab. In Section 6.1 we build prior for measurements in the lab using our prior for F. We cannot exactly condition on the implicit clone, so in Section 6.2 we approximate the posterior over the latent clone with a posterior over concrete clonal families XI:a-- In Section 6.3 we describe how to sample from our approximate posterior using twisted sequential Monte Carlo (SMC) procedure. Finally in Section 6.4 we describe how to suggest sequences to perform iterative optimization with CloneLM: we call our method Clone-informed Bayesian Optimization (CloneBO).
6.1 BUILDING A PRIOR ON LABORATORY MEASUREMENTS
In our bodies, antibodies are optimized to stably and strongly bind their targets. In principle, we are interested in doing same in the lab. We therefore assume that the function we optimize in the lab, f. approximately drawn from our prior on fitness functions, F, while allowing for some discrepancy due to mismatches between measurements in the lab and those in our bodies. For simplicity, we assume that ff is an affine linear transformation of F and that the deviation between experiment and fitness is independent normal with error o²: that is, for some T > 0, C, calling Fn log p( Xx, clone),
Yn clone N(TF++C.O C,oO)..
To reflect our vague uncertainty about T and C' we use uniform priors: C ~ Uniform(-oo oc) and Uniform(O, oc). With these priors, we get an analytical expression for the marginal likelihood.
Proposition 6.1. (Proof in App. D.1.)
For some constant
D, and R
VN St Cor(FL.N YI,N), with &b as the Gaussian CDF,
log.(YI.N - log Cov (ov(FF.)) + R2 + log ((R) + D.
(2)
The first term in Eqn. 2 pushes the fitness values of the measured sequences, FL,N, to be different, while the later terms push FLN to strongly and positively correlate with YINN
As before, we assume we have starting candidate X, that belongs in the clonal family. Therefore,
CelonneYi.N XINN x p(clone)x(Xx(llong))

---

Published as a conference paper at ICLR 2025
15
tSMC
(SMC:
30
-S: 300
(R² 0.48)
(R² 0.76)
(SMC: 4
IS: 100
IS: 1000
Bad X
Melting Temp.
70
350
300
Good Y
80
50
150
100
1
2
3
6
FM FM
(a)
(b)
(c)
Figure 4: We accurately sample functions from the posterior with a twisted SMC procedure. a) To sample from our posterior, we bias our generated sequences to look more like those sequences that were measured in the lab to be good. b) A sample from tSMC better fits the data than an importance sample. We show a line of best fit between FL (fitness from a clone of M sequences) and YI.N (measurement) for example clonal families sampled by importance sampling or twisted SMC, with M 6, D 300 particles for IS, and D for twisted SMC. c) We quantify the result from (b) across 10 replicates for various clone sizes M.
6.2 APPROXIMATING THE POSTERIOR
To infer given measurements YI.N XI.N. XI.N we would like to sample from the posterior F p(F|Y]:N X1.N, Xo). As we only implicitly represent clone, we approximate the posterior by swapping the latent clone for a concrete clonal family, X1:a;; then, as in Sec. 5.2, we can approxi- mately query the fitness function F(X) log
To create an approximate posterior, we replace Fn with F = -oogp((n
Pm PMMMM |YEN, X Xo) x p(X (X)) (YINNIANN
(3)
As .M + oc, FM + FLNN. so, pM converges to the distribution one obtains by sampling clone from the posterior and sampling Xm p( p(Xmnclone) iid:
Proposition 6.2. (Proof in App. D.2) As M oo, PM converges to the true posterior of Xo.M-
In practice, we approximate p(Xi:M (x)) and FM with CloneLM.
6.3 SAMPLING FROM THE APPROXIMATE POSTERIOR WITH TWISTED SMC
To sample from РM*. we need to generate M sequences from CloneLM such that the probabilities of the (M + 1) st sequence matches experimental measurements. Naively, might sample X1:M from CloneLM and then importance sample. However, the space of sequences is large, and we may fail to resample mutation that improved measurements in the lab.
Instead we bias generation at each letter by adding measured sequences to our clonal family so that mutations that improve measurements are encouraged and those that harm measurements are avoided (Fig. 4a). We call this bias a "twisted" distribution, a term from the sequential Monte Carlo literature. In practice, this bias does not exactly sample from the posterior, so we efficiently correct for the discrepancy with a sequential Monte Carlo procedure.
Twisted distributions Define ( to be the letter at the Ith position of sequence X and ( as the first letters in X; if is greater than the length of X, we define (") as an empty position. To build our twisted distributions, we decompose the likelihood of Xn given M + 1 sequences into contributions from the first M sequences and contributions from each letter sequence M 1:
M++ log ((n = log p(X M+ |NOM, + log P(XM+1lXoMM)
((X M+ Ia ((1-1) (x))
log
M+1 No.M, x(1-1) + FM
(4)
p(x() (
FMM+10 F. n
7

---

Published as a conference paper at ICLR 2025
We can calculate Fn (M++1(()) by adding Xn to the end of the clonal family X1:ar and calculating how much the conditional likelihood of the I-th letter of X increases. To approximately sample from the posterior therefore, when sampling each letter we bias towards letters with F, F.++1.0) that better match experimental measurements (Fig. 4a); we call these approximations of the posterior the "twisting distributions" and we define them formally in App. A.2.
The twisted distributions are not the exact marginals. We correct for this discrepancy by sampling D sequences at time and importance sampling in sequential Monte Carlo procedure (SMC), which we also describe in App. A.2. The final method is known as twisted SMC, and when D = 1, it is equivalent to sampling from the approximations described above.
Empirical results We sample clonal family conditional on laboratory measurements of the melt- ing temperature of 75 related antibodies from an experiment described in Sec. 7. In Fig. 4b and 4c we that clonal families from twisted SMC fit the experimental data substantially better than clonal families importance sampled from unconditional samples from CloneLM. We also see in Fig. 4c that correcting for bias with likelihood p(Y F. plateaus, reflecting the convergence of fMs to the true posterior. We show similar results for laboratory measurements of binding App. C.2
4 also improves the fit the data and that as M increases, the
6.4 BAYESIAN OPTIMIZATION WITH CLONEBO
After sampling from the posterior we would like to suggest sequences to test in lab. We take Thompson sampling approach (Russo et al., 2018): propose testing the sequence predicted to maximize F(X). and therefore maximize f(X), in the lab. We cannot optimize F(X) over all sequences, in practice we start with 4 sequences with the highest measurements Y and iteratively optimize F(X) over the top substitution for up to 3 substitutions.
In theory, Xo represents a candidate sequence optimize. In practice, we found it helpful to take greedy approach- - we randomly select Xo from the 4 sequences with the highest measurements Y.
Conditioning on large number of measurements NI.N YI.N is computationally expensive. To accommodate large N, other Bayesian optimization methods build summaries of the measurements, for example by fitting neural network to them (Stanton et al., 2022). In our case, we only condition on the measurements of sequences predicted to be most informative we calculate the probability that each X,, appears in a clonal family with Xo. p(X,, Xx)) and condition on the measurements of the 75 most likely sequences. Additional details of CloneBO provided in App. A.3.
7 EXPERIMENTS
Now we demonstrate that CloneBO efficiently optimizes antibody sequences in practice. In Section 7.1 we demonstrate that our method efficiently optimizes fitness functions or laboratory measure- ments in silico. In Section 7.2 we also show that our method suggests mutations that optimize sequences the lab. We provide experimental details App. B.
7.1 OPTIMIZING ANTIBODIES in silico
We show that CloneBO efficiently optimizes fitness functions or measurements the lab silico. To simulate fitness functions or lab measurements, we train oracles I on real data. We show that CloneBO outperforms naive and informed basclines.
7.1.1 BASELINES
First we consider a naive Greedy baseline which suggests a random substitution of one of the top 4 sequences. We also compare to an informed greedy baseline which randomly picks one of the top 4 sequences and introduces the mutation predicted to make the antibody look most like a typical antibody, where typicality in measured by the likelihood of masked language model trained on antibody sequences, Sapiens (Prihoda et al., 2022); this is a popular strategy for making antibodies that are more stable in the body. We also compare to LaMBO (Stanton et al., 2022), a state-of-the-art Bayesian optimization method for sequences which builds latent space using pool of sequences and fits experimental measurements in this latent space; by conditioning on this latent space, is less likely to suggest atypical sequences. We also build a LaMBO model informed by the space of antibodies by pretraining its latent space using 100000 antibody sequences from the observed antibody space database (Olsen al., 2022a), LaMBO-Ab. We also compare to other popular
8

---

Published as a conference paper at ICLR 2025
Fitness
Binding
Stability
log (og((xlcooe))
-12
Norm. -1ogra
0.4
Melting Temp.
74
0.35
72
-18
70
0.3
%
-44
0.25
%
-30
CbAS
Dyna-
CMA-E5
EvaBO
Genetic
Adalead
Greedy
LaMBO
Sapiens
LaMBO. LaMBO-Ab
CloneBo
Cbels Dyna-PRO
CMA-ES
EvaBo
Genetic
Adalead
Greedy
LaMBO
Sapiens
LaMBO. Ab
CloneBo
40
100
Steps
Model
Model
(a)
Figure 5: CLoneBO efficiently optimizes antibodies in silico. We show the mean and standard deviation of the best acheived value across 10 replicates. (a) CloneBO efficiently optimizes a fitness function. The blue line is CIoneBO; the grey LaMBO-Ab, LaMBO, Sapiens, and Greedy. (b) CloneBO optimizes binding and stability in silico over 100 steps of iterative design (p value is Mann-Whiteey). It does significantly better than the next best method for binding (p=0.018 Mann- Whitney) and stability (p=0.006 Hann-Whitney!
strategies for iterative optimization of sequences that do not have antibody-based priors, Genetic, AdaLead (Sinai et al., 2020), and EvoBO (Sinai et al., 2020); CMA-ES (Hansen & Ostermeier, 2001), Dyna-PPO (Angermueller et al., 2020), and CbAS (Brookes et al., 2019). In total, we compare CIoneBO to 10 baselines that represent state-of-the of-tth-attttttt industry practice".
7.1.2 RESULTS
Oracle of a fitness function of a clonal family We first demonstrate the potential of the CIoneBO prior to accelerate optimization. We build oracle to simulate the fitness function of real human clonal family, that is, a function from the CloneBO prior. We trained a language model on a heavy chain clonal family of 10015 sequences from our test set and try to maximize f(X)
the log
likelihood of X of this model. We start with single measurement X₁. Y1 where X,, is a sequence from the clonal family, Very few mutations of an antibody improve fitness, so in Fig. 5a we see some baselines struggle to identify positive mutations. CloneBO's 's prior on the other hand gives knowledge about what sorts of mutations are most likely to improve fitness allowing it to quickly optimize f even at very low N.
Oracles from laboratory measurements of melting temperature and binding We next demon- strate the utility of CI in an in silico simulation of a realistic in-lab setting. We trained oracles on lab measurements from an experiment that aimed to optimize VHH domain, sequence similar to an antibody heavy chain, for binding and stability at high temperature. We trained neural network ensembles on the melting temperature and binding (measured in - log KD) measurements and try to maximize f(X) the mean predictions of these ensembles. We simulate starting part way through this experiment by starting with 1000 measurements X110000 Y1:1000 where X1:1000 are the first. 1000 sequences measured in the experiment and Y1-1000 are oracle predictions. We do not expect mutations outside of the CDR regions of an antibody substantially affect binding, so we only allow mutations in these regions of the sequence when optimizing for binding. In Fig, 5b, we see that after 100 steps, greedy methods and methods informed by an antibody prior optimize antibodies more efficiently than previous methods against these oracles; in particular, CloneBO outperforms all baselines. App. C.3 we plot these results against and table form.
Comparison to structure-base based design model for binding SARS CoV. In App. C.4 we show that CLoneBO can also efficiently optimize antibodies in silico for SARS CoV binding as measured by predictor trained on Co (Raybould al., 2020). In particular we see CloneBO beats structure-based design method DiffAb (Luo et al., 2022).
A Ablations and sensitivity In App. C.5 we show that CloneBO accelerates optimization by build- ing an accurate posterior the performance of CO is harmed when we ablate sampling large clonal families, conditioning on experimental data, or our twisted SMC sampling strategy. We also
Note that some of these methods are developed for different regimens, such as short sequences or large amounts of training data. Their performance here does not necessarily represent their performance for the problems they optimized for.

---

Published as a conference paper at ICLR 2025
Binding
Stability
Binding
Stability
1.0
1.0
Cl
CloneBO LaMBO
P[Synth)
P(Synth)
0.8
0.8
LaMBO
0.6
0.6
Previous
Previous
LaMBO
CloneBO
LaMBO
CloneBO
.0.2
45
50
55
60
65
70
75
Normalized
-log:o Ko
Melting Temp.
(a)
(b)
Figure 6: CloneBO efficiently optimizes antibodies in vitro. LaMBO in this plot refers to LaMBO- Ab. (a) CloneBO design sequences predicted to be synthesizable. (b) CloneBO designs strong and stable binders in the lab. Measurements from previous rounds are shown a histogram. The vertical black line represents the best value previously achieved.
perform two other ablations demonstrating that our results above are reliable we show 1) CIoneBO is robust to different starting sequences and starting pool sizes, and 2) CloneBO can efficiently op- timize antibodies when f deviates from the CloneBO prior, especially at low N. Finally we sweep hyperparameters to show that CloneBO is not particularly sensitive to hyperparameters other than the amount of noise in the data; we describe heuristic that allows us to make a good choice for the amount of noise.
7.2 OPTIMIZING AN ANTIBODY in vitro
We now demonstrate that CloneBO can design sequences as part of real-world antibody optimiza- tion campaign. We started with 1000 lab measurements of binding and melting temperature" (visual- ized in Fig. 7) and designed 200 sequences using CloneBO and our strongest baseline, LaMBO-Ab, for one wetlab iteration of optimizing binding or melting temperature. In real world optimization campaign, this one step would be repeated many times.
Before measuring designed sequences, sequences need be synthesized; sequences which are atyp- ical can fail to synthesize, making their measurement impossible. In Fig. 6a we plot the predicted synthesizability of sequences from CIoneBO and LaMBO-Ab: sequences from CloneBO are signif- icantly more synthesizable (Mann- MhinneWy < le 5), suggesting they are more realistic. We next measure 20 sequences designed by CloneBO and LaMBO-Ab that are predicted synthesize.
In Fig. 6b plot sequences were able to measure; include any sequences that were proposed that we had measured in previous experiments. We see that sequences from CloneBO achieve the best binding and stability. Our strongest binder is only beaten by 2 997 previously measured sequences and our most stable sequence beats the previously measured sequences by a large margin. We also conclude that sequences from CloneBO significantly stronger binders than those from LamBO-Ab (Mann- Mann-Whitney Whitney p = 0.021). We discuss these results in more detail in App. C.6.
CONCLUSION
To develop new disease treatments, antibodies must be optimized for range of properties. By learn- ing from the human immune system's 's approach to antibody maturation and therefore substantially accelerating optimization in the lab, CloneBO can help build safer and more effective therapeutics.
important direction of future work addressing theoretical and practical limitations of CloneBO. First, CloneBO currently assumes simple relationship between the fitness of a clonal family and measurements in the lab (Section 6).. Future work may account for hetcroskedasticity or nonlinear relationships. As well, CloneBO evaluates the fitness of sequence by assessing how likely it is to belong to a clonal family of Xy- Future work may attempt to incorporate patterns learned from measurements of diverse sequences which unlikely to belong to the same clonal family. Finally, the cost of sampling from the CLoneBO posterior seales with the number of laboratory measurements N (Section 6.3), CIoneBO scales by conditioning only subset of measured sequences. Future work could instead build a more scalable model or approximate sampling procedure.
Another important future direction extending CloneBO multi-objective iterative Bayesian op- timization of antibodies for binding and stability simultaneously. We can do so for example by swapping Thompson sampling for other acquisition functions (Daulton et al., 2020).
/94 1000 sequences were measured for melting temperature and 997/ 1000 were measured for Kp-
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We include weights for our CloneBO models and code for implementing sampling in our code release. We describe how to train a CloneBO model, including the parameters that were used to build the training data, in the appendix. Sequences from the iterative optimization experiment are proprietary; we release all other data: we include the trained oracle for Fig. 5a in our code release; we describe how implement baselines and the oracle in Fig. 12 in the appendix.
ACKNOWLEDGEMENTS
This work is supported by NSF CAREER IIS-2145492, NSF CDS&E-MSS 2134216, NSF HDR- 2118310, Capital One, and an Amazon Research Award.
REFERENCES
Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy J Colwell. Model-based reinforcement learning for biological sequence design. Int Conf Learn Represent, April 2020.
David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 Proceedings of Machine Learning Research, pp. 773-782 PMLR, 2019.
Deborah L Burnett, David B Langley, Peter Schofield, Jana R Hermes, Tyani Chan, Jennifer Jackson, Katherine Bourne, Joanne H Reed, Kate Patterson, Benjamin T Porebski, Robert Brink, Daniel Christ, and Christopher C Goodnow. Germinal center antibody mutation trajectories are determined by rapid self/foreign diserimination. Science, 360(6385)
April 2018.
Paul Carter and Greg Lazar. Next generation antibody drugs: pursuit of the high--innnngg fruit'.
Nature Reviews Drug Discovery, 17(3):197-223, 2018.
Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective bayesian optimization. In Proceedings of the 34th In- ternational Conference on Neural Information Processing Systems, number Article 826 in NIPS '20, pp. 9851-9864, Red Hook, NY, USA, December 2020. Curran Associates Inc.
Jared I. Dopp and Nigel Reuel. Simple, functional, inexpensive cell extract for vitro prototyping of proteins with disulfide bonds. Biochemical Engineering Journal, 164: 164:107790, 2020.
Fabian Falck, Ziyu Wang, and Christopher C Holmes. Are large language models bayesian? mar- tingale perspective on In-Context learning. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, April 2024.
Clara Fannjiang, Stephen Bates, Anastasios N Angelopoulos, Jennifer Listgarten, and Michael I Jordan. Conformal prediction for the design problem. February 2022.
Edwin Fong, Chris Holmes, and Stephen G Walker. Martingale posterior distributions. J. R. Stat. February Soc. Series Stat. Methodol., 85(5):1357-139) Februrrr 2024.
Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K Min, Kelly Brock, Yarin Gal, and Debora S Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 599(7883):91-95, November 2021.
Peter Frazier. A tutorial bayesian optimization. July 2018.
Nate Gruver, Samuel Don Stanton, Nathan Frey, Tim G Rudner, Isidro Hotzel, Julien Lafrance- Vanasse, Arvind Rajpal, Kyunghyun Cho. and Andrew Gordon Wilson. Protein design with guided discrete diffusion. In Thirty-seventh Conference on Neural Information Processing Sys- tems, November 2023.
11

---

Published as a conference paper at ICLR 2025
N Hansen and A Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evol. Comput., 9(2):159 195, 2001.
E Hewitt and L J Savage. Symmetric measures cartesian products. Trans. Amer. Math. Soc.. 1955.
Brian L Hie, Varun R Shanker, Duo Xu, Theodora U Bruun, Payton A Weidenbacher, Shaogeng Tang, Wesley Wu, John Pak, and Peter Kim. Efficient evolution of human antibodies from general protein language models. Nat. Biotechnol., 42(2):275-22) 283, April 2023.
Alexander Jarasch, Hans Koll, Joerg T Regula, Martin Bader, Apollon Papadimitriou, and Hubert Kettenberger. Developability assessment during the selection of novel therapeutic antibodies. Journal of pharmaceutical sciences, 104(6):1885-188)) 1898, 2015.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7B. October 2023.
Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. Iterative refinement graph neural network for antibody sequence-structiee co-design. In International Conference of Learn- ing Representations 2022, October 2021.
Nal Kalchbrenner, Espeholt, K Simonyan, Aäron van den Oord, Alex Graves, and Kavukcuoglu. Neural machine translation in linear time. arXiv.org, 2016.
Xiangzhe Kong, Wenbing Huang, and Yang Liu. End-to-end full-atom antibody design. In Proceed- ings of the 40th International Conference Machine Learning, January 2023.
Dieterich Lawson, Michael Li, and Scott Linderman. NAS-X: Neural adaptive smoothing via twisting. In Thirty-seventh Conference on Neural Information Processing Systems, November 2023.
Dieterich Lawson, Allan Raventós, Andrew Warrington, and Scott Linderman. SIXO: smoothing inference with twisted objectives. In Proceedings of the 36th International Conference on Neural Information Processing Systems, number Article 2815 in NIPS '22, pp. 38844-38858, Red Hook, NY, USA, April 2024. Curran Associates Inc.
Hyungi Lee, Eunggu Yun, Giung Nam, Edwin Fong, and Juho Lee. Martingale posterior neural processes. In The Eleventh International Conference on Learning Representations, September 2022.
Ruei-Min Lu, Yu-Chyi Hwang, I-Ju Liu, Chi-Chiu Lee, Han-Zen Tsai, Hsin-Jung Li, and Han- Chung Wu. Development of therapeutic antibodies for the treatment of diseases. J. Biomed. Sci., 27(1):1, January 2020.
Shitong Luo, Yufeng Su, Xingang Peng. Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen-specific antibody design and optimization with diffusion based generative models for protein structures. In Advances in Neural Information Processing Systems 35. Cold Spring Harbor Laboratory. July 2022.
John R Mascola and Barton F Haynes. HIV-1 neutralizing antibodies: understanding nature's path- ways. Immunol. Rev., 254(1):225 244, July 2013.
Jeffrey W Miller. A detailed treatment of doob's theorem. arXiv: Statistics Theory, 2018.
Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena-Hurtado. Aidan Gomez, Debora Marks, and Yarin Gal. Tranception: Protein fitness prediction with autoregressive transformers and inference-time retrieval. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepes- vari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 17990-17017 PMLR, 2022.
12

---

Published as a conference paper at ICLR 2025
Tobias H. Olsen, Fergus Boyles, and Charlotte M Deane. Observed antibody space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences. Protein Sci., 31(1):1441-44, January 2022a.
Tobias H Olsen, lain H Moal, and Charlotte M Deane. AbLang: an antibody language model for completing antibody sequences. Bioinform Adv, 2()):bb,, June 2022b.
Tobias H Olsen, Brennan Abanades, lain H Moal, and Charlotte M Deane. KA-Search, method for rapid and exhaustive sequence identity search of known antibodies. Sci. Rep., 13(1):1-11, July 2023.
Ganesh E Phad, Dora Pinto, Mathilde Foglierini, Murodzhon Akhmedov, Riccardo L Rossi, Emilia Malvicini, Antonino Cassotta, Chiara Silacci Fregni, Ludovica Bruno, Federica Sallusto, and Antonio Lanzavecchia. Clonal structure, stability and dynamies of human memory B cells and circulating plasmablasts. Nat. Immunol., 23(7):1076-1088 June 2022.
David Prihoda, Jad Maamary, Andrew Waight, Veronica Juan, Laurence Fayadat-Dilman Daniel Svozil, and Danny A Bitton. BioPhi: A platform for antibody design, humanization, and human- evaluation based on natural antibody repertoires and deep learning. MAbs, 14(1):2020203 2022.
Jacob T Rapp, Bennett Bremer, and Philiip A Romero. Self-driving laboratories to autonomously navigate the protein fitness landscape. Nature Chemical Engineering, 1(1):97-107, January 2024.
Matthew IJ Raybould, Aleksandr Kovaltsuk, Claire Marks, and Charlotte M Deane. CoV-AbDab: the coronavirus antibody database. Bioinformatics, 37(5):734-735, August 2020.
Eve Richardson, Jacob Galson, Paul Kellam, Dominie F Kelly, Sarah Smith, Anne Palser, Simon Watson, and Charlotte M Deane. A computational method for immune repertoire mining that identifies novel binders from different clonotypes, demonstrated by identifying anti-pertussis toxoid antibodies. MAbs, 13(1):1869406, 2021.
Adam J Riesselman, John B Ingraham, and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nat. Methods, 15(10):816-22) October 2018.
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling. Foundations and Trends Machine Learning. 11(1):1-96, 2018
Guy Sella and Aaron E Hirsh. The application of statistical physics to evolutionary biology. Proc. Nail. Acad. Sci. U. A., 102(27):9541-944) July 2005.
Jung-Eun Shin, Adam Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C Kruse, and Debora S Marks. Protein design and variant prediction using autoregressive generative models. April 2021.
Richard W Shuai, Jeffrey A Ruffolo, and Jeffrey Gray. IgLM: Infilling language modeling for antibody sequence design. Cell Syst., 14(1 (1:199999-99e. e4, November 2023.
Sam Sinai, Richard Wang. Alexander Whatley, Stewart Slocum, Elina Locane, and Eric Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint, 2020.
Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Green- side, and Andrew Gordon Wilson. Accelerating Bayesian optimization for biological sequence design with denoising autoencoders. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 20459- 20478. PMLR, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull. David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom,
13

---

Published as a conference paper at ICLR 2025
Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Brian L Trippe, Luhuan Wu, Christian A Naesseth, David Blei, and John Patrick Cunningham. Practical and asymptotically exact conditional sampling in diffusion models. July 2023.
Timothy Fei Truong, Jr and Tristan Bepler. PoET: A generative model of protein families as sequences-of-sequee sequences. In Thirty-eevnnth Conference on Neural Information Processing Sys- tems, November 2023.
Kaixuan Wang, Xihao Hu, and Jian Zhang. Fast clonal family inference from large-scale cell repertoire sequencing data. Cell Rep Methods, ((10):10000) October 2023.
Eli Weinstein, Alan Amin, Jonathan Frazer, and Debora S Marks. Non-identifiability and the blessings of misspecification in models of molecular fitness and phylogeny. Adv. Neural Inf. Process. Syst., December 2022.
Nick Whiteley and Anthony Lee. Twisted particle filters. aos, 42(1):115-141, February 2014.
Kevin K Yang, Zachary Wu, and Frances H Amold. Machine- leaininggnnnneeedd directed evolution for protein engineering. Nat. Methods, 16(8):687-694 August 2019.
Kevin K. Yang, Alex Xijie Lu, and Nicolo Fusi. Convolutions are competitive with transformers for protein sequence pretraining. In ICLR2022 Machine Learning for Drug Discovery, 2022.
Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language models via twisted sequential monte carlo. In Proceedings of the 41st International Conference on Machine Learning, April 2024.
A DETAILS OF CLONELM AND CLONEBO
A. DATA COLLECTION AND TRAINING CLONELM
Clonal family data We downloaded all data units of human single chain data on OAS (Olsen et al., 2022a). For both light and heavy chain data, we put 10% of these units into test set, 10% into a validation set, and 80% into a train set. We annotated clonal families in each these data units with FastBCR (Wang et al., 2023) using the default parameters cl cluster_thre overlap_thre
3, 0 8. We removed any clonal families with fewer than 25 sequences. We ended up with 731 thousand heavy chain clonal families for training, 81 thousand for validation, and 96 thousand for testing; and 26 thousand light chain clonal familics for training, 4 thousand for validation, and 4 thousand for testing.
0. 1, consensus three
Training CloneLM We trained CloneL.M using Mistral 7B as our base architecture (Jiang et al., 2023). We scale down the model size to 24 layers of attention blocks with 16 attention heads and a hidden size of 1024 across embedding and all intermediate hidden states. We set our maximum context size 2048. We end up with Mistral model containing 377 million parameters. For training, we use a batch size of 2, gradient accumulation step of 4, and we sweep learning rates over {0.0005, 0.00025, 0.0001} using constant learning rate scheduler. All training is performed on NVIDIA A 1-00-XXM480000 GPUs. For human light chain data, we train for 24 hours for 40 epochs. For human heavy chain data, we train for 48 hours for epoch.
14

---

Published as a conference paper at ICLR 2025
A.2 TWISTED SEQUENTIAL MONTE CARLO
Twisted distributions We approximate the marginal PM+1 (X (X Xo.as) 1:
just as Section 6.2
by replacing F in Eqn. with FINN Futh:,++1, (M11) E F 1,(1)
+ F.
(1) PM+1 M41 Xo. M. M41 (o0pX FFN M+L1))
(1)
We call this approximation the "twisted" distribution.
If we pretend that these twisted distributions are exact marginals and that par is also the marginal of Pas-1, we can therefore sample each sequence letter-by-leter according to
((+1) ((++1) M+1 PM++1 (x (= M Xo.M) p(X (1+1) (cf)
M+1 M+ Xo.M (oo.)))(Yinn PiNN (1N)))))))) (5) x
The first term in Eqn. 5 samples the next letter according to the unconditional distribution. The 4,M+1 for which FM++1((++1) correlates with Y1:N;; F
second term is bias that upweights letters x x((+1)
this usually means upweighting letters that are more likely if sequences that were measured to have high Y,, X,, were included in the clonal family.
Importance sampling The twisted distributions are not exactly the marginals. We can correct for this discrepancy with sequential weighted importance sampling. Say we have Xo:ar, x-f)) approximately sampled from ((:)) (Xo.M, 1441) with importance weight w++1(( Then we can calculate the importance weight of Xo:Ms x(++1) (f+1)
M+11
M++ by multiplying by the ratio between M+ Xo.M (aM)) (Xo:M, X 14.
1(++1) (Xo.M, x(++D) and (xx
PM
M+1
w (+1
p(X
(I+1) (1+1)
(:f)
Xoar)
x P(YENIPENN (MMALA4
M+1 M411
M+160
M+1 M+1 Xo.)
(1+1) (i)
p PYYINN/FEN (+10))
PM--
00 with weights WID then we can approximately sample from PM+1 by sampling x 0.M+1 with probability in M+1 wy
Therefore if we have D samples XIID
M+1
Ex
Sequential Monte Carlo Say we are iteratively sampling and weighting D sets of sequences and M,(:.) becomes much smaller than that the others. Ideally wouldn't waste any more compute sampling the rest of the sequence. This is the idea of sequential Monte Carlo - while generating each set of sequence letter-by-letter, every so often, we calculate resample the sets of sequences with probabilities W1.D (M.(::) To decide when to resample,
the importance weight for one set w w.
the essential sample size (ū M.(())) and resample when it goes below vD, a classic heuristic. After we resample, we reset the weights wi 1/D. As D do expect approximate pM++ arbitrarily well.
We also note when using the predictive distributions of CloneLM, Eqn. 4 is approximation. The discrepancy comes from the fact that FM++1 the conditional probability of X,, the M 2nd sequence while E = Σ, 1 M+1()) + F.M is the probability of X,, as the M + 1st sequence. For p. these probabilities are identical, but this may not be exactly the case for CloneLM. Therefore, once we have sampled all the letters in a sequence XM then we have sampled from a distribution proportional
p(Xo:s )p YIN FM
Therefore also resample this stage after multiplying the importance weight of sample d, wx by
p(Y P. p(Y.N FM F
A.3 EXPERIMENTAL DETAILS OF CLONEBO
Before conditional generation, we normalized YI.N to to Y, = (Y, (aattmean//starttiee where startmean and startsid the mean and standard deviation of the initial dataset Yı.N. In our
15

---

Published as a conference paper at ICLR 2025
experiments used D 4 during twisted SMC, and generated clones of size M 6. We tempered by the maximum number of sequences we conditioned on, i.e. we used // 75 where ō 0.25. We run each experiment on single NVIDIA A100 GPU with 80GB memory; 100 steps with CloneBO takes roughly 10 hours.
B EXPERIMENTAL DETAILS
B. BASELINES
We implemented Sapiens using the code in https://gi wow/Merck/ Sapi ens under the MIT licence. We suggested mutations to a sequence by taking the highest likelihood mutation suggested by
scores that had not previously been measured.
We implemented LaMBO using the code in https: / /github.t... com/ samue lateelstannt staston////ooooo lambo under the Apache-2.0 licence. We used default hyperparameters for the masked language model version of LaMBO. To restrict mutations to the CDRs, we kept sampling mutations from LaMBO until only the CDR was modified.
To build LaMBO-Ab we pretrained the latent space of a LaMBO MLM model on training set of 100000 antibody sequences. We built the training set by taking one sequence from each of 100000 random clonal families from the CloneLM training set.
We also compare to two other genetic algorithms with trained surrogates, Adal.ead and Genetic; a NN ensemble Bayesopt method, "Eyo-BO"; an evolution method, CMA-ES (Hansen & Ostermeier, 2001); an RL method, Dyna-PPO (Angermueller et al., 2020); and an adaptive sampling method, CbAS (Brookes et al., 2019). The first three methods are described in Sinai et al. (2020). We implemented these methods with code from FLEXS (Sinai et al., 2020) using the code from https: //github. com/ sass i naii / FLEXS ree wwweeeeeeeeeeeeeeeeeeeeeeeeeeeee master under Apache-2. 2.0 license; used default settings for all methods.
B.2 TRAINING ORACLES AND INITIALIZING OPTIMIZATION
Oracle of fitness function of clonal family We trained an oracle language model adapted from Llama2 (Touvron et al., 2023) a single reference human heavy chain clone. There are in total 10013 sequences in the clone and we split them into 90% train, 5% validation, and 5% test sets. Due to the scarcity of our data, downscaled a Llama2 model with billion parameters by keeping 12 hidden layers with hidden state size of 512. We used 4 attention heads, key-value heads, and kept the context size 2048. We ended up with language model containing 50 million parameters. We trained our model on a single NVIDIA A A100-SXM4-80GB GPU for 10 epochs with batch size of 32, a gradient accumulation step of 2, a learning rate of 0.0005. We obtained perplexity value of 1.2634 on the validation set. Using the clone oracle, we start optimization with 2 randomly chosen sequences from the test set X1 and predictions from the oracle Y1.2-
Oracles from lab measurements of therapeutic Antibodies We were provided temperature and binding measurements of 6880 sequences from an iterative optimization experiment performed in reference antibody sequence and trained ensembles of 10
the lab. We aligned these sequences CARP/Bytenet models (Kalchbrenner et al., 2016; Yang et al., 2022) on the one hot encodings of the aligned sequences to fit the temperature and binding data. The K, model ensemble had measured predicted spearman correlation in crossvalidation of 0.95, while the the Tm model ensemble had an spearman correlation of 0.72. We use the mean prediction of the ensemble as f. We start optimization given the measurements of the first 1000 sequences of this experiment X1:1000- and predictions from the oracle Y1.1000-
16

---

Published as a conference paper at ICLR 2025
B.3 TRAINING DATA FROM ITERATIVE OPTIMIZATION
Siiiin Unno
asstance
UMAP1
(a) Sequences with binding data.
(b) Sequences with stability data.
Figure 7: Starting pool for (a) binding and (b) stability optimization. We plot the Hamming distance matrices and UMAPs. sequences are coloured by predicted property
B.4 LAB VALIDATION
We built a predictor of synthesizability from measures expression just the same predictors of melting temperature and binding in Sec B.2. The predictor achieves a test AUROC of 0.87.
Designed sequences were synthesized via cell-free protein synthesis (Dopp & Reuel, 2020) in 96- well format and purified via Protein A binding on Pierce magnetic beads. Purity and yield were confirmed before further analysis. Affinities (-log p)) were measured with Bio-Layer Interfer- ometry (BLI) Octet instrument, with the antigen immobilized at three different dilutions of antibody. Thermostability (melting temperature) was measured by Nano differential scanning inter- ferometry (SanoDSF) Uncle instrument.
C SUPPLEMENTARY RESULTS
C.I MORE EXAMPLE GENERATED CLONAL FAMILIES
In this section we show more light and heavy chain clonal families generated from CloneL.M. In Fig. 8b and Fig.s 9b, 9c, we see that sequences generated by CloneLM include can introduce in- sertions and deletions. The large sets deletions the sequences of the clonal family in Fig. 8b are due to the fact that some sequences in OAS are missing the beginning end of their sequences (Olsen et al., 2022b).
Prompt
Clonal family
070100050 momoor sunne
Sample
Sample
Sample 3
OVGLLOPFEEeS- MI50 www.
Prompt
Clonal family
Sample 1
Sample 2
Sample 3
www.
(b)
Figure 8: Examples of heavy chain clonal families generated by CloneLM.
17

---

Published as a conference paper at ICLR 2025
Prompt DSONTORRRSLAVSS..RR o/w.
Clonal
D19MTgARGaEAVV GRAAT www.geoo www.saagrrsooonnt..........
family
Sample
D29KTO SAA.OOFAAAAAS (AAATTMOKI www.ccaoogggg..a.oooo....
Sample
D
D2VMTOEFA WAA.PAAAAAAA.... SWWATIICKI waraaaagaee
Sample
Prompt
Clonal
DEOHTOSROS www.OOLLASS..... WWAATACKSSS
family
DIVEEgSSIGHLAAA AAA.S//AAAA...AA.......... AAA.S// www.e/ CAR.CCCCECEEEAOOOEETT Airrog www.goeoeeeeeo..o
Sample
Sample
DEWEEQSISSRLAVEE
30 RESH158 Airoggo BO www.Tii.YYS..ooo.. DOw. wOw. CES DS orr.rrr R/DIX
Sample
Prompt
ETGGGIEVEIR
Clonal
20000 LIFUCCIEVEIE
family
Sample
RIVN
R1 1ww.0//A7A4522000801111888 oww
Sample
AITOGGIRVEIS
Sample
RI1WNOS wwt.oo/SwTTLLLSSRREAATTL.... www.://wwwww....oooooooooo
GUPSGOERVETR
Figure 9: Examples of light chain clonal families generated by CloneLM.
C.2 TWISTED SMC FITTING AFFINITY DATA
In Fig. 10 we show similar results to Fig. 4c for 75 laboratory measurements of binding.
IS
tSMC
LSMC:
IS: 30
IS: 300
(R² 0.51)
(R2 0.61)
LSMC: 4
IS: 100
IS: 1000
Norm. -o
250
X
200
-1
150
log
-2
100
-45
-35
-25
-30
-20
5
6
7
FK
FM
M
(a)
(b)
Figure 10: Twisted SMC fits laboratory measurements of affinity. Experiments are similar to those in Fig. 4c.
18

---

Published as a conference paper at ICLR 2025
C.3
EFFICIENT OPTIMIZATION VERSUS NUMBER OF STEPS
Binding
Stability
74
Norm. 1og10
0.4
Melting Temp.
Greedy
0.35
72
Sapiens
-00
LaMBO
0.3
LaMBO-Ab
68
Cione80
20
100
100
Steps
Steps
Figure 11: Results of Fig. 5b for various for representative baselines.
Method
logp( X (|clone)
Norm. logio KD
Melting temp.
CbAS
-27.42
0.2572
66.76
Dyna-PPO
-27.42
0.2572
66.76
CMA-ES
-27.42
0.2853
66.76
EvoBO
-27.42
0.2572
66.76
Genetic
-25.94
0.3001
70.19
AdaLead
-25.30
0.2998
69.42
Greedy
-26.65
0.2986
69.28
LaMBO
-24.99
0.3085
69.40
Sapiens
-24.51
0,2875
70.39
LaMBO-Ab
-24.51
0.3381
70.44
CloneBO
-7.29
0.3713
72.92
Table I: Result from Fig. 5 in table form. We report the mean best value achieved across 10 repli- cates.
C.4 OPTIMIZING ANTIBODIES TO BIND SARS COV
Above, validated CloneBO using predictors trained large-scale mutational data from lab as well as in vitro experiments - these are the most reliable evaluations of CloneBO. As another eval- uation of CloneBO, and to compare it structure-based design models, here explore optimizing the CDRH3s of antibodies for binding SARS CoV I and in humans as measured by predictors used inJin al. (2021) (downloaded from https: / tub. com/ wengong jin/ Re i neGNN). We caution however that while we expect CIoneBO to be a good prior for SARS binding, these predictors have large epistemic uncertainty as they are trained on only few thousand extremely diverse sequences. We thus expect we are optimizing an objective similar to those in Fig. 14 so that CloneBO should perform best at low N.
These predictors were trained on only functional antibodies and may not generalize outside this set. So, we optimize binding humanness (measured by IgLM likelihood); we standardize both binding and humanness to the same variance.
We start with starting sequences, CloneBO is consistently among the most efficient methods for CoVI and, at low N, for CoV2 as well.
sequence from CoVAbDab. Fig. 12 we see that for randomly selected
We were interested in seeing if structure-based design methods could also be used for iterative design. When structure is available, we therefore also compare to a state-offthe--a structure method, DiffAb (Luo et al., 2022). To perform iterative design with DiffAb, greedily pick one of the best measured sequences and optimize as in Sec 4.3 Luo et al. (2022). We see that CIoneBO beats this algorithm.
19

---

Published as a conference paper at ICLR 2025
Greedy
Sapiens
LaMBO
LaMBO-Ab
DirTAb
CloneBO
SARS CoV1
SARS CoV2
A binding +humanness
0.9
0.5
0.4
0.8
0.9
1.6
100
100
100
100
0
100
100
Steps
Figure 12: CIoneBO efficiently optimizes sequences from CovAbDab for binding to SARS CoV1 (first 3 columns) CoV2 (last columns). In particular it outperforms seueture-basee bascline (DiffAb) for the sequences with an available structure. We show mean and standard deviation achieved across 3 replicates.
C.5 ABLATIONS in silico OPTIMIZATION
CloneBO efficiently optimizes sequences by building an accurate posterior We now show that CloneBO efficiently optimizes sequences by conditioning on experimental measurements and accurately sampling from martingale posterior. In Fig. 13 we see optimization is often harmed by I) not accurately sampling from the martingale posterior by only sampling clones of size M 1, 2) using a naive importance sampling procedure instead of twisted SMC, or 3) not conditioning on previous measurements. We the M = 1 or importance sampling ablations have less of an effect when optimizing fitness, potentially because it is easier to condition on data from a function from the CloneBO prior.
Binding
Stability
Fitness
Norm. -logno Ko
Melting Temp mmm
74
R X ccloee
0.4
-6
M=1
0.35
72
-12
(D 300)
70
0.3
No cond.
0.25
68
CloneBO
66
-30
100
Steps
Steps
Steps
Figure 13: CloneBO better optimizes antibodies than models that ablate accurately sampling from the posterior. We shade standard deviation across 3 replicates. We show results for optimiz- ing binding in silico, stability in silico, and the fitness function of a clone.
CloneBO is robust to different starting pool sizes and deviations from its prior at low N. In Fig. 14a we optimized for stability as Fig. 5b of the paper with a starting pool of various N mea- surements. We see that CloneBO outperforms our baselines across these different starting stabilities and starting pool sizes, especially low N.
In Fig. 5a in the paper we showed that when optimizing an objective from CloneBO's prior (the fitness function of clone), F, CloneBO strongly outperforms baselines. Here we mix F with random neural network, G, and optimize wF + (1 w) w)G; E [0 controls how well the CloneBO prior describes the objective. We start with = sequences. this small N setting, CloneBO outperforms other methods matches the prior (w = 0.4). Even at w 0.2, CloneBO is the best method at very low (up to N=25).
Fig. 14b we see that in when the objective only somewhat
20

---

Published as a conference paper at ICLR 2025
Greedy
Sapiens
LaMBO
LaMBO-Ab
CIoneBO
Melting Temp.
N=30
N=100
N-300
N-1000
N 3030
75
72
(a)
69
66
0
100
100 0
100
100
100
Steps
w= 0.2
w = 0.4
0.6
0.8
1.0
Relative Improvement
(b)
100
100
100 0
100
100
Steps
Figure 14: CloneBO is robust to ablating (a) the starting pool size or (b) how well its prior de- scribes the objective. show the mean and standard deviations the best achieved value across replicates.
Sensitivity to hyperparameters in silico We now investigate the sensitivity of our results to hyperparameters: the noise level in the likelihood ö, the maximum number of allowed mutations L, and the number of sequences we draw Xo from K, as described in Sec. 6.4. In our experiments above log,
L 3, and K = 4.
We picked and based on intuition. We picked ē by looking at the clones generated when con- ditioned the initial binding data in silico when suing values log, -4, -3, -2, -1; we noted that when log, = -4, -3, generated clones contained very short or long sequences, indicating it hard to find clones that fit the data with little noise. Thus we picked log, data with as little noise as possible. Performing the same procedure for the in silico stability data, we arrived at the same value log, experiments.
-2 fit the
2 was good choice. We fixed this value for all our other
We see in Fig. 15a that K performs badly when optimizing binding, likely due to higher chance of getting stuck in local minima; otherwise CloneBO is not very sensitive to K. In Fig. 15b we see CloneBO is also not very sensitive to L. Fig. 15c we note CloneBO sensitive to the choice of ō but procedure described above picked the optimal ā for binding and stability. For fitness, we noted the posterior is easy to sample from even at log, ā = -3 (we see this manifest in Fig. 13 well) and indeed smaller value of ō is optimal. This suggests one optimize antibodics more efficiently by improving the CIoneBO likelihood, potentially by picking a better or by adding prior and marginalizing over it.
21

---

Published as a conference paper at ICLR 2025
log p(xcco
-6
-6
-12
-12
-18
-18
-24
-24
244
-30
-30
16
5
-3
-2
-1
K
log2 0
Norm. -log Kp
0.4
0.4
0.4
0.35
8
0.35
0.35
0.3
0.3
0.3
0.25
0.25
0.25
16
5
-3
-2
-1
K
log2
Melting Temp.
74
74
74
72
72
72
70
70
70
68
68
68
66
66
5 66
1
16
-3
-2
-1
K
log2
(a)
(b)
(c)
Figure 15: CloneBO sensitivity to hyperparameters for fitness, binding and stability. Experi- ments are run with 10 replicates; blue boxes represent hyperparameters used in the main text a) Sensitivity to size of pool X₂ randomly selected from, K. b) Sensitivity to maximum allowed number mutations, c) sensitivity to noise in likelihood, Ø.
C.6 ADDITIONAL DISCUSSION OF vitro RESULTS
We were able to measure the affinities of 9 sequences for CloneBO, and 11 for LaMBO-Ab, and the melting temperatures 19 sequences from CIoneBO and sequences from LaMBO-Ab. Adding in previously measured sequences, were able to get more affinity measurements for CloneBO and 2 more for LaMBO-Ab, and no other affinity measurements. Note when interpreting these results that affinity and stability may be correlated with dropout and whether not sequence was previously measured. Fig. 16 we plot the affinity measurements removing the previously measured sequences; the results qualitatively similar those of Fig. 6.
Binding
CloneBO
LaMBO
Previous
-0.2
0.2
0.8
Normalized
log Kp
Figure 16: The result of Fig. 6b for affinity with only sequences that were newly measured.
22

---

Published as a conference paper at ICLR 2025
D THEORETICAL RESULTS
D.1 ANALYTIC FORM OF MARGINAL LIKELIHOOD
Proposition D.1. (Proof of Prop. 6.1) For some constant C,
Dog((YLN/FiNN) - log Cov(Fi:N) R2 log (R) C"
Proof. We first put wide prior on M, M N(0, 72). and then later send T + oC. Then we can marginalize M out to get
Yn
where we define be the vector with 1/vN in each position, e e= I/VNN Calling E o²I + n @ e, o3 - F M4 of of 2 -YIN. we get
OFIN V F
FIN d3
=22 P(N > >)
2o 2 3/0 X0
49
Now,
M =f
=0 (1- + c
tF 2( aST
1 N
1+)
=No
Therefore, as - 00, FT EEN--I No times the variation of FLN, Var(Fi:v), and E is No times the covariance of FI:v and YI.N, Cov(Fi:N, Y..)).
Therefore, if we call Ha -Noo--Con Cor((Fi. YI. SSt((Y))) = R then we get that when we send - + 00,
pogg(YY::/ FINN) log Cov(Fi.N) R² + + ( C"
for some constant C..
D.2 CONVERGENCE OF APPROXIMATE POSTERIOR
We show that the approximate posteriors converge defined in Eqn. 3 converge to the true posterior. We make the mild assumption that the hypothetical latent variable clone has been defined such that clone +> p(X (clone) is measurable and no two variables clonej, clonez have the same conditional distribution p(X clone,) * p(X (cloneg)) We also assume that all measured sequences are antibod- ies that can plausibly, although maybe with extremely low likelihood, appear in a clone together, i.c. p(.N) * O. Finally, we assume the sequences X are sufficiently diverse so that their log likelihoods cannot all be identical for any clone, i.c. for some O, Cov(Fi.N) e with probability 1 under (clone]Xo)*
Proposition D.2. (Proof of Prop. 6.2) Assume clone + p( []clone) is measurable and injective, p(Xo, X1.N) F O, and Cov(Fi:N) > c or some > 0 with probability 1 under p(clone| Then, as M + ce, the approximate and true posteriors converge in total variation
1 (X1:M NI., & Xo) - p(X1.M YI.N Xo)xx - 0.
'We need this assumption to ensure the density p(Yi.N FMx is bounded above. Altemntively, one assume proper prior M by picking large but finite in the proof of Prop. D.I.
23

---

Published as a conference paper at ICLR 2025
Proof. By our assumptions, by Doob's theorem (Miller, 2018), if clone p(clone(Xx) and Xm p(X]clone) iid, with probability 1,
p(X,|XI:M) p(X.|Xx:ar) ((X./)
for each n. In particular, we get that FiN + FIEN and therefore, since, assuming Cov(Fi.N) > P(YIN FM E:N is a bounded function of FI:N.
Ex clonex-x(op [ (p) (FEN) |( + 0.
Now we show that the the normalizing constants of the approximate and true posteriors converge.
ZM J | (dx))) Xo))
M
PFYI:N/FMN I:N)
II de( (oo)
m==1
=
Fi:N) (6)
- (cboeeeppcccammex
Xo) =: Z
By our assumption that p(Xo, XI:N) * O, for a of clone of probability greater than 0 under p(clone| (o)), (clone) > 0 all vı; for this set, p(Yi.)] FI.N) > 0 and therefore >
Now we show that the approximate and true posteriors converge in total variation:
P(YnN
1.N ( (
(Xo)
p |c(Xe_//coone)p((X. FI:N )ap[clopee (o))
ZM
Z
XI.
I (((MM Xo) 2 z-1
XT.
p(X (p(Yi.N) p(Y..) (Fin)) dp(clone|
7
X
Z
Zaal
1
Eptctone| X ( P(Yi:N 0.
2
Z
24


Reviews:
[{"review_id": "LndvJb15je", "reviewer": "Reviewer_Bdzt", "summary": "This paper treats antibody design from pure sequence view, using clonal family to guide the model. The paper shows CloneBO can optimize antibody sequences better than former methods. Outstandingly, some experiments in vitro also support the effectiveness, which is often ignored in similar works.", "questions": "1. Does the optimization process in Fig. 3a converge? It seems that fitness is still rising in the end.\nSee weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The idea of applying martingale in antibody optimization is novel, and surprisingly fits the nature of evloving process of antibody.\n2. Wet lab experiment is a highlight.", "weaknesses": "1. The experiment seems weak. Many influential works in recent years are not taken into comparison, such as[1][2]. They authors claim that structure based de novo design cannot make use of previous measurements and must have access to structure so they are not suitable for this task, which I highly suspect. Antibody is a specific protein type that highly reliable on its structure to perform. Even though stucture data are scarse, authors should demonstrate CloneBO’s supriority by showing that using massive sequence data can lead to higher performance only using sequence.\n2. The representation is vague. Without any specific table, it’s hard to comprehend directly. This could also be a potential problem for following works to follow.\n\n[1]Kong X, Huang W, Liu Y. End-to-end full-atom antibody design[J]. arXiv preprint arXiv:2302.00203, 2023.\n\n[2]Lin H, Wu L, Yufei H, et al. GeoAB: Towards realistic antibody design and reliable affinity maturation. ICML2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CsyyoxL0cd", "reviewer": "Reviewer_9Ldu", "summary": "In this paper authors propose CloneBO - a procedure which aims at streamlining the antibody sequence optimisation. The approach relies on a language model which training is heavily inspired by the evolutionary mechanisms present in the immune system and allows for iterative guidance of generation by taking into the account previously measured samples. Authors validate their approach both in silico and in vitro demonstrating significant improvements over other methods.", "questions": "Training of CloneLM relies on a data pre-processing step (grouping of sequences into clonal families) done with FastBCR. Since this is a critical initial step I wonder if the authors examined the effects of different preprocessing tools or hyperparameters on downstream performance or at least, given the large resource demands of LM training, the changes in distribution of processed data.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper focuses on antibody sequence optimisation, an important problem of multi-objective nature that is encountered in every drug discovery project. Approaches that improve this drug pipeline stage have a potential of streamlining development of antibody-based therapies by shortening the development process and decreasing its costs.\nThe main original contribution of this paper lies in jointly modelling the clonal families (groups of evolutionarily connected antibody sequences that are computationally inferred from large scale sequencing data). \nAuthors validate their approach on several benchmarks and showcase potential of improving individual relevant traits of antibodies like binding affinity, humanness and thermal stability. Reported results significantly outperform state-of-the-art methods. Improvements are reported both in silico (through better predicted scores of trained oracles) and with in vitro experiments, which significantly strengthens the contribution.", "weaknesses": "Method requires initial, viable sequence i.e. the starting point from which the optimisation begins. This limits the applicability in some design scenarios where such sequence is not known and therefore hinders the impact of the approach compared to parallel lines of research that focus on e.g. structure based binder design and optimisation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Izf3GzrmLi", "reviewer": "Reviewer_gX85", "summary": "The paper introduces a method, termed CloneBO, for optimizing antibody sequences for stability and binding. Their approach uses an LLM to model the distribution of a collection of clonal populations of antibodies, drawing inspiration from the evolutionary trajectories of antibodies in the body to predict beneficial mutations. The authors introduce several ideas to improve sampling quality from their LLM, including sampling from a martingale posterior and constructing a twisted MC scheme that biases towards observed antibodies. Their approach is tested both on in silico benchmarks and in vitro experiments and shows outstanding performance in the design tasks they’ve attempted.", "questions": "* I’m not clear on the exchangeability claim, evolutionary processes have a clear arrow of time, with sequences ordered in a way that can be predicted, unless we are only looking at the final leaves of the tree, which in my understanding is not necessarily true for immune populations. I appreciate a clearer exposition here.  \n* I’m unsure why the authors have chosen to model the light chain and heavy-chains separately. A clear explanation would be helpful. \n* Line 39: “up to thousands of previous iterations” this is somewhat misleading, not all iterations are the same as in many cases these are done in batches (“measurements” are always done in batches), and batch-iterations rarely exceed 10. In silico iterations can reach thousands but the description here is blurry and especially “measurements” is best reserved for real world queries rather than oracle queries.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* Context sharing and placement of the solution: The exposition is strong (on the antibody design problem), code is shared, well-organized and readable. The paper targets an important problem and is lucid about how to get to impact in that domain. \n* Innovation: Multiple interesting and recent ideas have been introduced together for this problem domain (Simulating clonal evolution, Martingale posteriors, combining twisted Montecarlo and LLMs), and the results are convincing in silico and in vitro. \n* Benchmarking: multiple relevant design approaches have been benchmarked, and the method performs well comparatively. \n* In vitro validation: The authors conduct extensive validation, both in silico and in vitro. In vitro validation is rare among ML papers but absolutely crucial in evaluating whether the method has practical use. The authors have done ablation studies on the effects of their sampling scheme and show each individual component helps the performance of the optimizer.", "weaknesses": "* The experimental data are not shared. This is not disqualifying for an ML paper in my view, but should be noted as a weakness. \n* The paper is well structured but somewhat hard to read, partly due to the many ideas that it tries to cover in the main text. I suggest moving some of the mathematical exposition into the SI and describe the intuition more concisely but clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H18vqD2mCb", "reviewer": "Reviewer_mvea", "summary": "The paper proposes a new way to design antibodies using Bayesian optimisation. In contrast to most closely related existing aproach (Lambo) the authors propose a different way to capture the prior antibody distribution. They train a PLM on clonal families, to capture how the imune system diversifies antibodies. They also introduce a twisted sequential Monte Carlo procedure to better incorporate previous lab measurements (sucessful designs). The in silico and in vitro results show that the proposed procedure improves upon Lambo in terms of antibody synthesizeability and stability.", "questions": "When training the ClonalPLM are the clonal families (their order) re-mixed (e.g. for Light chain as it uses many epochs) if not, how is the ordering chosen?\n\nWhat is the edit distance distribution for the in-vitro designs? In the method description (Section 6.4) you state \"iteratively optimize F(X) over the top substitution for up to 3 substitutions\", does this mean that designs can only have maximum edit distance of 3 in your experiments? If yes were the baselines constrained to the same edit distance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I find it quite a unique and nice idea to try to produce local diversification around the seed that matches diversification our body produces during imunization and it's implementation is going a bit of the beaten path with for example the martingale posterior. The experiments are quite thorough and convincing, with a strong baseline. The paper itself is well written and easy to follow.", "weaknesses": "This is not a big weakness, but I'd say that the CloneLM model makes more Fv mutations than one would expect. Also, in appendix the clonal familly does not fix the missing starting gap '-' in some cases, which is not ideal as it would always be a 'Q' for human Ab with the given subsequent amino acids.\n\nThe melting temperature and binding oracle experiment uses VHHs, which I would say is not ideal. While humanized VHHs do look kinda like human VH chain, they are still different, even in trivial ways (e.g. they often have much longer CDR H3). It's nice that the proposed method deals with this distribution shift, but it's not entierly fair to compare to the baselines, that were only developed for 'normal' Abs.\n\nIn the paper you write \"de novo design method DiffAb\", but DiffAb it's not de novo, it needs a co-crystal structure as a starting point.\n\nFigure 4 (b) second plot, only 1 point beats best point of Lambo, the average of Lambo also looks better so I'm not fully convinced that the proposed method produces more stable antibodies. Of course, the KD results are quite convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)