Data Distillation for extrapolative protein design through exact preference optimization
Authors: Mostafa Karimi
Sharmi Banerjee
Tommi Jaakkola
Bella Dubrov
Shang Shang
Ron Benson
Venue: ICLR 2025 Poster (2025)
Keywords: Protein design
Protein Language Models
Preference Learning
Extrapolation
Data distillation
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Steering protein design in the extrapolation region by using data distillation through preference optimization.

Abstract: The goal of protein design typically involves increasing fitness (extrapolating) beyond what is seen during training (e.g., towards higher stability, stronger binding affinity, etc.). State-of-the-art methods assume that one can safely steer proteins towards such extrapolated regions by learning from pairs alone. We hypothesize that noisy training pairs are not sufficiently informative to capture the fitness gradient and that models learned from pairs specifically may fail to capture three-way relations important for search, e.g., how two alternatives fair relative to a seed. Building on the success of preference alignment models in large language models, we introduce a progressive search method for extrapolative protein design by directly distilling into the model relevant triplet relations. We evaluated our model's performance in designing AAV and GFP proteins and demonstrated that the proposed framework significantly improves effectiveness in extrapolation tasks.

Published as a conference paper at ICLR 2025
DATA DISTILLATION FOR EXTRAPOLATIVE PROTEIN DESIGN THROUGH EXACT PREFERENCE OPTIMIZATION
Mostafa Karimi, Sharmi Banerjee
Tommi Jaakkola Massachusetts Institute of Technology t oomii @csail 1.mmt.... edu
Amazon
(mkarimii, Sharmiba@@amazon
Bella Dubrov, Shang Shang, Ron Benson
Amazon
(belladub, shashang, romben]aamazznn.cooo
ABSTRACT
The goal of protein design typically involves increasing fitness (extrapolating) beyond what is seen during training (e.g., towards higher stability, stronger binding affinity, etc.). State-of-the-ar methods assume that one can safely steer proteins towards such extrapolated regions by learning from pairs alone. We hypothesize that noisy training pairs are not sufficiently informative to capture the fitness gradient and that models learned from pairs specifically may fail capture three- way relations important for search, e.g., how two alternatives fair relative to a seed. Building on the success of preference alignment models large language models, we introduce a progressive search method for extrapolative protein design by directly distilling into the model relevant triplet relations. We evaluated our model's 's performance in designing AAV and GFP proteins and demonstrated that the proposed framework significantly improves effectiveness in extrapolation tasks.
INTRODUCTION
We focus on the challenging task of extrapolative protein design (Chan al., 2021 Padmakumar et al., 2 Lee et al. (002). The problem involves creating novel protein sequences with fitness values above and beyond those seen during training. Extrapolation
challenging for machine learning methods methods are primarily trained to recognize patterns within the range of the training data (Xu et al., 2020). Existing extrapolative protein design methods can be roughly categorized into two groups. Scorer-based approaches optimize sequences by following a learned score model with the help of reinforcement learning (Lee et al. 2024) or Gibbs sampling (Kirjner et al.) 2024), progressively modifying sequences towards higher predicted fitness values. Since the has been trained primarily on sequences belonging to the training region fitness values, its ability generalize sequences further away may be limited. Edit-based methods, the other hand, effectively learn gradient directions from pairs of examples derived from data or from an estimated score model, and learn propose sequence improvements. the learned patters of improvement generalize, then the methods are able to continue to improve the sequences beyond those in the original training region. Current generative models for sequence improvement approximate the gradient direction through differences between protein pairs, e.g., learning the ranking through contrastive discriminatory objective (Chan al.) 202T), token-level machine translation (Padmakumar al., 2023) and Bradley-TTer (BT) model (Bradley & Terry 1952) with maximum likelihood objective (Lee et al. 2023). The primary limitation that noisy pairs alone may not provide sufficient gradient information to effectively guide protein generation.
model in these methods
In this paper, we build on edit-based methods to address these limitations and further enhance the methods with data distillation. We draw inspiration from recent successes of preference learning (Christiano et al. 2017 Rafailov et al.) 2023) in text generation. In order to improve the extrapolation power of the generative models, distill carefully selected higher order (triplet) relationships into the models. These triplet relations are meant to better approximate the gradient direction in sequence space and therefore aid in progressive search for better sequences. We focus on improvements

---

Published as a conference paper at ICLR 2025
relative to seed with the help of offline preference learning approaches such as direct preference optimization (DPO) RRaailov et al. 2023) and efficient exact optimization (EXO) ( et al.
2024). We
introduce hard triplet relations additional guidance to the generative model pre-traied from pairs alone. Specifically, given a conditional generative model P ((.)x,, , we identify examples x1 < X2 < X3 (based on their fitness) where the current model fails, i.e., P x2) > P(xx|x2). We consider these mistakes similar to harmful text generation in LMs. The proposed data distillation approach guides language models towards higher fitness in the extrapolation region while preventing the generation of lower fitness sequences.
We evaluate our method on the well studied Green Fluorescent Proteins (GFP) by Saarisyan et al.) (2016) and Adeno- Associated Virus (AAV) by |Bryant et al.) (202I). We utilize the carefully created medium and hard difficulty splits provided by Kirjner et al. (024).. Our contributions are summarized as follows:
We develop a novel data distillation approach for extrapolative protein design through preference learning. Our approach distills model-dependent hard triplewise ranking into generative model through reverse KL offline preference learning (Section 3.4 for preference data creation and Section for general model alignment).
Our benchmark shows that approach can drastically improve the performance upon prior methods (S)ction(5).
Through ablation studies, we show the importance of training on hard triplewise rankings in comparison to other methods for preference dataset creation (Section 6.1
We benchmark our proposed approach against state-of-the-art ccorer-based approaches and our approach becomes state-of-the-art in 3 out of datasets (Section 6.2).
We benchmark the reverse KL preference learning approach against other state-of tottte-ofthh---- -teth-rrtt preference learning approaches such as DPO (Rafailov et al., (023). IPO (Azar et al. 2023).
IRPO (Pang et al. 2024). NCA (Chen et al. 2024), SPPO (Wu al. 2024). AOT (Meinyk
et al. 2024) (Section 6.3
2 RELATED WORK
Controllable Biological Sequence Design The approaches for controlled biological sequence design be categorized into (i) conditional protein design with control codes such protein families (Karimi et al.] 22220] Madani et al. 2023) (ii) controllable generation via scorer functions such as
Gibbs sampling on smoothed fitness landscape (Kirjner al., 2024 (iii) protein optimization in
latent space such as reinforcement learning in latent space of large language models (Lee et al., (024).
property-guides variational auto-encoder based models (Gömez-Bombarde Bombarelli et al. [0188 Ghaffari
2024) (iv) extrapolative protein design: extrapolating to fitness beyond training data (Chan et al.
al. 2021 Padmakumar et al., 2023 Lee al. 2023) All extrapolative protein design models have the inherent assumption that extrapolation can be learned sufficiently well through pairwise ranking of protein finesses. Chan et al. (202T) developed a contrastive learning approach of ranking pairs through discriminator of the latent space and extrapolating biological sequences through traversing it. PPadmakumar et al. (2023) proposed to learn local editor for translating sequences with low fitness to sequences with slightly higher fitness through machine translation. Recently, [Lee et al. (2023) modeled the ranked pairs through Bradley-Teriy (BT) model via a maximum likelihood objective, named align-plm. The align-pim model is an unconditional auto-regressive generative model, does not have any notion of seed (starting sequence). Therefore, the align-plm model has been used as an oracle rather than a generative model rank the fitness of heuristically generated sequences. Particularly, starting from seed sequences, they will exhaustively search all single site mutations and rank them based on align-plm model to choose the top ranking sequences and iteratively continue this process.
Preference Learning Aligning language models toward human feedback has improved their capa- bilities in following instruction (Ouyang et al. 2022) and translation (Kreutzer et al. 2018). LLM aligmment originated from the seminal work of Christiano et al. (2017) on reinforcement learning with human feedback (RLHF). However, training RLHF was shown to be challenging due to training instabilities, reward hacking and catastrophic forgetting (Peng al. 2023). Recently, there has been
2

---

Published as a conference paper at ICLR 2025
momentum for closed- and direct optimization of offline preferences. Direct optimization of human feedback can be categorized into sequence likelihood calibration (Zhao et al. 2023), 3), direct
preference optimization (DPO) through Bradley-Terry (BT) model (Rafailov et al. 2023 Mitchell 2023), more generalized version of DPO named Vi preference optimization (Azar et al. 2023
has recently been shown that, under probability matching perspective, DPO is optimizing the forward KL divergence (L (T) where T is the optimal target distribution and Te the parameterized model distribution. However, the RLHF objective function is equivalent of minimizing the reverse KL divergence (L(TTol(TT)) (| Recently, efficient exact optimization (EXO) has been proposed minimize the reverse KL divergence of the general alignment objective (Ji et al., 2024). It has been shown that DPO only covers the support of z3 (mean secking) and EXO is capturing its modes (mode secking). Direct preference models not only perform on par with RLHF but are also simpler to implement, single-stage training and computationally efficient in practice.
METHODS
3.1 PROBLEM DEFINITION
with N samples where x" Let's assume there is supervised dataset D {(x", )
(a"....., ..),)) is rihh protein sequence with length L and y" is its corresponding fitness value (i.c. stability, binding affinity). Let's assume the fitness value y in dataset D is bounded y [ymin: ymuax]
We define this region as training region and try to generate sequences with fitness value Ugen > Umax
or Ygen < Ymin which is defined as the extrapolation region.
3.2 OVERVIEW
The core concept behind the proposed method is to gradually learn the higher order relationships among ranked proteins. We start with auto-regressive unconditional pLM such Pxot-T5-XL (Elnaggar et al. 2021) that is trained on unsupervised data to model x Po(.) where x is generated protein sequence. Inspired by the ICE model (Padmakumar al. 2023). we trained a local editor with the desired direction (c.g. increasing the binding affinity) to learn the first order relationship among ranked proteins (approximating desired gradient direction through pairs). The model learns to generate X2 ~ Po(-(x]) where the fitness of x2 (designed sequence) is expected to be better than X1 (starting sequence). Inspired by direct preference optimization (DPO) (Rafailov et al., 2023) and EXO (Ji et al. 2024). we aligned the pairwise model based on hard triplets by directly optimizing newly created preferences where the pairwise model makes the worst mistakes. With this alignment, the model updates its belief of gradient direction from triplewise relationships where the pairwise model is going in the wrong direction. The overall schematic of the proposed method illustrated in Figure
3.3 LOCAL EDITING THROUGH PAIRS
Given a supervised dataset D, we trained scorer function fs to predict the fitness of a query sequence. We expect fs to perform well on the training region and perform poorly on the extrapolation region, since it has not seen these fitness values during its training. Then, following Padmakumar et al. (2023) we generated perturbed sequences by masking-infilling starting from the training sequences (seeds). Scorer function fs is utilized to assess whether the newly generated pair (seed, sequence) has small but meaningful improvement in the desired direction. We created the dataset if increasing fitness is desired and vice versa. Finally, fine-tuned the Prot-T5-XI. model (Elnaggar et al. 2021) through MLE in
D Dpair = {xx",, zz' ) with M samples where fx < fz"
an ruto-regressive manner to predict the next amino acid: paie (z|x) I1 P(z
3.4 PREFERENCE DATASET CREATION
To better approximate the gradient direction toward improved fitness in the extrapolation region and directly model higher order relationship among proteins, created preference dataset of size K based on triplets D Duriplet (Xproompe x x² x> 1 where Xprompt is the seed sequence, x,, is the desired response and x; is the undesired response. We interested in increasing fitness by moving
3

---

Published as a conference paper at ICLR 2025
telling
Teclesarco Mide
Demrge sadwaad
Design
X
Fiadhanded traido
Eoat prdweree
Porel
Oeiign
Figure 1: Schematic overview of extrapolative protein design through triplet preference learning.
from Xprompt toward Xw where fo (Xprompt)) < fo (x.)) while guarding it against sequences with same worse fitnesses (undesired ones). Therefore, create the following preference datasets (i) Don't 't go backward: triplets should satisfy the following order (xi) < f& (Xprompt)) < (x()w) (ii) Don't I get stuck at the same fitness: triplets should satisfy the following order fx(x;) (xpromp))) fs (x()).
In addition for better approximation of gradient direction and higher order modeling, we would like focus more on the triplets which are the most confusing for the pairwise model. We mathematically Xw:; Xi) with respect to the pairwise model:
define hardness for a given triplet (Xprompt>
Shardness = log P pair (xi Xprompt) - log P pair (x (prompt))
(I)
By definition, triplets with Shardness > 0 are considered to be hard examples, since the pairwise model
prefers the undesired sequences to the desired sequences given the prompts, and Shardness <= 0 are easy examples similarly. In our internal experiments, we found out that 65%-70% of the proposed triplets are already easy. We have hypothesized that one should focus on the hard triplet examples in the entire training fitness range. We have performed ablation studies to investigate the impact of these components. The details of preference creation has been explained in section 4.3
3.5 PREFERENCE LEARNING THROUGH TRIPLETS
We model the triplewise relationship through offline preference optimization recently developed for model alignment. Ji et al. (2024) proposed general model alignment which is explained in details in supplementary section A We will utilize their approximate formulation for offline preference setting. Firstly, one can define the empirical distribution fo for given prompt Xprompt and a response x; where j {w. 1} follow:
9xx log
Palx, Sprompt) Tpate (xprompe)
fo Vixxs X1, Xprompt)
Pa(Kw (prompt) Be log Ppair (F pprommee]
Bo log (
+ paii (3g prompt
Secondly, one can define the empirical distribution Pr,, as follow:
- E: j E ==
Pr. (j(xy: Xi, Xprompt)
4

---

Published as a conference paper at ICLR 2025
Where is a small number (e.g. 10- Finally, the offline alignment objective is the reverse KL between P. and Pr
Lexa B (Xx (promps )- Dariples
[II X1, Xpromp:) (1 Xprompt))
3.6 INFERENCE AND EVALUATION
During inference, the model starts with initial seed sequence, iteratively edits it and is expected
to improve its fitness. At iteration i, given the seed sequence X and the trained extrapolative would sample x, ~
protein design model P priplet (- x). predefined iterations. (Fan et al., 2018) introduced "top-k" as fixed number of highest probability vocabulary tokens keep in inference. (Holtzman al. 2019) introduced nucleus sampling avoid degenerate sampling by filtering the unreliable tail of the probability distribution. Particularly, for given top-p probability, the smallest set of most probable tokens with probabilities that add up to top-p or higher are kept for generation. Temperature (r) hyper-parameter is used to control the randomness in generation and as a way to balance between greedy search (token with max probability when 0) and uniform sampling (when + oo). Inspired from Padmakumar et al. (2023), for each initial seed sequence, we sample N (i.c. 10 for AAV and for GFP) sequences using a combination of top-k and top-p sampling with & 10, p 0.95 and temperature of 0.7 (1.0) without (with) scorer in inference. At the end of each iteration, we randomly select M (i.e. 10,000 for AAV and 2,000 for GFP) samples from all generated sequences and them seeds for next iteration when scorer is not used in the inference. We have only chosen the best sequence out of N generated ones for each seed based scorer model when used in inference. At the last 7'th iteration, we evaluate the final M samples. For iisiiccco evaluation of GFP and AAV datasets, we used the evaluators trained by Kirjner et al. (2024).
until f reaches T (i.e. 10)
EXPERIMENTS
4.1 DATASETS
In order assess the extrapolation ability of models on both sequence and fitness landscape, we have utilized the Adeno-associated virus (AAV) and Aequorea victoria GFP (avGFP) datasets processed by Kirjner et al. (2024). They proposed to use nutational gap, defined as the minimum number of mutations required from the training set to achieve the optimal fitness, in order to measure the extrapolation ability of protein design models. We used the medium and hard difficulty split of datasets where mutational gap are and 7 mutations respectively. The characteristics of the datasets are explained in Table
Table 1: Characteristics of datasets for benchmarking
GFF
AAV
Medium
Hard
Mediam
Hard
Training region
[133,3002]
[1.30_156]
[5.64,7.48]
[4.7.642]]
Extrapolation region
>3.02
ST.6
>748
>6.42
Mutational gap (99th)
6
7
6
7
Number
100
Avg. fitness of soeds
2,28
2,28
1.49
1.49
Avg. fitness of top100 extrapolation sequences
4.02
4.02
16.62
16.62
4.2 BENCHMARKED MODELS
When scorer has not been utilized in inference, we compared our proposed method to (i) Sampling: unconditional protein design through Prot-T5-XI. (Elnaggar et al., (021]) (ii) Iterative Controlled Extrapolation (ICE): extrapolation through learning local editor by translating proteins with lower fitness to slightly better fitness (Padmakumar et al. 2023 (iii) Align-plm: extrapolation via Bradley- Terry (BT) model of ranked proteins with big enough distances (Lee et al. 2023). We could not compare our method against Genhance (Chan al. 202I) we couldn't it run their code. When scorer has been utilized in inference, we compared with (i) Iterative sampling: unconditional protein design through Prot-T5-XI. (Elnaggar al. 2021) with scorer ranking, (ii) ICE
scorer: ICE with
5

---

Published as a conference paper at ICLR 2025
scorer ranking, (iii) BiGGS: Gibbs sampling with Graph-based Smoothing in the smoothed fitness landscape (Kirjner et al. 2024), (iv) LatProtRL: The recent state-of-thee t--thee-rt model for protein fitness optimization through reinforcement learning in latent space of large language models (Lee et al. 2024).
4.3 IMPLEMENTATION DETAILS
We used the CNN models trained by Kirjner al. (222)) and Dallago al.) (2021) on smoothed
fitness landscape of the training regions of GFP and AAV datasets respectively and utilized them
as scorer functions S.. Following Padmakumar et al. (2023), we created the pairs dataset D Dpairs =
{(xi. x=)
Jiml with .M = 900K (100K) training (validation) samples where they follow (x1)
(x2) < 0.5 and {x,, X₂) } can either be from original dataset or masking-inflling (e.g. 5% masking).
We trained the local editor model on pairs for 10 epochs with the Adam W optimizer (Loshchilov &
Hutter 2017). a learning rate of le-4 and batch size of 384.
Next, we created the preference dataset for both proteins following the principles of (i) Don't go backward and (ii) Don't I get stuck at the same fitness. For GFP, we binned sequences based on their smoothed fitness into buckets of [O, 0.25, 0.75, 1, 1.25, 1.5, 1.75, 2, 2.25] and [O, 0.2, 0.4, 0.6, 0.8, 1, 1.2, 1.4, 1.6] for medium and hard difficulty datasets. For AAV, we binned sequences based on their smoothed fitness into buckets of [0. 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5.5, 6, 6.5, 7] and [0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5] for medium and hard difficulty datasets. In total, created 500K (50K) training (validation) samples in which half of them based on Don't go backward and the other half based Don't 't get stuck the same filness. For Don't go backward, we sampled triplets from every three consecutive buckets where Xprompts- X₁ and X are from buckets with middle, lowest and highest fitness respectively. For Don't get stuck at the same fitness, we sampled triplets from every two consecutive buckets where Xprompts and X₁ are from the bucket with lower fitness and X is from the bucket with higher fitness. As shown by Padmakumar et al. (2023) in pair creation, utilizing masking-inflingg sequences might further improve the triplets as well. Instead of investigating masking-infiling from unconditional generative model for triplets, we have created two preference datasets utilizing paired conditional extrapolative seq2seq model in combination with scorers. These datasets are named scorer distillation and combined distillation which are explained in details in section 6.
Then, we assessed the hardness of triples defined in equation I We chose the top 100K (10K) hardest triplets as training (validation) samples for offline preference learning. We further fine-tuned the local editor model based on triplet-basee based preference learning through EXO loss function defined in equation 3 for epoch with batch size of 32, learning rate of 5e-7, B 0.1 and the Adam optimizer (Loshchilov & Hutter 2017).
4.4 EVALUATION METRICS
We use four evaluation metrics: (i) Extrapolation percentage: Our primary goal to generate sequences in the extrapolation region. We use evaluator models to assess the percentage of generated sequences that have fitness in the extrapolation region. (ii) Fitness joo: Our secondary metric the average fitness of the top 100 generated sequences that measures how far away generated sequences are from the training region in the fitness landscape. (iii) Distance the third metrie measures the edit distance of the top 100 generated sequences from the top 100 ground truth sequences in the actual assay that have not been seen by any model (extrapolation sequences). We measure the average of the closest distance between these two sets of sequences. (iv) Diversity Our final metric measures the diversity of the top 100 candidates of each model. Diversity is measured as median of the distances between every pair of top 100 candidates. We should emphasize that higher diversity does not correlate with better performance since a random algorithm can achieve maximum diversity. However, it may provide insights on the exploitation-exploaai trade-off.
RESULTS
Figure shows that for 3 out of datasets, the proposed method (EXO) significantly outperforms baseline methods while being slightly better on the hard difficulty split of the GFP dataset. Table
2 shows that (i) extrapolation percentage has substantially increased by 1.54 (20.76% 52. 75.5,

---

Published as a conference paper at ICLR 2025
17.53 (4.59% to 85.07%), 4.07 (18.52% 92.92%) times hard difficulty AAV, medium difficulty AAV and medium difficulty GFP datasets respectively in comparison to the best baseline method. It has decreased from 54.45% to 24.27% on the hard difficulty GFP dataset. (ii) The average fitness of the top 100 candidates has increased drastically by 21.53% (9.01 to 10.95), 39.76% (9.43 to 13.18), 10.19% (2.55 to 2.81), 69.03% (2.39 4.04) on the hard/medium difficulty splits of the AAV and GFP datasets respectively, in comparison to the best baseline method. (iii) The top 100 generated candidates from EXO are closer to the top 100 unseen ground truth extrapolation sequences in all datasets except being second best on the hard difficulty split of AAV dataset. (iv) The diversity of the sequences generated on the hard splits of AAV and GFP datasets is on-par with prior methods (balance exploitation vs exploration), while having lower diversity on the medium splits of AAV and GFP datasets (more exploitation vs exploration). First, we performed nyper-paramett tuning for B since it is used in the preference learning stage. Based on the results shown in Figure [5 and Table = 0.1 is outperforming other values on medium splits while performing decent and robust on hard splits. We also performed prprr-paraeeter tuning for sampling temperature (7) which is important at inference stage. Similarly, based on Figure 16 and Table 17 T = 0.7 is outperforming other values on medium splits while performing decent and robust hard splits. In addition, Figure [3] and Table 14 shows that our results are reproducible by running with five different random seeds for both the hard and the medium splits of AAV and GFP datasets.
Table 2: Comparison of ii-sioico fitness evaluation for baselines and proposed method on medium/hard difficulty splits of GFP and AAV datasets. We report average (standard deviation) of different runs.
Method
Exiraplation
Finnssou
(hard)
Distanctuus
Divenstyws
Extrapoattvn
Finnewus
AAV (medium)
Distanceios
Diversit
Top 100 ground truth
16.02
501
16
500
Sampling
16495-0.31)
7.4/00330)
4.4910.35)
7.1800.61)
(.44%(0.31)
7.42/0.30)
4.19(0.35)
7.18(0.61)
ICE
5.59%)0::04)
8.18/0.01)
2,08/0. 14)
13.56,0.18)
4 4.599500.15 15)
9.43/0.04)
7.22(0.08)
11.49(0.24
20.16%/0.00
9.010.000
7.6010000
8.1600.00
1.49%)(0.00
8.66/0000
7.29:0
6.22(0.00)
EXO
(2.75%((.74)
10.5(0177)
8.460.27)
85.07%
13.180.33
16410.41)
1.08/044)
GFP (ham))
GFF (medrum)
Method
Exirspoation
Fimesaoo
Disterstion
Diversilyoos
Extrapolation
Finnessuo
Disanccioo
Diversly
100 pround
4.02
4.12
4.64
Sampling
18.5255:0.80
1.940.09)
10.21:0.50
20.10X1.01>
18.52%
1.9:00.09)
10.2110.00
20.10(1.01)
ICB
27.105:0.800
20070.913
10.93/0.18)
15.7600.62)
0.1695(0.19)
2.390003)
8.470021))
14.410087)
Align-pim plm
(0.05%(
2.5500000)
9.6210.00)
4,1700.80)
0.0095:0.00)
2,12/0.00)
6.13(0.00)
5,4100.00)
FXO
24.27%(1.20)
281(0117)
9.08144944
149967.44)
92,92% (0.30)
4.04/0.01)
2.13/.14)
2.57(0.14)
Distribution of fitness values- A/V medium di fficulty split
Distr bution of Fitness values AVV hard difficully split
05-
Probablity dencity
Extrapotation Region
Probabiiity density
Extrapatction Region
02
19.09
new
news
18.11
2.0
19.00
mais)
11230
(1.14)
Atness value
Fitness
Distribution Fitness values- GFP
Distribution fitness velues-
medium difficully split
difficully split
Probeebiitt dersity
Probebility denaity
Extupe Megion
Extrapolation Hagion
3.-
36
32
RS5. 7m
(22.00
13. ml
123.5.
mg
30
mazd
pase
pusill
my
Plzzto
(64.
iK
Fitness value
Fitness value
Figure 2: Comparison of in-silico fitness evaluation on GFP and AAV datasets.
7

---

Published as a conference paper at ICLR 2025
6 ABLATION STUDIES
6.1 EFFECT OF PREFERENCE DATA
To evaluate the effect of preference data on the proposed method, we have created five types of preference datasets (1) All: random triplet creation through Don't 't go backward and Don't 't get stuck at the same fitess principles that includes a mix of easy and hard examples.
(2) Mistakes: focusing on the hardest triplets created from similar principles (this is the default preference of data creation in the proposed method). (3) Mistakes in extrapolation: focusing only on the hardest triplets close to the extrapolation region but in the training region. The main question behind it: Are the hard examples close to the extrapolation region sufficient for good extrapolative generative model in extrapolation region? (4) Scorer distillation: In the previous three preference datasets, perform data distillation by ranking triplewise sequences based on hardness for local editor. Another alternative approach is to utilize the local editor to propose sequences and rank them based on the scorer in order to create a preference dataset. Particularly, for a given Xprompt> one can sample K sequences YI:K = 12,,y2,- } from pair (.) Xprompt). Then, choose the triplewise rankings for all pairs based on Si.j = f(y,) - f(y:) where f(y;), < [xprompst) < f(y)). By definition, Sij > can be considered as hard example since the pairwise model has wrongly generated yj with lower predicted fitness than prompt. We would like to encourage the pairwise model to generate more sequences similar to y/: since its predicted fitness higher than seed. This approach can be considered as scorer distillation since we are attempting to distill the scorers knowledge in training and extrapolation region into the generative model. (5) Combined distillation: Finally, we combine the hardest triplets from Mistakes with scorer distillation to utilize the best of both worlds.
3sows that (I) training on hard examples is drastically beneficial in enhancing the performance of EXO. We observe that in medium difficulty splits of AAV/GFP datasets Mistakes significantly outperforms All version and perform worse (34.55% to 24.27%) and (54.78% to 52.75% 5)) on hard split of GFP and AAV datasets. In addition, average fitness of top 100 candidates have increased drastically for all datasets except hard split of GFP where it slightly performs worse (2.88 vs 2.81) on hard split of GFP dataset. (2) learning from hard examples close to extrapolation region is not sufficient for EXO. Similarly, we observe that in 3 out of 4 datasets Mistakes outperforms Mistakes in extrapolation version and perform worse (56.65% on data distillation outperform scorer distillation one for EXO. We observe that in 3 out datasets Mistakes outperforms scorer distillation version and perform worse (76.89% to 52.75%) on hard split of AAV, and (4) training on Combined distillation perform well on hard split of AAV and GFP datasets in comparison to the default one but perform worse on medium split. shows that in hard difficulty split where there is much less data to learn from it is beneficial to augment data through scorer distillation. However, it is computationally expensive to create triplets through scorer distillation since we need to run local editor in inference for many samples. Table 7 indicates that scorer distillation is more than 200 times more expensive to run on GPU machines in comparison to the default. In general, we can conclude that based on Table 3 and Figure 7 Mistakes perform better in comparison to other preference datasets with almost no computational overhead.
24.27%) on hard split of GFP, (3) training
Table 3: Comparison of in-silico fitness evaluation for various preference data on medium/hard difficulty splits of GFP/AAV datasets. We report average (standard deviation) of 5 different runs,
Triplet creation
Extrgolation
Fimessecco
(hard)
Distanc
Diverniyi2o
Eximpolation
Fireehym
(mediuun)
Distimcingc
Diversit um
All
54.78% 361
7.600000
(.K00.00
(1001.10)
(1.19910099
11
6.33(0116)
6180073
Mistakes in extropolarion
6.725500 19)
8.28/0.11)
8.060.20)
128330.30)
(.8856(0511
10.800.004)
5.61(0.27)
8.75(0.29)
Scorer distillation
76.899613448)
10.15/0.155 18)
3.000.00)
0.000.00)
(5.22%(147))
9.8300.00
5.9910.00)
0.0:0.00)
Combined distilatioon
78.19%((2.66
13.9700111)
1.740.25)
1.7000.45)
20.5910.72)
13240011
2.8510.19)
2.110.41)
Mistakes (default)
52.75%((L.4)
10.95/0.7)
5.30v0.39)
8.4600227)
(1.07%
18000.))3
1.6400.41)
1.08(0.6)
Triplet creation
Estrapolation
Filnesscco
(hand)
Distancepo
Divensayse
Extrapolation
GFP (medium)
Distaxceac
Diversib m0
All Mistakes in extrapolation
24.55%(2.223
2.88/0 261
4.300
6.0(2199))
17.544(280)
3.9900001
2503(0155)
24470.10)
56.65%<:1.19
3.6940.02)
2.1340.06)
3.1640.06)
0.00%
L99c0.02)
13.44/2.58)
17.79(4.28)
Scurer distillation
(.00%;00.00)
1.35/0.01)
9,78/0.05)
0.3340.06)
9.85%(5.01)
3.35:0.28)
3.7000.5))
3.02(0.89)
Combined distillation
(1.06)
3.68/0.02)
2.420.10)
4.07(0.08)
28.44%(..77
3.82:0.02)
2.24/0.10
3.14(0.19)
Mistakes (default)
24.27%6(1350
2.8100.00)
9.08/4.04)
14.96(7.44)
92.92%(00.30
4.060.001
2.13/0.14)
2.57:0.14)
We have further assessed the impact of 1) creating triplets from consecutive vs non-consecutive bins; 2) sequence similarity cutoff between prompt vs desired/undesired responses. Table 4 highlights that naively creating preference data from non-consecutive pairs would confuse the model's learning
8

---

Published as a conference paper at ICLR 2025
and worsen the performance on 3 out of 4 splits. We hypothesize that, in order to properly learn from non-consecutive bins, one needs to develop a more sophisticated prompt to incorporate the distance between bins well. In addition, Table T suggests that max mutation 15 between seed and desired/undesired sequences would slightly improve the performance on hard splits of AAV and GFP datasets while worsen it for medium splits. However, calculating sequence similarity for large dataset is computationally expensive therefore we would not suggest it as our default.
Table 4: Comparison of in-silico fitness evaluation for ablation studies on triplet creation.
Triplc creation
bits
Max musatoon
Taragataion
AAY (wedd)
AAY (mediami
Minzkes
Cussecative
None
52.75%411:4
(03500117)
5.3000.99
3.3600
95.07% 107761
13.1800.33
L.400 41)
12660340
Mizzkes
Cossecutive
7.24860.255
2.200.04 (4)
9.0000122
13.1800.34)
(0846600000
10.1410
7 7.2200.00
9.0640.64)
Misskks
Cineazulive
i0
53, 51,539442288 K3)
56.0447)
6.050067)
2.5814 333
57. 47.475.0.0
10.0300013
5.000.200
4.2500.55
Miszaes
Consecutive
15
(1.22
11.3068.3
531/0.19
7,3700 199
01. 11,389641.50 560
11 11,7200.22 12)
3.100.400
26300.27
Triplet creation
Max monations
Tarrestation
OFF
CFP
Mhizkes
Cossecutive
Nove
282
14.9677444
92.92% (0.33)
4.3400.833
21130..44
23570.041
Misskos
Nen-Canecaaiiee
None
51.61366 169
2 -900009)
7.061 1.45
7.721 855
92.87541 5.6
40500.05
0.900020 209
0.690:256
Mizzies
Cousecuttve
27:57541 42)
20000001
10.5300.16
14.7200655)
0.0555-00 10)
2.3500.05)
8.600.15 15)
14.430020)
Miaskes
Cimeazative
11:199200330
3.52/0.00)
4.960
3.9855/0 641
3.4800113
34310
4.50x0.333
Mizzkes
Cosserathv
(2.015
2.34000))
6.3309.21)
7.5000.99)
101000
3.5000000
0.300.055
0.000.000
6.2 EFFECT OF SCORER
Prior to our work, scorer-based generative models such as LatprotRL (through reinforcement learning (Lee et al. 2024)) or BiGGS (through Gibbs sampling (Kirjner et al., 2024)) were the state-of-the-ar and outperformed non-scorer generative models by large margins. Tables shows that our method can fill the gap, compete favorably with the scorer-based generative model and outperform them in various tasks. Particularly, EXO outperform scorer-based generative model on medium difficulty split of AAV (85.07% vs 38.63%) and GFP (92.92% 55.50%) datasets. In addition combination of EXO + scorer ranking can furthermore outperform LatproLL/BiGGE in 3 out of datasets. In addition we can observe that, on the hard split of GFP dataset even though extrapolation percentage of EXO + scorer is 50.91% % vs BiGGs 99.53%. However, its average top 100 generated sequences are almost na-par with them (3.79 vs 3.83). In addition its top 100 generated sequences are closer to the unseen extrapolation sequences (1.73 vs 3.48 mutations). Figure [14 and Table inndicaes that our results are reproducible by running with five different random seed generator for both hard and medium splits of AAV and GFP datasets.
Table 5: Comparison of in-silico fitness evaluation for scorer-based methods.
Method
Exingolation
Flozauue
AAV (hard)
Distaccue
Divesily
Estrpdaaiin
AAV (mulluum)
Distanstan
Divenaly
herative sampling
1.375/0 12)
2.670.25)
6.5 6.54(0.4)
11.240.00 701
0.406 10)
7.620014
6.500.65)
(1450097)
ICE scnner
37.01960 96)
10 10.3600
6.500.30 (0)0.30)
0.900477
53. 17% 90
10,000 1110
6.500049)
0.39(0.55)
BXKSS Lat
1680925.377 37)
10.500.51)
5.701106 (36)
(.44)
10.110885
8.0510.84)
8.34(0.93)
64822641 02)
13 32.000006)
2.450116))
4.60022)
58 38.69908886
12.53(0.88)
2.830.15)
5.210.077)
EXO
62.75%81174
10550.17)
5.3061.39)
2.600023)
(5.17((.72)
13.180.33)
L64(8-41)
12800.66
EXO scorer
94.4%:11155
14.550.111
1.52/0.21)
2.530.31)
94.23%
1399N015)
2.000.00)
3.2511.29)
EXO (Combined distillatios)
78. 23.66)
13.970.11)
1.74/0.25)
1.700045)
(0.859
13.240111
2.85(0.19)
291(0.41)
EXO (Combined distillation) + scorer
98.6%)
14,2100.02)
1.51/0.23)
1008/0.19)
64.799(0.39)
13.1900003)
3,3410.16)
3.34/0.28)
Metaed
Estrigolation
Fiteessue
on
Estrapsbation
(medium)
Diversity an
sampling
(11778)
207 45(145
1320004)
174,59(1766)
ICE
14.87900775
1,960.004
9.52(125)
a.25
2.53/004)
8,25
1566.0
(9.33%(
3.830.02)
3.8(0.36)
6.000.51)
55.509(6.75)
3.890.03)
4.1310.35)
5.4(0.711)
Latpeeelt
98:284<1 05)
148/004)
58.22%5
392/0001)
1.5610.05)
70)
14.967.44)
4.0400.11
2.1310.14)
2.5200.14)
50.15c3
1750012)
EXO (Combined distillation)
65.869210
1,650000)
2.42/01000
4.070008)
1.220.22)
2.3410
3.14(0.19)
(Cambined distillation)
71.159/2.3)
3,75/0.03)
2.100007
3.4600099
(1.45%(1.34)
3.86/002)
2,1600111)
3.19/0.16)
6.3 EFFECT OF PREFERENCE LEARNING ALGORITHM
We benchmark against the the ofte-of-the-arn offline preference learning algorithms such as direct preference optimization (DPO) (Rafailov et al., 2023), its generalized version of dlentity-mapping preference optimization (IPO) (Azar et al. 2023), addition of NLL loss function on favorable direction to DPO (IRPO) (Pang et al. 2024), noise contrastive alignment (NCA) (Chen et al.) 2024), Nash equilibrium based self-play preference optimization (SPPO) (Wu et al. 2024), alignment through optimal transport (AOT) (Melnyk et al. 2024) and our default model EXO through reverse KL distribution matching (Ji et al. 2024). Tables 6 shows that EXO outperforms other preference learning algorithms on 3 out of 4 tasks. Particularly, EXO outperform the best alternative preference

---

Published as a conference paper at ICLR 2025
learning algorithm by (52.75% 50. 787 attained by AOT), (85.07% vs 74.39% attained by DPO) and (92.92% vs 47.26% attained by IRPO) on hard split of AAV, medium split of AAV and medium split of GFP, respectively. IRPO through addition of NLL loss function to DPO loss has shown to outperforms others on hard split of GFP by 55.15% vs 42.80% attained by IPO. In addition, training curves for preference learning based on these preference learning methods have been shown on In general, we can observe that EXO is robustly performing well.
Figures 17
18
20
19
Table 6: Comparison of in-silico fitness evaluation for various preference learning algorithms.
AAV (hard)
AAV (medium)
Method
Extrapolation
Fimesssjp
Distanco
Diversity
Extrapolation
Firnession ?
Distanecino
Diversity
DPO
46. 46.31910.33)
(1.1700112)
5.68(0. 17)
9.15(0.34)
74. (4.999(2241)
(2.96(0.19)
2,7700.45)
4. 4.15(0.33)
IPO
38.74%(0.81)
9,97(0.15)
7.20 19)
11.15(0.13)
30.83%)(0.34)
11.47(0.08)
5.4(0.10)
7.3000144
IRPO
37.67%)0.76)
10.62(0.29)
5.36(0.31)
8,35(0.29)
35.13%%
11 11.37(0.10) 10)
5.39(0.22)
7.86(0.45)
NCA
39. 39.10%((1.5 89)
10.07(0.23)
6.90(0.56)
10.38(0.43)
(1.15%
11.95(0.229
4.19(0.55)
5.92(0.81)
SPPO
(0.51)
9.18(0.07)
8.22(0.08)
11.3000.40)
17.54%
10.88/0.10 10)
6.5700.09)
7.97(0.11)
AOT
30.78% 55)
10.97(0.08)
5.7910.62)
9,31(0.58)
(9.95%((1.11
12.83(0.09)
2.68(0.45)
3.61(0.94)
EXO
82.75%
10.95 17)
5.30(0.39)
8.46(0.27)
83.07 (1.72)
13.18(0.33)
1.6(0.41))
1.080.46)
(hard)
GFP (modium)
Method
Extrapolation
Fitnessuoo
Distance Loo
Diversity
Extrapolation
Foressson
Distance soo
Diversity 100
DPO
38 (6.999999)
3.73(0.06)
1.920. 19)
3.26(0.28)
13. (4.66)
3,53(0.12)
3.770.60)
5.0500.51)
IPO
42.80%
3.47(0.08)
2.05(0.11)
3.10(0:24)
(0.35%
3.69(0.03)
2.45(0.09)
1.000..19)
IRFO
(1.94)
3.68(0.06)
1.85(0.44)
2.94(0.54)
(1.16%((114)
3.85(0.01)
2.10(0.03)
3.68(0.08)
NCA
30.75%((101)
2.55(0.09)
5.65(0.97)
9,08(1.5))
24.67%(..75)
3,92(0.02)
2.32(0.05)
2,980,114)
SPPO
30.70%
2.32(0.07)
9.07(0.37)
12.03(1.05)
0.58%(0.12)
2.51(..07)
7.29(0.19)
11.63(068)
AOT
35.51%6(5.60 60)
3,38(0.09)
9,90(4.50)
18.15(8.12)
(5.54%
3.65(0.03)
3,95(0.12)
2,84(0,22)
EXO
24.27%((170)
2.81(0.07)
9.08(4.04)
14,96(7.44) 44)
(0.30)
4.04(0.01))
14)
2.57(0.14)
7 DISCUSSIONS
We utilized the ProsTTS model (Heinzinger et al. 2023) trained in multi-modal fashion (sequence and structure) to embed the unique sequences generated by each method. ProstT5 would enable us to better assess the closeness of the top generated candidates to the top unseen extrapolation sequences sequence-structure latent space rather than defining closeness based on edit distance in purely sequence space, which does not consider the characteristics of mutations. Two dimensional visualization of embeddings through t-SNE (Van der Maaten & Hinton| 2008) in Figure 3 highhlghts that top candidates generated by EXO and EXO+scorer have supports by unseen ground truth extrapolation sequences in the latent space of ProstT5 on the hard split of AAV dataset.
As mentioned, used the in-silico evaluator trained GFP and AAV datasets by Kirjner et al. (2024) and utilized by the state-of of of-thhe-n art methods (Lee et al. 2024| (023). We calculated the mean-squared error and Spearman's 's rank correlation coefficient between scorer trained on specific split of the dataset and the evaluator trained on the entire dataset. Figures 4 and 5 highlighhs that (1) the mean square error of scorer deteriorates significantly in the extrapolation region. (2) on the AAV dataset the correlation deteriorates as we move further away from the training region, as expected. However, on the GFP dataset, the scorer and evaluator have a very non-smooth rank correlation relationship. An experimental pipeline with wet validation in the loop is the ultimate evaluation pipeline which is, unfortunately, very costly and time-intensive.
CONCLUSION
We present novel data distillation approach enhance extrapolative protein design models. Our main contribution is to create model-aware hard preference datasets in order to better approximate the direction of the gradient and learn higher order relationships such as triplewise rankings. In order to evaluate our contribution, we have utilized the carefully curated medium/hard difficulty splits of AAV and GFP datasets where the model needed to extrapolate both on sequence and fitness spaces. Our framework outperforms both extrapolative non-scorer and scorer-based generative models baselines. Through ablation studies, we have investigated (I) the effects of various approaches of creating preference datasets and (2) effects of state-of-the-art preference learning algorithms. Potential future directions include (1) assessing the effect of higher order relationship (quadruples etc.) through Plackett- Luce ranking models (Plackett 1975 Luce 2005) on extrapolation, and (2) utilizing reasoning approaches such as tree of thoughts (Yo et al. 2024) to boost performance of the proposed extrapolative protein design model.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences. arXiv preprint arXiv:2310. (tti::2300.12000,, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
Drew H Bryant, Ali Bashir, Sam Sinai, Nina K. Jain, Pierce J Ogden, Patrick Riley, George M Church, Lucy J Colwell, and Eric D Kelsic. Deep diversification of an aav capsid protein by machine learning. Nature Biotechnology, 39(6):99-666, 2021.
Alvin Chan, Ali Madani, Ben Krause, and Nikhil Naik. Deep extrapolation for attribute-enhanced generation. Advances in Neural Information Processing Systems, 34:144:: 14084-14096, 2021.
Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Christian Dallago, Jody Mou, Kadina E Johnston, Bruce J Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. Flip: Benchmark tasks in fitness landscape inference for proteins. bioRxiv, pp. 2021-11,2022
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):11227722) 2021.
Angela Fan, Mike Lewis, and Yann Dauphin. Hicrarchical neural story generation. arXiv preprint 1Ef.:18805008333 2018
Saba Ghaffari, Ehsan Saleh, Alex Schwing, Yu-Xiong Wang, Martin D Burke, and Saurabh Sinha. Robust model-based optimization for challenging fitness landscapes. In The Twelfth International Conference on Learning Representations, 2024.
Rafael Gómmez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobate Benjamín Sánchez- Lengelin.. Dennis Sheberla, Jorge Aguilera-Iparragun Timothy D Hirzel, Ryan P Adams, and Alán Aspuru- Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268-276, 2018.
Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Martin Steineg- ger, and Burkhard Rost. Prostt5: Bilingual language model for protein sequence and structure. bioRxiv, pp. 2023 07, 2023.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv: 1904: 09751, 2019.
Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient and exact optimization of language model alignment. arXiv preprint arXiv:2402.00855 2024.
Mostafa Karimi, Shaowen Zhu, Yue Cao, and Yang Shen. De novo protein design for novel folds using guided conditional wasserstein generative adversarial networks. Journal of chemical information and modeling, 60(12):5667-563 2020.
Andrew Kirjner, Jason Yim, Raman Samusevich, Shahar Bracha, Tommi S Jaakkola, Regina Barzilay, and lla R Fiete. Improving protein optimization with smoothed fitness landscapes. In The Twelfth International Conference Learning Representations, 2024.
11

---

Published as a conference paper at ICLR 2025
Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learaaillity of human bandit feedback for te-sueenee- reinforcement learning. arXiv preprint 18X5::880..1000 2018.
Minji Lee, Kyungmin Lee, and Jinwoo Shin. Fine-tuning protein language models by ranking protein fitness. In NeurlPS 2023 Generative Al and Biology (GenBio) Workshop. 2023.
Minji Lee, Luiz Felipe Vecchietti, Hyunkyu Jung, Hyun Joo Ro, Meeyoung Cha, and Ho Min Kim. Robust optimization in protein fitness landscapes using reinforcement learning in latent space. arXiv preprint arXiv:2405. 18986, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
R Duncan Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2005,
Ali Madani, Ben Krause, Eric R Greene. Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et Large language models generate functional protein sequences across diverse families. Nature Biotechnology, 41(8): 1099-1106, 2023.
Igor Melnyk, Youssef Mroueh, Brian Belgodere, Mattia Rigotti, Apoorva Nitsure, Mikhail Yurochkin, Kristjan Greenewald, Jiri Navratil, and Jerret Ross. Distributional preference alignment of llms via optimal transport. arXiv preprint arXiv:2406.05888, 2024.
E. Mitchell. A note on dpo with noisy preferences and relationship to ipo, 2023. URL https c/t.rimitceell.aaa..//
pdf
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Vishakh Padmakumar, Richard Yuanzhe Pang, He He, and Ankur P Parikh. Extrapolative controlled sequence generation via iterative refinement. In International Conference on Machine Learning. pp. 2-7898. PMLR, 2023.
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024.
Baolin Peng, Linfeng Song, Ye Tian, Lifeng Jin, Haitao Mi, and Dong Yu. Stabilizing rlhf through advantage model and selective rehearsal. arXiv preprint 230.22309.10202, 2023.
Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statisties, 1932):993-200 1975.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2023.
Karen S Sarkisyan, Dmitry A Bolotin, Margarita V Meer, Dinara R Usmanova, Alexander S Mishin, George Sharonov, Dmitry N Ivankov, Nina G Bozhanova, Mikhail S Baranov, Onuralp Soylemez, et al. Local fitness landscape of the green fluorescent protein. Nature, 533(7603):397-400 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675. 2024.
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. arXiv preprint arXiv:2009.1 1848, 2020.
12

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
13

---

Published as a conference paper at ICLR 2025
A
GENERAL MODEL ALIGNMENT
Ji et al. 2024) proposed the following definition for general model alignment:
Definition A.1. Let 3 0, B, and 3, B. In addition for simplicity, let's assume the prompt sequence, y is the desired or undesired sequence and (x y) is the reward model between prompt and generated sequence. Therefore, the generalized alignment objective is defined as:
J = Ex Dx-ppeepe
(v) DRD. (P (y(x)]|P pair (y xx)x))
(4)
Where p (y satisfies:
Ps (y|x) x Po(y|x) pair (ylx)==B+
(5)
In addition, given unlimited model capacity, the optimal policy is analytically defined as:
P3 Papir (y|x) (aro())))
(6)
(x)
Where Zz(x) = Ey'~o/ Paii (y' xxee xr. is the partition function.
Ji et al. 2024) have proposed a practical way of approximating the general alignment objective through sampling and coff-normaiiztionn Following their proposed approach, for given S generated sequences Y1.s = {y1, Y2, "..".s Ys} sampled from P (-ar(-|x), one can define the empirical distribution fo based on S samples as:
9e log Pa(yalb)) paic (F1R
P Po
(7)
Σ, B+ Polyzix) pasr Org
Similarly one can define the empirical distribution ra based on reward function as:
e (x(.y.)))
Pr. (sly
(8)
Σ, Eperroxxyy
Finally, the original general alignment objective can be translated to the reverse KL between P and Pry:
Lexo = Exer-Desitttt V s (L
(9)
vi.s
B MUTATIONAL ANALYSIS
Without
With
1.5
0.8
Uimeession
0.2
0.0
2.0
24 Dimension
Dimersion
Figure 3: 1-SNE visualization (2 dimensions) of top 200 generated sequences for EXO vs baselines based on ProstT5 embedding (Heinzinger al.] (203) (left) without scorer generative models (right) with scorer generative models on the hard split of AAV dataset.
14

---

Published as a conference paper at ICLR 2025
C SCORER VS EVALUATOR
C.I ABSOLUTE PREDICTIONS
Scorer trained on Medium spli AW detaset evalunter
Scorer trained on land split AN detaser: eval ustor
Potopoaation ling on
Putropoo Region
Mean squared
Mean squared
Ground truth fltness range
Ground truth fliness range
spli
Eersssladian
Extrapolsttin
Mean squered M1101
Moaa scuurr and 101
Raap
pich
pich
range
range"
Figure 4: Comparison of in-silico scorer trained on specific subset of the data versus in-silico evaluator trained on all the data from absolute prediction perspective.
C.2 CORRELATION WITHINN BUCKETS
Scorer trained on Medium spiit AJV cataset vs evaluctor
Scorer trained on Hard split AA detasot vs eveluator
Speamann corteation
Speamann cortecttion
Fstrppiiation Aerpinn
174
Ground truth fknoss rango
Ground truth fknoss rango
trained Mediam
bevanekaour Regios
Extrpptatton Regian
Spearmmn conteatton
Spearmmn con Feot tom
(22035))
range
Ground
range
Figure 5: Comparison of silico scorer trained on specific subset of the data versus in-silico evaluator trained on all the data from ranking within buckets perspective.
15

---

Published as a conference paper at ICLR 2025
C.3 RANKING ACROSS BUCKETS
Percenaage correct ranking scroes
Percentage correct acrees
trained Medium dataset
with trained Hand solt AM detaser,
IL22-
IL2--
IMC-
IVK-
3131-
3438
34.30
34,00
Percenitage correct ran king buckets with scorer trained on Medium split of GFP detacet
Percenitage of ranking buckets
with scorer trained on Herd spit GFP dataser
12.2 75
D0000- 10425)
(6425)
DA
3322
CAN2 DALDI
23521
Figure 6: Comparison of in-silico scorer trained on specific subset of the data versus in-silico evaluator trained on all the data from ranking across buckets perspective.
D ABLATION STUDIES
D.1 TRIPLETS
Distribution
Distributiin hard difticulty spit
medium dificultyy
Edtrpo aiien Ragion
Dxtpppatio Region
Pobabitty
04
Pobabiliy
64
02.01
122.0
Value
Distributien of finees values- GRP medium difficulty split
Distribztion of Fineess values- GRP hard difficulty split
Probebiity deeniity
Probebiity dieniity
Derasaaees Region
pempsation degion
30
Finess
Ftness
Figure 7: Comparison of in-silico fitess evaluation for various preference data on medium/hard difficulty splits of GFP and AAV datasets.
16

---

Published as a conference paper at ICLR 2025
D.2 EFFECT OF SCORER
Distribution fitness velues with scorer- A/M medium ditiicuity spit
Distribution fitness velues with scorer AM hard citficuity spilt
Book
Log
ut Boo
Pobabilty dendity
Ertrpslation Ragion
Probabiity deadtty
Extropolatio Regicn
Fitress value
Fitress value
Distribution of
GIP
Distribution
scorer-
medium dificultty split
hard difficllty
Probebiitt density
Probebiiiyy density
Parppaation Rejinn
30
30
1asas 50
R4420
Brsat rureas
13..5
1152700
Fitness value
Atness value
Figure& 8: Comparison of in-silico fitness evaluation for scorer-based baselines and proposed method on medium/hard difficulty splits of GFP and AAV datasets.
D.3 EFFECT OF PREFERENCE LEARNING
Distributior of Fineess values- AAV medium difficully splt
Distribution of Fineess values: AAV hard dfficcity splk own
Dxtrapdaaion Region
Dxtrapdiation Regionn
12248
B214
Firisss value
Fineess value
Distribution
Distribution
medium difficuity
hard ditfiultty spilt
Probebiitt density
Profebiiit denilty
Tamppo on Reghts
Temppoteiinn #eglon
30
30
Baso
12:200
Ras
Figure 9: Comparison of in-silico fitness evaluation for various preference learning algorithm on medium/hard difficulty splits of GFP and AAV datasets.
17

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of computational costs of generating triplets with P3 (V100) GPU machine. We report average (standard deviation) of seconds needed for one triplet creation.
Triplet creation
AAV (seconds)
GFP (seconds)
Mistakes
0.025(0.001)
0.031(0.001)
Scorer distillation
0.136(0.001)
8.944(0.007)
D.4 CONSECUTIVE NON-CONSECUTIVE TRIPLETS
D.4.1 WITHOUT SCORER
Table 8: Comparison of in-silico fitness evaluation for preference data created based on consecutive or non-consecutive bins medium/hard difficulty splits of GFP and AAV datasets. We report average (standard deviation) of 5 different runs.
Triplet ereation
Estrapolatoo (1.75% $2,869(0884)
Pitness
AAV (hard)
Distanceuu
Diversity
Extrapolation
liinesssoo
AAV (medium)
Distance too
Divensity
Consecntive
10.95(0.17)
5.300.39) 7.4900366)
8.66002777
(5.07%((1.22)
13.1800.33 11.06/0.03)
164(0.41) 6.77(0.20)
1 46)
Non-Consecutive
9.460.09)
9.0200755)
(0.41
7.83/0.30)
Triplct creation
Extrapolation
Fireestuo
Distanceiio
Diversitynne
Extrapolation 92.92%
Pinessico
Distanccion
Divensity
Consecutive
24.27%(1.70)
2.81(0077)
9.08(4.044 (144
14,96(7.44)
4,04(0.01)
2,130 14)
2.57(0.14 14)
Con-Conecuttvve
D.4.2 WITH SCORER
Table 9: Comparison of in-silico fitness evaluation for preference data created based on consecutive or non-consecutive bins on medium/hard difficulty splits GFP and AAV datasets with scorer. We report average (standard deviation) of different runs.
Triplet creation
AAV (hanl)
AAV (moclum)
Extepolatco
Consecutive
14.2500.11) 10.6510.18)
2.53/0.31)
13,90/0.55
2.0010.06)
Can-Consecutive + scurer
51.92%/0.53)
6.12(0334)
8.56c0.63)
41.9%/0.80)
11.43:0121
3.5110.35]
7.75(0.33)
Triplet creation
Exurpolation
Fimessgu
(hard)
Distancuux:
Divenity
Fimessoeo
GFP (medium)
Distancqus:
Divesity
Conseculiv +scorer
30.91%(3.16) 35. 35.46922297)
179.0.01)
1.330.12
300500.22
3,960003)
1.7500.16
4.000.100
Non-Consecutive + scurer
2.110.099
3,23/0.77
5.2900.4)
(1.76)
3.96/0086)
1.810.29)
2.28/0.26)
18

---

Published as a conference paper at ICLR 2025
D.5 SEQUENCE SIMILARITY CONSTRAINT
D.5.1 DISTRIBUTION OF SEQUENCE SIMILARITY IN CURRENT PREFERENCE DATA
Distribution of number mutatore ons with to prompt medium difficully solit
Distribution of number mutation ons with respect to promptt AAV with hard difficully split
Probaaiity dercity
Probaaiity dereity
613
6.3
6:3
Graad pid intaki
intas
Number of mutations
Number of mutations
Distribution mutations respect
Distribution mutitions
prompt for GTP with medium difficllty
prompt GIP with difficuity spit
Probabiliy denslt
Protabbityy denalty
650
ie is
Intad
(50
oz.10
az lntae
Rumber mutations
Rumber mulations
Figure 10: Distribution of sequence similarities on hard triplets of AAV and GFP datasets.
D.5.2 CORRELATION OF SEQUENCE SIMILARITY IN PREFERENCE DATA WITH ITS HARDNESS
Comdlation of sequence dimilarty with hardhets
Comdistion of srquence dimilrrty with hardhees
Cormaction sequence with hardnees mesium culty spit.
Commecton Siillitty hardiess med dificcly sph)
-42
+12
Figure 11: Sequence similarity does not have correlation with its hardness in triplets on both AAV and GFP datasets.
19

---

Published as a conference paper at ICLR 2025
D.5.3
WITHOUT SCORER
Table 10: Comparison of in-silico fitness evaluation for sequence similarity cutoff preference dataset on medium/hard difficulty splits of GFP and AAV datasets. We report average (standard deviation) of 5 different runs.
Triplet creation
Estrapalaioo
AAY (hard)
Diversity mo
Extrapolation
(medium)
us
mav sontation
8.25(0394)
(31300334)
(0:84520003
9.060004)
max mutation 10
58.338 83)
9.58(0.47)
8,0500887)
2.504.33)
57.47%5
10.03/0.01)
5.9900.20)
4.25(0.45)
max mutation = 15
55.62%
11.2000.9))
1.110..19
7.3700 19)
81. (89 86)
11 12)
3.190040)
2.630.23 27)
No max mutation constraint
12.75%
10.95(0.17)
5.3000.39)
8.16(0.27)
(5.0%%
13.18/0.39)
16410.41)
108(0.46)
Triplet creation
Extrapolation
Pinessauo
Distanccato
Deversity
Extrapolation
Fimesssco
(mediun))
Distanceuo:
Diversiiv
max mutation 5
27.5%
2.05(0.01)
10.830.118 165
14.2200653
0.055%400.10
2.35/0005)
8.68:0.15)
(4,350026)
max motation 10
11.16%:0..8)
2.78(0.04)
3.52/0.30)
4.5%(0.71)
3. (.98%%(0.84)
3.48(0.11)
3.4360.32)
4,500..82)
mutation 15
(6.88)
2.540..07)
6.32/0.21)
7.51(0.59)
63,109 58)
3 3,950000)
0.30/0.65)
0,00/0.00
No motation
24.275K1120
2.81(0077)
9.08/4.04
14:99744))
92.925%00.30)
4.04(0.01)
2.130.14
2.57(0.14)
D.5.4
WITH SCORER
Table 11: Comparison of silsio fitness evaluation for sequence similarity cutoff preference dataset on medium/hard difficulty splits of GFP and AAV datasets. We report average (standard deviation) of different runs.
Triplet creation
Extngolalinn
FiinesSue
AAV Chaed
Distaceum
Divenaty up
JUW meedium)
Distanceree
LiversHS
mis mutation 5 scorer
17.145:127)
0,2300.01
19100.00
$251 30)
6.3000.71
8.61000451
max matation= 10 scuree
(7.14%(8.70)
9.600 24
8 5.000.15)
0.520.236
95. 649
13 13.560009)
1.540.15))
2.5560.2 2500.221
mark mutatiin = 15 scoree
02.975(1991)
14.37(0.86)
148(0177
2.32/.12)
95.01% 757
1388000)
(.800-22)
3,000 37)
No max mutiaioon constrsint scorer
84 84045((111)
14.2500 11)
15200.113
2.5303313
94.23%% (8.60)
2.000006)
3.85(0299)
GFP
Estrguuattoon
Diverity um
Fatrspoiation
max motation scorer
2081%61151)
2.060.02)
109120119
15.90(103)
0.605(0.14)
2.6100.09)
8.15(0U8)
13.9HH055)
max
21.67%/22766
3.220.09)
4.1500222
mix
2.710.091
3.930.041
max mutation constraint scorer
50.91%/33316
3.790.01)
1.300.12
3.08/122))
58,09926.355
3.96/1883)
2.75/0.16)
4,94/011 11)
D.6 EFFECT OF NUMBER OF ITERATIONS
Distribution hitness values- medium difticutty split
Distribution of hitness values-
medium difficutty split
Probabiliyy denalty
0.5
Pubbbiiity
0.8
02
12.00
10:20
Finess
Einess value
Figure 12: Comparison of in-silico fitness evaluation for various iterations of EXO on medium difficulty split of GFP and AAV datasets.
Table 12: Comparison of in-silico fitness evaluation for various iterations of EXO on medium difficulty split of GFP and AAV datasets.
AAV (medium)
GFP (mediun)
Estrapolation
Funcesico
Disanceion
Diverstty
Extrapolation
Flinesspp:
Distanceyo
Diversit
18.58%
11.75
3.68
6.13
76.5%
3.56
0.38
0.78
11.99
69.17%
13.01
1.67
1.95
92.05%
4.05
2.14
2.47
10
83.83%
13,50
1.00
0.99
93.10%
4.04
1.93
2,50
20

---

Published as a conference paper at ICLR 2025
D.7 EFFECT OF TOP CANDIDATES
Table 13: Comparison of in-silico fitness evaluation for baselines and proposed method on medium/hard difficulty splits of GFP and AAV datasets. We report average (standard deviation) of different runs.
AAV (hard)
AAV (medium)
Method
Fitnessio
Fitmess
Fitness
Fitness.1
Fimessuu
FiteeShro
FitnesSpopp
Fitness
Sampling
9.33(0.43)
7.42(0.30)
5.32(0.20)
3.32(0.07)
9.33(0.43)
7.42(0.30)
5.32(0.20)
3.32(0.07)
ICE
(0.11) 9.1(0.00) 12.63(0.47)
8.18(0.01)
6.67(0.01)
3.83(0.01)
10.71(0.05)
9.43(0.04)
7.64(0.03)
4.42(0.03)
Align-plm
9.22(0.00)
7.90(0.00)
5.35(0.00)
9.54(00.0)
8,70(0.00)
7.33(0.00)
4.99(0.00)
EXO
10.95(0.17)
9.41(0.08)
6.54(0.04)
13.32(0.42)
13.18(0.33)
12.46(0.14)
9.87(0.13)
GFP (hard)
GEP (medium)
Method
Pitnessio
Filnessoou
Pitness
Finesss.n
Fiinessnu
Fiineshoo
Finnesspop
Fiteessai
Sampling
243(0.13)
L.94(0.99)
1.28(0.12)
1.220.01)
2.43(0.13)
1.9400.99)
1.28(0.12)
1.22(0.01)
ICE
2.22(0.02)
2.07(0.01)
1.610001))
1.320.01)
2.94(0.10) 10)
2.39(0.03)
1.72(0.01)
1,45(0.01)
Align-plm
2.86(0.00)
2.39(0.00)
1.60(0.00)
1.33(0.00)
2.27(0.00)
2.12(0.00)
1.87(0.00)
166000.00)
EXO
3.74(0.07)
2.81(0.07)
1.70(0.02)
1A4K0.02)
4.09(0.01)
4.04(0.01)
3.85(0.01)
3.58(0.01)
REPRODUCIBILITY
E.1 WITHOUT SCORER INFERENCE
Distribution Fitness values. AAV med lum difficulty split
values. hard difficulty split
Probability density
Probacility density
Extrepolaton Region
Extrepolation Region
64
02
60
1831
11.)
12.5
B.4
A3
101
(1.13)
(2210)
10.31
12.0
17.1
101
(1215)
f18.44
value
Distribution of Fitness values- GFP
Distribution of Fitness values- GFP
med difficulty split
hard difficulty split
Tay2
Dar
Pan
Fan
Bun
Pa14
faxt
1.5
1.5
Probablity density
Procabliity density
1.2
Extrapateti
3.2
Extrapelatior
86
03
03
(
(230029)
(6.75.1
f2.125)
1.331.3)
(128.00
(.221)
(0022))
10.25..5)
8.78.9
f.t25)
(125.13)
(I 2
(2.73.1
(3.339)
(1.75.4)
Fitness value
Fitness value
Figure 13: In-silico fitness evaluation for EXO algorithm on medium/hard difficulty splits of GFP and AAV datasets with different random seed.
21

---

Published as a conference paper at ICLR 2025
Table 14: In-silico fitness evaluation for EXO algorithm medium/hard difficulty splits of GFP and
AAV datasets with 5 different random seeds.
Method
(hard)
AAV (medium)
Extrapolation
Fiieesyn
Distance
Diversity
Extrapolation
jireess
Distance
Diversity
50.41
11.12
8.60
83.66
13.50
1.30
0.84
Run
54.58
11.19
5.58
8.65
83.71
12.80
2,00
0.75
Run
54.79
10.83
5.53
8.77
87.94
13.32
1,90
2.00
Run
52.7
10.85
5.11
8.05
86.22
12.76
2,00
0.93
Run
51.27
10.76
4.61
8.24
83.83
13,50
1,00
0,90
AVG
52.75
10.95
8.46
85.07
13.18
1.64
108
STD
1.74
0.17
0.39
0.27
1.72
0.33
0.41
0.46
GPP (hard)
GFP (meddium)
Method
Extrapolation
Fimeessyu
Distancoyuo
Diversity
Extrapolation
FitnesS
Distance
Diversityia
Run
22.40
2.70
6.69
10.36
92.65
4.06
2.20
2.52
25.70
9,47
92,95
2.08
2.56
Run
23.25
2.90
16.63
29.09
92.30
4,04
2.09
2.84
Run 5
23.15
2.86
6.14
9.92
93.10
4.04
1.93
2.50
2.81
STD
1.70
0.07
4.04
7.44
0.30
0.01
0.14
0.14
E.2 EFFECT SCORER
Distribution Fitness values. AAV med lum difficulty split
values. hard difficulty split
Probability density
Probacility density
Extrepolation Region
Extrepolation Region
64
02
60
18.1
11.)
12.5
B.4
M55)
B.0
0.101
(1.11
(1213)
10.31
12.0
D.4
M5
181
111.29
f20.44
value
Distribution of Fitness values- GFP
Distribution of Fitness values- GFP
med difficulty split
hard difficulty split
Tay2
Dar
Pan
Fan
Bun
Pa14
fant
1.5
1.5
Probabiity density
Procabliity density
1.2
Extrapolation
3.2
Extrapelatior
06
03
03
(
(330029)
(6.75.1
f2.125)
11.511
25.0
(C425)
(00.2))
80.25..57
fl.25)
(125.13)
(22.3))
(25.15)
(22.7
(1.75.))
(4422)
(I
Fitness value
Fitness value
Figure 14: In-silico fitness evaluation for EXO algorithm with scorer on medium/hard difficulty splits of GFP and AAV datasets with 5 different random seed.
22

---

Published as a conference paper at ICLR 2025
Table 15: In-silico fitesss evaluation for EXO algorithm with scorer medium/hard difficulty splits of GFP and AAV datasets with 5 different random seeds.
Method
(hard)
AAV (medium)
Extrapolation
Fiinessn
Distance
Diversity
Extrapolation
Fireess
Distance
Diversityjae
83,25%
14.20
L.84
3,10
93.97%
13.83
2.09
3.50
Run
83.59%
14.46
1.29
2,48
95. 14%
13.96
1.95
2.62
Run
80..
14.25
1.35
2.58
94.53%
13,92
1.91
2.88
Run
86.22%
14.23
1.72
2.28
94.19%
13.92
2,00
3.17
Run
83.04%
14.13
1.42
2.19
93.32%
13.85
2,06
3.07
AVG
84.04%
14:25
2.52
94.23%%
13.89
2,00
3.04
STD
1.14%
0.11
0.21
0.31
0.60%
0.05
0.06
0.29
GPP (hard)
GHP (meddium)
Method
Extrapolation
FimtesSyu
Distanco
Diversity som
Extrapolation
FitnesS
Distance
Diversityia
Run
53.00%
3.78
1.78
3.10
53.94%
3.96
3.07
4.18
52.84%
3.81
2.65
69.39%
3.92
Run
10%
3.82
1.72
3.16
56.69%
3.92
2.69
4.05
Run 5
49.00%
3.77
1.81
3,18
$9.50 59%
3.96
2.71
3,91
STD
3.16%
0.02
0.12
0.22
6.38%
0.03
0.16
0,10
F HYPER-PARAMETER TUNING FOR PROPOSED METHOD
F.1 PREFERENCE LEARNING PARAMETER: B
Distribution of Fitness values- AAV med lum difficulty split
Distribution of Fitness values- AAV hard difficulty split
Probablitty density
Probability density
Extrapoiation Region
Extropolation
04
04
1831
(na)
12.41
D3
90.0
1o.
R2.13
18.1)
12.11
12.01
17.1
88.00
19.00
Fitness value
Fitness value
Distribution of Fitness values- GFP difficulty split
Distribution of Fitness values- GFP
difficulty split
1.5
1.5
Probabilit dersity
Probability density
Extropotation Region
Ectrapelation Region
6.8
0.6
(6239.3)
R2.
11.31.9
115,1
12.25
122359
25.)
12.221
(3.2% (1.35))
(88225)
(825..5)
25:5
R2. 201
(1.2...)
(2222)
22232335 51
12.35.3
(22220
12.5..55
(24.2.79)
(2.33.)
(44.3))
Fitness value
Fibress value
Figure 15: In-silico fitness evaluation for EXO algorithm with scorer medium/hard difficulty splits of GFP and AAV datasets with different beta sypeetpprameters.
23

---

Published as a conference paper at ICLR 2025
Table 16: In-silico fitness evaluation for EXO algorithm medium/hard difficulty splits of GFP and AAV datasets with different 3 hyper-parameters.
AAV (hard)
AAV (medium)
Methou
Extrapolatinn
Distanco
Diversily 300
Extrapolation
DistaneC +
Diversityrar
3=0.02
64.14%
8.57
8.00
0.00
39.20%
9.85
7.00
0.00
9=0.05
59.84%
8.32
0.00
24.50%
0.0
6=0.1
51.27%
10.76
4.61
8.24
83.83%
13.50
1.00
0.99
J=0.2
47.06%
12.54
67.29%
3-0.5
33.20%
10.25
6.07
9.35
32. 19%
11.56
5.25
7,80
3-0.9
29.95%
9.66
7.38
10.95
29.08%
11.25
6.13
8.01
GFP (hard)
GFP (mediuu)
Method
Extrapolation
F'inessipo
Distance
Divensily 100
Extrapolation
Tiness
Distance
Divensilyio
7-0.22
62.55%
1.87
0.38
1.17
1.84%
2.59
2.78
4.82
8=0.05
11,40%
2.68
14.16
24.65
0.10%
2.55
5.85
6.22
8=0.1
23.15%
2,86
6.14
9.92
93.10%
4.04
1.93
2.50
d=0.2
51.60%
3,77
3.56
25
3.84
3w0.5
37.25%
3.13
2.72
4.20
33.20%
3.94
1.96
3,00
8=0.9
12.00
F.2 SAMPLING PARAMETER: TEMPERATURE (T)
Distribution of Fitness values- AAV med lum difficulty split
Distribution of Fitness values- AAV hard difficulty split
(012
Probablitty density
Probability density
Extropolation Region
Extropolation
04
04
1831
(12)
12.4)
13
10.0
R2211
10.1)
12.01
19200
111.22
Fitness value
Fitness value
Distribution of Fitness values- GFP
Distribution of Fitness values- GFP
difficulty split
difficulty split
1.-
1.5
Probabiltt dersity
Probability density
Extrapaation Region
Extrapelation Region
64
0.6
06
(6239.3)
(8.75.1
RR.
11.,177
1.7
12.75
122.99
(1.25
(8a25))
(825..5)
IId 25)
11225.1
(2225)
25
(39.2.99
12.33.30
(22220
12.5..5)
(24.2.m
(2.55.)
(44.0)
(2
(5.
Fitness value
Fibress value
Figure 16: In-silico fitness evaluation for EXO algorithm with scorer medium/hard difficulty splits of GFP and AAV datasets with different sampling temperature (r) hyper-parameters.
24

---

Published as a conference paper at ICLR 2025
Table 17: In-silico fitness evaluation for EXO algorithm medium/hard difficulty splits of GFP and AAV datasets with different sampling temperature (r)
AAV (hard)
AAV (medium)
Methou
Extrapolation
FatacaSqpp
Distanco
Diversity sou
Extrapolation
FiuncsS
DistaneC
Diversityror
T=0.7
51,27%
10.76
4,61
8.24
83.83%
13,50
1,00
0.99
57.4%
13.14
3.16
5.54
76.03%
2.78
4.28
r=1.2
44.73%
12.00
4.33
7,80
62.94%
13.63
3.01
4,93
5,95
GFP (hard)
GFP (mediuu))
Method
Extrapolation
Fimnessi00
Distanccino
Diversityion
Extrapolation
Fitness
Distancc1o0
Diversityio
Tw0.7
15%
2.86
6.14
9.92
91.10%
1.93
2.50
r=1.
49.25%
3.76
1.88
3.26
36. 19%
3.86
2,90
4.57
35.00%
3.15
3.67
6.59
29.84%
3.92
2.34
4.10
r=1.5
19.30%
2.45
12.77
24.18
41.80%
3.96
2.25
4.18
G HYPER-PARAMETER TUNING FOR BASELINE MODELS
G.1 WITHOUT SCORER
G.2 SAMPLING
We have utilized Prot-T5-XL (Elnaggar al.) 2021
pre-trained generative baseline model. We used the same combination of top-k and top-p parameters (k: 10, p = 0.95) as the proposed method since we have done any tuning for these parameters in the proposed method as well. Similar to the proposed method, we have performed
tuning for temperature (T) with the same grid search of = [0.7, 1.0, 1.2, 1.5). Table suggggss that r 0.7 is performing slightly better than others however within the margin of standard deviations.
Table 18: In-silico fitness evaluation for Prot-T5-XL algorithm medium/hard difficulty splits of GFP and AAV datasets with different sampling temperature (r) hyper-
AAV (hard)
AAV (medium)
Method
Extrapolation
Finess
Distance
Diversity
Exiapolation
Fitees
Distance
Diversity
(defauk))
164%
7.42(0.30)
4.49:0.35)
7.1800.61)
(0.1)
7.42(0.30)
4.49(0.35)
7.1600661)
r=1.0
1. 1.62%(0.12
7.45(0.14)
4.42(0.10)
7.24(0.12)
(.62%(0.12)
7.4500.14)
4.42(0.10)
7.24(0.12)
1=1.
I. 1.566(0.18) 18)
7.31.0.12
4.59(0.2))
7.4900225)
1.56%8(0.18)
7.310..22
4,59.0.23)
7.49(0.25)
160%(0.12)
7.4400.07)
4.43/0.05)
7.240.17)
(60%)0011)
7.44(0.07)
4.43(0.05)
7.24(0.17)
GTP (hand)
GFP (meshhum
Method
Extrapolation
Fitnessio:
Distanceioo
Diversityico
Extrapolation
Filnes::00
Distance 100 à
Diversáyyop
P-0.7 (defanlt)
18.52% (0.80)
1.94(0.09)
10.21(0.50)
20.10(101)
18.22%
1.94(0.49)
10.21(0.50)
20.10(1.01)
(0.85))
1.87:0.02)
(7.74%(0.85)
1.87(0.02)
17.67%:(1.27)
1.8((0.0))
10.58(0.25)
20.85(0.53)
17.67%((1.77)
1.870.0))
10.58(0.25)
20.85(0.53)
yml.5
(8.28%(0.399
1.890.01)
10.54(0.17)
20,7000.33
18. (8.28%((
1.890.01)
10.54:0.17)
20.70(0.35)
G.2.1 ICE
Iterative Controlled Extrapolation (ICE) Padmakumar et al. (0233 utilizes ranked pairs of sequences learn local editor. We used the same combination top-k and top-p parameters (k 10, p 0.95) as the proposed method since we have not done any tuning for these parameters in the proposed method as well. Similar to proposed method, we have performed hyper-paramen tuning for temperature (7) with the same grid search of (0.7, 1.0, 1.2, 1.5). Table 19 suggests that
T 0.7 performing slightly better than others however within the margin of standard deviations.
Table 19: In-silico fitness evaluation for ICE algorithm on medium/hard difficulty splits of GFP and AAV datasets with different sampling temperature (7) hyper-parameter.
Method
Extrapolation
Fitncs
AAV (hard)
Distanceuo
Diversityio ico
Exuapolation
Fiines:um
(medium)
Distance
Diversity
=00.7 (d3fauli)
5.58%
8.1800.01)
9.18(0.14)
13,560 18)
(.15%)
9,4300.04)
7.72/0.08)
11:49/0:24)
8.10(0.07)
9.42:0.12)
14.3000.34
3.72%
8.15(0.14)
r-1.2
(00%00008)
8.060.06)
9.47(0.3))
14.410.
(0.13)
9.21(0.03)
8.060.19 12)
12.74(0.23)
vm1.5
4 (886400077
8.24(0.06)
9,190.12)
14,460022)
3.97900220)
9.1600.7)
7.950.15)
12.53(0.29)
Mothod 7-0.7 (defanlt)
Extrapolation
Pinnessjo
GTP (hard)
Distance.oo
Diversityiso
Extmpolation
Fimess,po
GTP (mediuum)
Distance
Diversity
(0.80)
2.07(0.01)
10.930.18
15.760.64)
(.16%)(0.19)
2.3900.03
2.470.21)
(4.1100.7)
1=1.0
19.97%
2,03/0001)
11.34(0.16)
16.760033)
0.32%
2,3400.04)
8.28(0.16)
15.6300.25)
(0.35)
1.94(0.01)
18.61(0.26)
0.12%
2.35(0.01)
831/0.08)
1-5.5
(0.66)
1.92(0.02)
9,90/0.10)
19.18(.14)
(.03%)(0.02)
2.38/0.02)
8.42(0.08)
16.52(0.19)
25

---

Published as a conference paper at ICLR 2025
G.2.2 ALIGN-PLM
Align-plm Lee et al. (223)) proposed a Bradley- Terry (BT) model of ranked proteins with big fitness distances between ranked proteins. We have utilized their implementation to train and run their model in inference. In inference, they proposed to use all single-site mutations with respect to seed sequence as generator and keep top M (e.g. M = 10) sequences ranked by likelihood of the proposed model. At the end of each iteration, they will keep the top N (e.g. N 10000) for being used as seed sequence for the next iteration. For inference, M is their only hyper-parameter. We have performed hyper-parameter tuning for M with the grid search of = [5, 10, 20J. Their model would need 10 days to finish 10 iterations on GFP dataset on a p3 GPU machine. Since align-plm approach uses all single-sir site--stteeee siteemmtaiions reporting its standard deviation as zero in the Table 20 We can observe that M 20 is outperforming the default (M 10), therefore, we will use this value.
its generative model, so it is deterministic model. Therefore,
Table 20: In-silico fitness evaluation for Align-plm algorithm on medium/hard difficulty splits of GFP and AAV datasets with different top M hyper-parameter.
Method
AAV (hard)
AAV (medium)
Extepolation
Pinesssyc
Distance
Diversity.o
Extropolation
Distance,ju
Diversilyop
M-5
(0.36%(
10.29(0.00)
6.98(0.0))
7.9100.00
(.13%(0..00
8.81(0.00)
7.32/0.00)
5.92(0.00)
M=10
22.49% 20.76R(0.00)
9.2200.00)
7.4600.00)
8.56(0.00)
(0000)
$.700.00)
6.83(0.00)
5.7700.00)
M=20
9.0100.00)
7.600.000)
8.16(0.00)
(0.00)
8.66(0.00)
7.29/0.00)
6.22(0.00)
Method
Extrapoation
Fimeessico
GFP (hard
Distanccooo 1 Diversity
Extrapolation
Fliness
(medium)
Distance
Diversity
M-5
20.154(03.00
2.5100.00)
8.64(0..00)
5.11(0.000
(.10%)(0.00)
2.23(0.000)
5,45(0.00)
4.95(0.00)
M=10 (default)
20.30%/0.00)
2.3910.00) 2.55(0.00)
6.8%00000
5.62/0.00)
0.009% (0.00)
2.12/0.00)
5.96/0.00)
5.2310.00)
M-20
(0.00)(0000)
9.64(0.00)
4.17(0.00)
(0.00)
2,12/0.00)
6.13/0.00)
5.41(0.00)
G.3
WITH SCORER
G.3.1 ITERATIVE SAMPLING
We have utilized Prot-T5-XL (Elnaggar et al. 202T in combination with scorer as pre-trained language model guided by scorer to increase fitness. We used the same combination of top-k and top-p parameters (k 10, == 0.95) as the proposed method since we have not done any tuning for these parameters in the proposed method as well. Similar to the proposed method, we have performed hyper-parameter tuning for temperature (r ) with the same grid search of = [0.7, 1.0, 1.2, 1.5). Table
21 suggests that 7 1.0 (default) is performing similarly to other J within the margin of standard deviations.
Table 21: In-silico fitness evaluation for Iterative sampling algorithm on medium/hard difficulty splits of GFP and AAV datasets with different sampling temperature (T) hyper-pppaae parameters.
AAV (hard)
AAV (medium))
Mathod
Extrapolation
Finesss oo
Distance (o)
Diversity 100
Extripolatio
Firness 100
Distance ioc
Divessity uo
r=0.7
118%/00110
7.64(0.07)
6.37(0.16)
10.83(0.37)
0.47%8
7.64(0.30)
6.57(0.36)
55.
==1.0 =10 (defaut) T=1.2
1.37%/0.12
7.67(0.25)
6.540.44
11.28/0.70)
0 0.40% 10)
7.67(0.14)
6.59(0.65)
11.4300.97)
(429)(0226)
7
6.42(0.38)
11.04(0.55)
(0.05)
7.95/0.11)
6.18/0.19)
10.74(0.44)
1-1.5
(.82)(
7.900.53)
6.43(0.13)
10.820.228
(.61%)
8.000.30 30)
22000.2
10.91(0.41)
Method
GFP (hard)
(mediunn)
Extrapolation
Fitncsson
Distance:
Disersity
Exirspolation
Fitness
Distanceur
Diversity co
=00.7
(.035(0.022
1,190.02)
180. 3618.35
205.14(1.25)
(0.00(00000)
1. (9(0.002)
187 217.11107778
207.74(2.5)
1. (defauly
0015-00.02)
1.1(0008)
184.71(10.78)
(.00%%(0.00)
1.22(0.04)
174-52(17.62)
201.22(6.04)
1=1.2
(.00%/0.80)
1.160.03)
178.61(8.32)
205.99(3.95)
(.009)(0.00)
1.2100.03)
206.30(3.75) 3913333)
15
0.01%/00.02
1.180.02)
175.05(16.85)
202.91(7.20)
(0.00)
1.20/0.22)
(83.63(13.60)
204.96(4.87)
G.3.2 ICE + SCORER
Iterative Controlled Extrapolation (ICE) Padmakumar et al. (0023) with scorer has the same hyper- parameter (7) to be tuned as most of the other models. we have performed hyper-parameter tuning for temperature (7) with the same grid search of = [0.7, , 1.0, 1.2, 1.5). 5 = 0.7 is the default value used by the authors. Based our hyper-parameter tuming reported in Table 22 we can observe that default value performs well on GFP hard split and T 1.5 performs well on both medium and hard split of AAV dataset. Therefore, will use 1.5 in our main reporting.
26

---

Published as a conference paper at ICLR 2025
Table 22: In-silico fitness evaluation for ICE algorithm with scorer medium/hard difficulty splits
of GFP and AAV datasets with different sampling temperature (r) hyper-parameters.
Method
AAV (hard)
AAV (inedium)
Extrapolation
Fiieessue
Distanceue
Diversilyicn
Exmmpollttonn
Fimessan
Distanceus
Diversitynon
==0.7 (Osfaul))
27.13%5(3
9.240019)
818(0.19)
807700887
(8.4%%(3.20)
10,34(0226)
7.2
8.37(1,22)
r=1.0
(0.49)
9.64 12)
7,78/0.21)
10.47(0.31)
25 (1223)))))
10.50(0.05)
6.89(0.17)
9.33(0.31)
x=1.2
34.41%(00774
987700188
7.940013)
11.1800.4))
29.948(1221)
10.6800.10)
6.85(0.20)
9.2910.2)
r=1.5
37,01 & (0.36)
10.26(0.12)
6.50(0.30)
9.90X0.47)
(0.59)
10.8000100)
6.52/0.49)
9.89(0.53)
Method
GTP (hard)
(medium)
Extrapolation
Finess
Distanecgo
Diversilyjeo
Extapolation
Frinessgg
Distancego
Diversily
--0.7 (defoul)
28.78 (0.40)
2.0%%0.01)
10.85(0.11)
(3.960083)
0.139)00.19)
2.66(0.05)
7.87(0.23)
(1.66(044)
1=1 10
20.6895 39)
2.05(0.01)
10.93(0.16)
16,49(0.56)
0 % (0.10)
2.5600.02)
8,0300 14)
14.2700.24)
1=1.2
14.30%
1.95(0.02)
10.29(0.05)
18.4000.30
(.29%))
2.54(0.05)
8.13(0.10)
15.1410.18)
1-1.5
(0.75)
1.96(0.04)
9,52(0.25)
18.19(0.46)
0.02% (0.02)
2.53(.044)
8.22/0.14)
15.68(0.28)
G.3.3
BIGGS
BiGGS Kirjner et al. (2024) proposed Gibbs sampling with Graph-based Smoothing in the smoothed fitness landscape. The main typer-parameter, in their inference is the temperature (T) used in Gibbs sampling. Their default is 0.01. We have performed hyper-parameter tuning for temperature (T) with the grid search of = [0.002, 0.005, 0.01, 0.02]. Based on Table 23 we can observe that 0.002 is having serious issues with diversity. Particularly, it has diversity of zero for medium splits of AAV and GFP datasets for top 100 designs. In addition, we can observe that for hard splits of AAV and GFP, default value (r = 0.01) is performing better than (T 0.005) while (r = 0.005) is performing better on medium splits of AAV and GFP datasets.
Table 23: In-silico fitness evaluation for BiGGS algorithm on medium/hard difficulty splits of GFP and AAV datasets with different sampling temperature (r) hyper- Saperrparameters.
Method
A (hard)
AAV (medium)
Extrapolation
Fitness
Distance
Diversily
Eximpolation
Fiiness
Distance
Diverslly
000000
31.2%(3.67
10.0%K.332
33950.00)
0.128 (0.25)
(5.97%(122))
11:56:0)
5.5%0)
0.00x0.00)
7-0.005
18.704<( 94)
9.35(0.30 30)
0.20)..5)
3.56(0.44)
9.3154(1.20 28)
12.37(0.19)
5.74(0.46)
5.3000224
==0001 (default)
16.80% 37)
10.55
5,7011.00 000
6,38(144))
4,88% 84)
10.210.88)
8.05(0.84)
8,34(0.93)
T=0.02
26.19%((240)
10:01/0.09)
4.28(0.31)
3.79(0.13)
7.07%v0.70)
11.28/0.43
4.65(1.04)
7.36(0.97)
GFP (hard)
GFP (medium)
Method
Exirapolatinn
Fimessso0
Distancegoo
Diversityoo uoo
Extmpolation
Finess
Distanceggo
Diveraly
D-00002
(0.0%(0000)
X9970.01
0.66(0.39)
0.24(0.2)
(0.0)
3.9900.04)
0.4000.48)
0.9000.00)
=00.005
89.30% (4)
3.800002)
20)
3.21(0:39)
(1 (1.58)
4.05(0.01)
2.21(1.15)
1,9600 70)
0000 (default)
99.53%/0.221
3.83(0.02)
3/48(0.36)
6.01(0.51)
55.50%
3,8900.03)
4.13(0.38)
5.740.71))
T-0.02
95.58%(0.97)
3.88(0.01)
2.36(0.15)
4.39(0.22)
67.91%((4227)
3.91(0.01)
2.64(0.22)
5.23(0.36)
G.3.4 LATPROTRL
LatprotRI. Lee et al. 2024) proposed protein fitness optimization through reinforcement learning in
latent space of large language models. Maximum number of mutations allowed per step is the only hyper-parameter controllable in inference of LatprotRL. We have performed hyper-parameter tuning for maximum number of mutations allowed per step with the grid search of = [2.3, 4, 5]. When we set maximum number of mutations allowed per step equal 2, we have encounter errors in the inference for hard splits of AAV and GFP. Based on the results reported in Table 24 the current default value (3) is already performing robustly well on these datasets.
Table 24: In-silico fitness evaluation for LatprotRI. algorithm on medium/hard difficulty splits of GFP and AAV datasets with different maximum number of mutations per step per-parameter.
Method
Extrapoiation
AAV Thad)
Distaneem
Diversity
Extropetaiion
Finnss
AAV (medium)
Distacee
Diversis
max mos.mmmtiinnga... per step-2
12,450019
2.780.05
43500.20
max, mutammttnnnoooooom per step-3 (defank))
64.22
13.29.006)
2.45:0.16)
4,67)023)
38.61% 86)
12,5300088
2.830.15)
5.210.077
max mutatiots per steped
62.57%
132220111
2
5.10(011)
45 38)
12.6000111
299004T)
5.470..14
mitations per Stopes
60 (0.467
13.000.000
3,550 (0)
2.60021)
35 35.7690099 98)
12.40000)
3 30800.11 11)
57100.14
Method
Exrmpoaation
Fimessiu (land)
Distancuun
Diveriiy.on
Exinpelation
(modiam) Fiinessor:
Distmott
Divensiyyn
max mutations per stepoo
35.7756 (04)
3.91/0011
1.610.013)
3.1500.44
mas mututions per step- (defaalt)
(8.28%((.06)
88/0.011
1.400004
2.994007)
36.2256 00)
3.93/0011
1.560.05)
3.040.055
max justationssppp per stcpend
87.81 88)
3.99000111
1.47/0094)
2.5940105)
39.87 (1.72)
3.03/0.01)
1 1.56/0.06)
5.0913.05)
mas.musatims.ppr.cmm Surp
56. 19%:00.97)
3,88/0.00 01)
1.480.002
2,8710006)
38.269x(1311)
3,930001)
15700.04)
3.06/0.07)
27

---

Published as a conference paper at ICLR 2025
H
TRAINING CURVES FOR PREFERENCE LEARNING ALGORITHMS
H.1
MEDIUM DIFFICULTY SPLIT OF GFP
Undesied Desttd
-20
Undestedd
-30-
log probabiity
probabiilty
-30
-70
Steps
Steps
IPO
IRPO
Undestee
Undesteed
-1
-202
-22
-22
leg probability
probability
-212-
-27
-003
-225-
-35
-375-
-36
SPPO
Undested Distte
Undesteed
log probabiiig
-2
Steps
Steps
EXO
-30
Undested Deated
-81
log probabiiity
-40
-00
Figure 17: Log probability of validation set's desired vs undesired sequences for preference learning models on medium difficulty split of GFP.
28

---

Published as a conference paper at ICLR 2025
H.2
HARD DIFFICULTY SPLIT OF GFP
AOT
-28
Undesied Destte
-20
Undesteed
log probabiitty
probabillty
-48
-10
-30
Steps
Steps
IPO
IRPO
Undestee
Undesteed
-24
-32
log probability
probabiiity
-33
+20
-5
SPPO
Undested Distte
Undestedd
log probabiitt
24
log probabiitty
-70
-28
Steps
EXO
Undested Deated
log probability
-30
Figure 18: Log probability of validation set's desired vs undesired sequences for preference learning models on hard difficulty split of GFP.
29

---

Published as a conference paper at ICLR 2025
H.3
MEDIUM DIFFICULTY SPLIT OF AAV
AOT
Undesied Destte
Undesteed
-33
log probablitty
probabiltty
+25
-25
Steps
Steps
IPO
IRPO
Undestee
Undested
-75
log probability
probability
-15
-38
26-
SPPO
Undested Distte
Undesteed
-28
-10
log probablitt
probabiltty
-2
12
-2
-24
Steps
Steps
EXO
Undestee Deated
log probability
Steps
Figure 19: Log probability of validation set's desired vs undesired sequences for preference learning models on medium difficulty split of AAV.
30

---

Published as a conference paper at ICLR 2025
H.4 HARD DIFFICULTY SPLIT OF AAV
DPO
Undesied Destte
Undesteed
15.9
log probabiitt
-300
probability
-203
-22
250
25.0
-125
-305
884
Steps
5teps
IPO
IRPO
Undesind
14-
Undesteed
-35
log probabilityy
36-
probability
SPPO
-14
Undested Distte
Undesteed
-1
-16
log probabiiity
-38
log probabiitty
-10
-28
-4
-24
20-
Steps
Steps
EXO
Undested Deated
log probability
20-
-25
-20
Steps
Figure 20: Log probability of validation set's desired vs undesired sequences for preference learning models on hard difficulty split AAV.
31

---

Published as a conference paper at ICLR 2025
1 TRAINING CURVES FOR TRIPLET DATASETS
1.1 MEDIUM DIFFICULTY SPLIT OF GFP
Figure 21: Log probability of validation set's desired vs undesired sequences for various approach of creating triplets on medium difficulty split of GFP.
1.2 HARD DIFFICULTY SPLIT OF GFP
205
Sep
Comblined dhiflition
2000
Figure 22: Log probability of validation set's desired vs undesired sequences for various approach of creating triplets on medium difficulty split of GFP.
32

---

Published as a conference paper at ICLR 2025
1.3
MEDIUM DIFFICULTY SPLIT OF AAV
probeattee
Scoree diliain
Figure 23: Log probability of validation set's desired vs undesired sequences for various approach of creating triplets on medium difficulty split of AAV.
1.4 HARD DIFFICULTY SPLIT OF AAV
Figure 24: Log probability of validation set's desired vs undesired sequences for various approach of creating triplets on medium difficulty split of AAV.
33


Reviews:
[{"review_id": "t1UuHhmSnt", "reviewer": "Reviewer_jBZU", "summary": "In this work, preference learning on triplets of sequences has been proposed to improve the performance of prior edit-based methods for the task of protein design. In prior edit-based method(s), the conditional generative model is only trained on pairs of sequences where the likelihood is maximized for generating the sequence with higher property conditioned on the sequence with lower property. However, faced with three (or higher)-way relationships, the conditional generation is not guaranteed to move the optimization towards the sequence with the highest property. To mitigate this issue, the set(s) of sequence triplets are designed to further optimize the conditional generative model through preference learning losses.\n\nProtein optimization tasks (four in total) with varying degree of extrapolation difficulty (designed by prior work) have been used to demonstrate the effectiveness of preference learning on sequential design of proteins. The proposed method works better than prior protein design approaches in terms of extrapolation and finding higher fitness samples in three out of the four design tasks.", "questions": "1. How were the pairs with true scores generated? Has the same binning of fitness been used to generate pairs as well as the triplets? Were the pairs taken from two consecutive bins?\nIn section 3.3 there is no mention of using the training pairs with their original scores, while it is mentioned in section 4.3 (first paragraph). Make the two consistent to avoid confusion.\n\n2. Why are the triplets generated from consecutive bins? Is there an underlying assumption that the sequences coming from two adjacent bins are more similar than the ones further apart? The fitness landscape of protein does not necessarily behave like this. The fitness can have sharp variations in a local neighborhood of sequence space.\n\n3. How is a small meaningful improvement of score determined for the perturbed seed sequence (section 3.3, line 163). I am assuming that having a small improvement is the requirement of the conditional modelling used. How is a meaningful improvement identified? Also, how different can a perturbed sequence look from the seed sequence? Is there any restriction on the dissimilarity between the perturbed and seed sequence in the sequence space?\n\n4. In the main text you should clarify what $r_\\phi$ (reward) stands for.\n\n5. Please explain top-k and top-p sampling and the role of temperature in the inference.\n\n6. Is the default operating mode of EXO with the “All” triplet creation (Table 2)?\n\n7. (Minor). Change 2023 to 2024 in the citation for,\nImproving protein optimization with smoothed fitness landscapes. In The Twelfth International Conference on Learning Representations, 2023.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Demonstrating the impact of preference learning in extrapolative generative protein design.\n\n2. Nice ablation study on the impact of a) the strategies/criteria to create dataset of triplet sequences for preference learning, b) the type of preference learning approach adopted.", "weaknesses": "1. The authors have specified a separate category of methods for extrapolation, whereas by the definition of the paper in section 3.1, almost all the methods for protein optimization are doing extrapolation. They are all searching for sequences not existing in the train set that have properties beyond the range of training samples. The extent of extrapolation may differ from one work to another as discussed by  [1] and [2] that introduces the concept of “separation”, which can be seen as the extent of extrapolation, and the challenges associated with that in protein optimization tasks.\n\n2. Given that this work is for protein optimization, the related work section should contain a category for methods that perform optimization in the latent space which could be guided by property values as well. The famous work of [3] is one of these methods which has been included and discussed in many protein optimization benchmarks. Also, the recent method of [2] (published at ICLR 2024), encodes the _preference_ of sample generation into the continuous latent space of the generative model. This makes the optimization robust to the extent of “separation\", i.e., extrapolation, in the design tasks.\n\nReferences.\n1) Kirjner, Andrew, et al. \"Improving protein optimization with smoothed fitness landscapes. ICLR, 2024.\n2) Ghaffari, Saba, et al. \"Robust Model-Based Optimization for Challenging Fitness Landscapes. ICLR, 2024. \n3) Gómez-Bombarelli, Rafael et al. “Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hy4X3rWpZt", "reviewer": "Reviewer_P9DR", "summary": "In order to optimize a certain protein of interest, some preliminary source of supervised data is collected around the property of interest and some mutational landscape of that protein. In the effort to achieve a higher functional capability than those derived from the supervised set, protein designers will propose, express, and assay new designs in hopes that they can achieve more fit proteins than their initial candidate pool. To that end, this work uses some of the reward modeling techniques from preference optimization literature to improve a conditional protein generation model’s ability to extrapolate to unseen fitness values. It argues to use the exact preference optimization (EXO) framework with a mixture of hard negatives to improve the generative model’s capability to sample fit sequences.\n\nTo assess the contribution of different components, it ablates the choice of preference optimization method and triplet generation techniques. In addition, it studies the performance of methods that explicitly use and do not use the scorer during sampling.", "questions": "* How is the sampling with scorer guidance performed?\n* In what order were $\\beta$ and $\\tau$ determined?\n* What choices went into the selection of hyperparameters for the baseline models?\n* How does performance of the baseline models change when benchmarked with the same attention as the proposed model?\n\nBecause of the simplicity, this algorithm seems like an attractive solution for protein designers, but still has room to grow towards evaluation. To clarify, it does not need uniformly SoTA results to merit publication, rather reproducible experimental parameters to give confidence in performance differences. If work evolves to address the above questions, I’d happily raise my score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The model’s intervention is well motivated and deserves recognition for its simplicity. Preference optimization is easy to implement, and the results against non-scoring guided samplers make this an intervention that’s likely to be adopted. I can see this becoming a well adopted strategy for these reasons.\n\nIn introducing the intervention, the work does a good job in beginning to analyze what alterations are causal in increased performance. The experiments with respect to triplet curation and reward methods are well thought out and thorough. Each study is run with 5 replicates to ensure that performance is not a result of randomness. In addition, Figure 2 is quite compelling towards understanding EXO’s performance compared to other alternatives, while also illuminating the core goal of the project.", "weaknesses": "As the paper notes, all experiments are derived from a blending of wet-lab data and estimated wet-lab experimental outcomes. As such, there might be a mismatch between the benchmark outcomes and real world outcomes. This should not be counted against the work too much, but kept in the back of readers minds.\n\nThe two major areas for improvement in this work lie in elements of clarity and the derivation of baselines. Overall, the work asks the right questions, but the exposition in how the answers are derived leaves out critical details that makes it hard to have conviction in the whole outcomes.\n\nOn the note of clarity, the work finds certain key components unexplained. In particular, while dedicating an entire subsection to the role of sampling with a scorer, the details of how this performs are left unstated. Furthermore, the exact reasoning behind certain choices is left unsaid: why does sampling with a scorer have a different temperature and what is the maxim used to select this? As a minor nitpick, on line 284, the mentioning of $\\beta=0.1$ should clarify a preference loss $\\beta$ as it’s confusable with AdamW’s two separate $\\beta$s.\n\nMy largest source of concern in the work centers around how baselines are evaluated and the steps leading to a hyperparameter selection. Looking at Table 11, hyperparameter choice can mean the difference between a SoTA performance  and unusable model. With this in mind, and the explanations towards those chosen for the preference models, the leading question then becomes how sensitive are the baseline models to hyperparameter selection? The text does not explain how they were optimized.\n\nEven for certain choices in the paper, the statements: “[b]ased on the results shown in Figure 21 and Table 10, $\\beta = 0.1$ is performing well” and “[s]imilarly, based on Figure 22 and Table 11, $\\tau=0.7$ is performing well” led to confusion amongst what “performing well” means. Additionally, in what order were $\\beta$ and $\\tau$ chosen? Even if not a hard and fast rule it would aid the reader to understand which metrics were being balanced to make this decision. \n\nLastly, a more minor comment, the claims around Figure 3 should be hedged to take into account the stochastic nature of t-SNE and the relationships that they may imply. Both panels of the figure are challenging to read from the choice in symbols due to small size. It’s challenging to draw the same conclusion as the paper about the “supports by unseen ground truth extrapolation sequences.” This feedback seems more splitting hairs about a qualitative study so I don’t think it’s a strong critique, rather room for the wording to get a little qualification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W8GFDd0taN", "reviewer": "Reviewer_jZ4t", "summary": "The paper introduces a data distillation method focused on extrapolative protein design by applying triplet-based preference optimization. The approach, termed Exact Preference Optimization (EXO), aims to improve the generation of protein sequences with fitness values that exceed those observed during training. By aligning model training with structured triplet preferences, the authors seek to direct the model toward higher-fitness extrapolation regions. Performance is evaluated on Green Fluorescent Proteins (GFP) and Adeno-Associated Virus (AAV) datasets, with results suggesting improvements over existing methods in metrics such as extrapolation rate and top-100 fitness.", "questions": "1. The triplet generation process, especially when involving scorer-based distillation, appears computationally intensive. Could the authors provide more detail on the computational requirements for this step? Additionally, how would the method scale to larger datasets or to proteins with significantly different characteristics?\n\n2. The results are presented for GFP and AAV datasets. To what extent do authors believe the method could generalize to other proteins, especially those with less predictable fitness landscapes? Have the authors considered additional datasets, or are there plans to test the model on other types of proteins?\n\n3. The ablation studies show some performance sensitivity to triplet dataset construction. Could the authors elaborate on the specific criteria used to select the triplets, and whether the approach is robust to different triplet selection methods? Additionally, how dependent is the performance on the accuracy of the scorer used in the distillation process?\n\n4. The diversity metric suggests some degree of balance between exploration and exploitation, but it’s unclear how this affects overall model performance. Could the author clarify how the model’s exploration-exploitation balance was tuned or controlled, and how this may impact its ability to generalize in truly extrapolative regions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The method provides a unique approach to data distillation for extrapolative tasks, using triplet preferences instead of traditional pairwise comparisons. This formulation may better approximate the gradient direction toward higher fitness values, which is often challenging in extrapolative settings.\n\n2. The paper includes benchmark results against several existing methods, both scorer-based and non-scorer-based, which helps to establish a comparative baseline for their approach. The presented improvements in extrapolation metrics provide an indication that the model is capturing some aspects of fitness not fully addressed by baseline methods.\n\n3. Evaluation across multiple dimensions, including extrapolation rate and diversity among top sequences, gives a broader perspective on how the model navigates the trade-off between exploration and exploitation. This helps to contextualize the method’s potential in extrapolative design, though the gains remain somewhat dataset-specific.\n\n4. Ablation studies on different strategies for preference data construction are included, offering insights into the model’s sensitivity to triplet selection and shedding light on the specific design choices that contribute to observed performance improvements.", "weaknesses": "1. Generating triplet preferences, particularly when using scorer-based distillation, is computationally intensive and may limit the scalability of this method in practice. The reliance on costly preference data creation steps could hinder its application to larger datasets or more varied protein types.\n\n2. The paper’s evaluation is confined to GFP and AAV datasets. While they are common in the field, they may not fully represent the diversity of protein design challenges. The extent to which the method generalizes across other types of proteins remains unclear, and additional benchmark results would strengthen claims.\n\n3. The approach relies heavily on in-silico evaluators that are trained on subsets of the data, which could contain biases and limit the evaluator’s ability to accurately assess extrapolative performance. The model’s extrapolative capability, therefore, may not fully translate to real-world scenarios without further validation, such as wet-lab testing. This dependence on in-silico metrics calls into question the practical value of the observed improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r9FUfupmfn", "reviewer": "Reviewer_qbi3", "summary": "This paper addresses the problem of designing novel protein sequences with fitness values beyond those training ones. The paper distills triplet relationships into a generative model using preference alignment to better capture the fitness gradient than alternative pair-based approaches. This model is validated on AAV and GFP and outperforms state-of-the-art scorer-based and non-scorer-based baselines in extrapolation tasks.", "questions": "* Some parts of this paper are dense and difficult to read; in particular the definition of triplets (defined in different parts of the paper in sections 3.4; 4.3 and 6.1). W.r.t. scorer distillation, what is defined as a hard triplet ranking wise (for the vanilla version, hardness is defined by the local editor) ? The term \"scorer distillation\" is confusing (if I understood correctly the vanilla version also uses a scorer).\n* How important is the mask infilling step ? How frequently are the mask infilled samples represented in the challenging triplets (where the editor does not capture the right fitness ordering) ?\n\n\n====Post rebuttal====\nI thank the authors for the rebuttal; I recommend acceptance.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall this paper is a valuable contribution to the field.\n* The problem of designing proteins with fitness value beyond the training range is an important problem and challenging in protein design e.g. lead optimization; the proposed work has high practical value.\n* The proposed approach is well-motivated; the use of triplet-information with preference optimization for protein design makes a lot of sense and is (to my knowledge) new. The analysis of the importance of using difficult triplets is insightful. The experimental validation is thorough (including ablation studies) and shows that the proposed model outperforms alternatives in the extrapolation regime.", "weaknesses": "* The preference optimization relies on a scorer function to define the triplets; in real-world setting, training such scorers is difficult and the scorer is often used in out-of-distribution settings where it does not generalize well. Adding some discussion on this topic would be valuable to the paper as it is not discussed to avoid settings where the proposed preference optimization would degrade the base pairwise model. In particular, could the authors explain how robust is the preference fine tuning model to noisy labels? How to avoid pathological cases where difficult triplets are defined by an imperfect scorer ? \n* The presentation is at times confusing and difficult to understand (c.f. clarifying question).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)