A Variational Perspective on Generative Protein Fitness Optimization
Authors: Lea Bogensperger
Dominik Narnhofer
Ahmed Allam
Konrad Schindler
Michael Krauthammer
Venue: ICML 2025 poster (2025)
Keywords: protein design
variational methods
generative modeling
Primary area: applications->health_medicine

Abstract: The goal of protein fitness optimization is to discover new protein variants with enhanced fitness for a given use. The vast search space and the sparsely populated fitness landscape, along with the discrete nature of protein sequences, pose significant challenges when trying to determine the gradient towards configurations with higher fitness. We introduce *Variational Latent Generative Protein Optimization* (VLGPO), a variational perspective on fitness optimization. Our method embeds protein sequences in a continuous latent space to enable efficient sampling from the fitness distribution and combines a (learned) flow matching prior over sequence mutations with a fitness predictor to guide optimization towards sequences with high fitness. VLGPO achieves state-of-the-art results on two different protein benchmarks of varying complexity. Moreover, the variational design with explicit prior and likelihood functions offers a flexible plug-and-play framework that can be easily customized to suit various protein design tasks.

A Variational Perspective on Generative Protein Fitness Optimization
Lea Bogensperger
Dominik Narnhofer? Ahmed Allam
Konrad Schindler 2 Michael Krauthammer'
Abstract
tory with a time-consuming, yet narrow random exploration of the fitness landscape. This highlights the need for effi- cient in-silico methods capable of exploring the space of potential sequences, so as to prioritize promising candidates for experimental validation.
The goal of protein fitness optimization is to dis- cover new protein variants with enhanced fitness for given use. The vast search space and the sparsely populated fitness landscape, along with the discrete nature of protein sequences, pose sig- nificant challenges when trying to determine the gradient towards configurations with higher fit- ness. We introduce Variational Latent Genera- tive Protein Optimization (VLGPO), a variational perspective on fitness optimization. Our method embeds protein sequences in a continuous latent space to enable efficient sampling from the fitness distribution and combines (learned) flow match- ing prior over sequence mutations with fitness predictor to guide optimization towards sequences with high fitness. VLGPO achieves state-of-the- art results on two different protein benchmarks of varying complexity. Moreover, the variational design with explicit prior and likelihood functions offers flexible plug-and-play framework that can be easily customized to suit various protein design tasks.
Many different approaches exist, from generative mod- els (Jain et al., 2022; Gruver al., 2024; Notin et al., 2022), evolutionary greedy algorithms (Sinai al., 2020) gradient-based sampling strategies, such as Gibbs With Gra- dients (GWG) (Grathwohl et al., 2021; Emami et al., 2023) and smoothed GWG variants (Kirjner et al., 2023). In gen- eral, these methods are challenged by the high-dimensional, sparse and discrete nature of the fitness landscape, charac- terized by ruggedness (Van Cleve Weissman, 2015) due to epistasis as well as holes due to proteins with very low fitness (Johnston et al., 2023; Sinai & Kelsic, 2020). To deal with the inherently discrete nature of amino acids, compu- tational methods tend to rely on discretized processes, and thus must cope with issues such as non-smooth gradients and the vast diversity of possible sequences (Kirjner et al., 2023; Frey et al., 2023).
Here we look at the problem from different perspective: instead of relying on token- based sequence representation, we embed sequences in continuous latent space and learn the corresponding latent distribution in generative manner. In this way, patterns and relationships can be captured that are difficult to model in the original discrete space, espe- cially if one only has access small data set containing only a few thousand mutations of protein.
1. Introduction
Protein fitness optimization seeks to improve the function- ality of a protein by altering its amino acid sequence, to achieve a desired biological property of interest called "fit- for instance, stability, binding affinity, or catalytic efficiency. It requires searching through vast combina- torial space (referred to as the "fitness landscape"), where the number of possible sequences grows exponentially with the sequence length d, while only a small subset of these sequences exhibit meaningful biological functionality (Her- mes al., 1990). Traditionally, protein fitness optimization has been addressed through directed evolution (Romero & Arnold, 2009), mimicking natural evolution in the labora-
To construct a prior distribution over latent protein se- quences, we leverage flow matching (Lipman et al., 2023; Liu et al., 2022), a powerful generative modeling scheme that learns smooth, continuous representations amenable to gradient- sampling and optimization. The prior is then integrated into a variational framework, making it possible to guide the search towards regions of high fitness with fitness predictor in the form of neural network, trained limited set of sequence mutations with associated fit- ness labels. See Figure 1. The versatility of the variational framework means that it can easily be tuned to different protein optimization tasks by suitably adapting the prior and the guidance function. We validate VLGPO on two public benchmarks for protein fitness optimization in limited data
Universitty of Zurich RETH Zurich. Correspondence to: Lea Bogensperger clea.bogenspergeree uuhh.g uzh.ch.>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Variational Perspective Generative Protcin Fitness Optimization
(2)(
At vor(#r)
At var(
0.0
0.4
0.8
1.0
[(x=)
p(æ)
Enc
Dec
p v))
vga(2))
At var(=i)
At (=((=))))
p(y(æ) ~ 3 lasta) all
V. Ply P(x))
P(y
p()P)(a))
t-00 0.0
0.4
0.8
1.0
Figure 1. Overview VLGPO sampling. The central section illustrates the VAE framework, showcasing protein sequences, their latent representations and the approximate posterior distribution. While the upper section depicts unconditional sampling from the prior p(x) using flow matching in the latent space, the lower section illustrates the modifications introduced by VLGPO during sampling. We additionally incorporate likelihood term 2(y]æ)) to condition the fitness y. enabling sequence generation from the posterior distribution p(xly)) and facilitating sampling from high-fitnes regions (as shown by the shifted and reshaped distribution).
regimes, namely Adeno-Associated Virus (AAV) (Bryant et al., 2021) and Green Fluorescent Protein (GFP) (Sark- isyan et al., 2016), as suggested by (Kirjner et al., 2023).
2. Related Work
We consider in-silico methodologies for protein fitness op- timization the sequence level, while emphasizing that there is an extensive stream of research centered around active learning frameworks, which iteratively integrate com- putational predictions with experimental validations (Yang et al., 2025; Lee et al., 2024). While these methods are often considered the gold standard due to their integration of ex- perimental feedback, our contribution is confined purely computational strategies, thereby complementing existing active learning approaches.
To summarize, our contributions are:
We introduce Variational Latent Generative Protein Optimization (VLGPO), variational framework for protein fitness optimization that enables posterior sam- pling of protein sequences conditioned on desired fit- ness levels. Our method combines learned prior and a fitness predictor to effectively guide the optimization towards high-fitness regions.
We perform fitness optimization in continuous latent representation that embeds meaningful relations be- tween discrete protein sequences. Within this latent space, we employ flow matching to learn generative prior that enables efficient exploration of the fitness landscape and thus facilitates the discovery of high- fitness sequences.
In typical directed evolution setup (Romero & Arnold, 2009), biologists simulate nature in the laboratory by run- ning multiple rounds mutagenesis, searching through the local landscape by sampling few mutations away from the current position in the sequence, effectively discovering sequences through random walks in the sequence space. This process can be resource-intensive do not enhance the fitness of the starting sequence, and the vast number of combinations makes exhaustive exploration impractical. Therefore, many techniques have been intro- duced to address these challenges by employing surrogate models guide the search more efficiently (Sinai al., 2020; Brookes al., 2019; Trabucco al., 2021; Ren et al., 2022). For instance, Adalead (Sinai et al., 2020) uses Blackboxx predictor model to inform greedy algorithm to prioritize mutations that are more likely to improve protein fitness.
many mutations
We demonstrate state-of-the-an performance on es- tablished benchmarks for protein fitness optimization, namely AAV and GFP, targetting in particular tasks of medium and high difficulty in limited data regime!.
'Source code available https: //gtthbb.. hhu......tt.... com/ wuzh wazh-ddpm-cmi- /VLGO..
2

---

A Variational Perspective on Generative Protcin Fitness Optimization
Moreover, generative methods have been explored for sequence generation. For instance, GFlowNets (GFN- AL) (Jain et al., 2022) are designed to sample discrete objects, such as biological sequences, with probabilities proportional to a given reward function, facilitating the generation of diverse and high-quality candidates. Further, discrete diffusion models have been used for sequence sam- pling, where gradients in the hidden states of the denoising network help to guide the sequence design (Gruver al., 2024). Alternatively, a walk-jump algorithm was proposed to learn the distribution of one-hot encoded sequences using a single noise level and a Tweedie step to recover the sam- pled sequences after MCMC sampling (Frey et al., 2023), which was further extended using gradient guidance in the noisy manifold (Ikram et al., 2024). Likewise, also GWG proposes a strategy to obtain gradients for MCMC sam- pling of discrete sequences (Grathwohl et al., 2021; Emami et al., 2023). The method Gibbs sampling with Graph- based Smoothing (GGS) builds upon this by additionally regularizing the noisy fitness landscape with graph-baac based smoothing (Kirjner et al., 2023).
specific fitness metric y= == f() with y R. which quanti- fies desired protein functionality such stability, activity, or expression.
Therefore, we work with paired data S = {( } (note that S is only a subset of the entire data S see Table 2) and we use the parameterized convolutional neural network (CNN) g: Vd + R (Kirjner et al., 2023) to infer the fitness for a given sequence. Specificallyy, we employ g and 9% as predictors, which are trained on small subset of the data (see Section 4.1) without and with graph-based smoothing, respectively. Additionally, we use Io as an in-silico oracle for the final evaluation, trained on the entire paired data set S Note that all models share an identical architecture, differing only in their respective weights which we re-use without further training.
Building on the predictive framework for estimating protein fitness, we now turn our attention to creating new sequences that may exhibit desirable properties. Generative modeling provides a powerful approach for navigating the vast sequence space, offering a data-driven way to propose candidate proteins beyond those observed in the training set. In the following, we introduce generative models and discuss how they can be leveraged to discover novel protein sequences with optimized fitness.
Since the search space of protein sequences grows with se- quence length, many works have started to recognize the significance of latent space (Praljak et al., 2023) or em- bedding spaces that are used in large-scale protein language models (Lin et al., 2023). LatentDE (Tran et al., 2024) for instance combines directed evolution with a smoothed space which allows to employ gradient ascent for protein sequence design in the latent space guided by a fitness predictor.
3.2. Generative Modeling
A recent class of generative models, known as diffusion models (Ho et al., 2020; Song et al., 2021; Sohl-Dickstein et al., 2015), has shown remarkable success in generating high-quality data across various domains. These models work by progressively transforming simple noise distribu- tions into complex data distributions through a series of iter- ative steps. During training, noise is systematically added to the data sample x at varying levels, simulating degradation process over time t. The model cp is then tasked with learn- ing to reverse this process by predicting the added noise for Ty at each step, effectively reconstructing the original data from the noisy observations. In detail, a model is trained to predict the added noise at each step by minimizing the objective
Within the diverse array of approaches protein fitness opti- mization, we propose a variational perspective that is known for its efficacy in other domains like inverse problems and image reconstruction. Specifically, we integrate a generative flow matching prior that learns the distribution of protein sequence mutations from data set of protein sequences and combine with a fitness predictor. The integration of this predictor helps to effectively guide sampling process that generates high-fitness samples. Moreover, we address the problem within compressed latent space, encoding protein sequences into a latent representation that facilitates contin- uous optimization techniques based on gradient information. This approach enables efficient exploration of the protein fitness landscape, leveraging the latent space to perform guided sampling directly on the encoded sequences.
[()) = Mo [[le- flc--cl co (Z,, 0)(]
Flow Matching. A more recent approach to generative modeling is given by the versatile framework of flow match- ing (Lipman et al., 2023; Liu et al., 2022). Rather than removing noise from data samples, flow matching aims to model the velocity of the probability flow V.,, which gov- erns the dynamics of how one probability distribution is transformed into another over time. By learning the velocity field in of the probability flow, the model vg captures the evolution of a simple base distribution at into a more
3. Method
3.1. Protein Optimization
Let æ E Vd represent a protein sequence, with dimension- ality d corresponding to the number of amino acids chosen from a vocabulary of 20 amino acids. Protein fitness optimization seeks find sequence æ that maximizes
3

---

A Variational Perspective on Generative Protcin Fitness Optimization
complex target distribution p(z) at 1 1, directly model- ing the flow between them. Since the velocity field oe is intractable, it was shown in (Lipman et al., 2023) that we can equivalently minimize the conditional flow matching loss:
2023). Note that for guiding the process towards the highest fitness, y is simply set to which represents the highest fitness in the normalized fitness spectrum.
3.4. Latent space representation
So far, we introduced general framework that in theory allows for sampling from learned distribution of protein sequences. However, these sequences represent proteins that are combinations of a discrete set of amino acids. As a re- sult, the underlying distribution of these sequences is likely sparse and complex, making it difficult to approximate or directly sample from in its original form. To overcome this limitation, we operate in an embedded latent space, where protein sequences are encoded as continuous representa- tions. This is achieved by a VAE framework (Kingma & Welling, 2022), which maps discrete sequences to a con- tinuous latent space through an encoder & : V -> IR' and reconstructs them back to the original sequence space us- ing a decoder network D R R VI, where << d. Because of the discrete nature of amino acid tokens, the de- coder produces logits, which are then mapped tokens via an argmax operation. The training objective of the employed 8- VAE (Higgins et al., 2017) is given by the weighted Evi- dence Lower Bound (ELBO):
min E EI, (2 - (21 x.)ll"]
(1)
where t ~ 10 I1 p(z) and zo No. and the conditional flow is given by Vy (=)) = (1-t)rg +t Once trained, samples can be generated by numerical integration of the corresponding neural ordinary differential equation (ODE) with t E [0, 1]:
d di
(1(x)= = (o,e(V,
(2)
Our aim is learn flow-based generative model that ap- proximates the distribution of sequenee variants of protein p(r). We then seek to leverage this model in order to gen- erate sequence variants with high protein fitness through guidance, as explained in the following.
3.3. Classifier guidance
Diffusion and flow-based methods can be guided by the log- log-likeldhood gradient of an auxiliary classifier during gen- cration, which enables conditional sampling (Dhariwal & Nichol, 2021). In detail, our goal is to sample from a con- ditional distribution p(z|go (ar) - y). where go represents the predictor and y denotes the desired fitness value. We therefore adopt the framework of classifier guidance which allows for a decomposition of p(æly)) ~ p(y pz))) Thus, we guide the sampling process by using the gradient of the predictor (z) with respect to the input sequence x. By introducing this gradient into the generative process, we can bias the sampling trajectory towards regions of the distribu- tion that are more likely to yield sequences with the desired fitness value. The velocity field to,t in the generative frame- work is modified to incorporate this guidance, yielding the following variational update:
min Be [s|=) - log Pa (x|=) =) (LT (4) 2.
where due to the discrete tokens log (xl2)) simplifies to the cross-entropy loss in our case.
Note that while the VAE is trained with variational inference, VLGPO goes further by introducing additional generative modeling components: flow matching as descrihed in Sec- tion 3.2 and classifier-guides sampling in Section 3.3. We show how these extensions refine the generation and help to better control it.
3.5. Variational Latent Generative Protein Optimization
From here on we will use the building blocks introduced ear- lier to describe our proposed VLGPO approach. We start by using our respective sequence data p(x:) to train a VAE with encoder E and decoder DD that compresses the higher- dimensional discrete protein sequences into a continuous latent space representation ~ p(z[æ:) In order to model and sample from the learned latent space in an effective way, we train a flow matching model vog, to learn the probability flow dynamics in the latent space. This network captures the transformation between simple base distribution (c.g.. Gaussian noise) at = 0 and the complex latent distribution of protein sequences at L = 1. By integrating the learned flow from t E [0, 1]. we can efficiently generate new latent representations 2 which can subsequently be decoded back into protein sequences via the VAE decoder, resulting in x ~ (a (m121) = D(=1), similar to (Esser et al., 2024).
1 (z) + as V. log z))
where Vz log ('gg)( - V vl? represents the gradient of the log-likelihone of a sequence x having desired fitness and 0g is a scheduler dependent constant (Zheng et al., 2023). Given the goal to maximize the fitness value of generated sequences, one could also set the gradient of the log- log-likelihood to 2 V_ (x) Both approaches perform very similar in practice. In this work, we chose the first version to demonstrate the possibil- ity of steering sequences toward specific fitness values.
similar suitable form.
For the classifier guidance we use the trained CNN-based predictors Is and the smoothed 93 from (Kirjner et al.,

---

A Variational Perspective on Generative Protcin Fitness Optimization
Algorithm 1 VLGPO sampling
4. Experiments
Require: K, J, y, 04
4.1. Data Sets
1: Initialize 20 p=(2) 10)
2: Set step size At < 1
We adopt the medium and hard protein optimization bench- marks on AAV and GFP from (Kirjner et al., 2023). Given the full data set S the task difficulty is determined by (i) the fitness percentile range of the considered sequences (20 40 for medium, < 30 for hard) and (ii), the required gap of mutations to reach any sequence of the 99th fitness percentile of S (a gap of 6 mutations for medium, 7 muta- tions for hard), see Table 1.
k
3: for k = 0to K- - 1 do
4: +
5:
of = + At vg,)
6:
j = to
do
7:
1Ι 4 2 + (1 - 1-- A1) (())o,((=() (={)
8:
2/ x 2 2 V (1l19((((8))) y/1
9:
end for
10:
E++-1 + =/z
11:
end for
12:
return x = (211)
Table I. Task definition.
Task
Range %
Gap
Medium
20-40
6
Sampling from the posterior. Instead of merely sampling from the sequence distribution p(æ), also seek gen- erate high- high-fitness sequences. To achieve this, we condition the sampling process on a fitness score y, ideally set to the maximum value = 1, which is estimated by predictor ÿ = (x).
Hard
30
7
Together, this results in four different tasks described in Table 2, each of which only sees a limited number of N sequences in limited fitness range. The idea of protein fitness optimization is to enhance these sequences to higher, previously unseen fitness values. The setting reflects re- alistic scenarios in terms of data set sizes. The diversity (Appendix A.2) in the training data sets of the four tasks is quite high: for GFP (medium) it is 14.5, for GFP (hard) 16.3, for AAV (medium) 15.9, and for AAV (hard) 18.4. On the other hand, the diversity within the top-performing (99th percentile) sequences of the full data set S* is 4.73 for GFP and 5.23 for AAV.
Although incorporating the likelihood term into the gradient, as shown in Equation (3), may appear straightforward, it is actually challenging because it becomes intractable due to the time-dependent nature of the diffusion//loww model. Moreover, explicitly including this term can push the gener- ated samples off the current manifold related to time point t (Chung et al., 2022). Inspired by (Chung et al., 2023; Ben- nen-Hamu et al., 2024), we employ a scheme that evaluates the likelihood d while the gradient of the likelihood is calculated at Ty, which has the effect of constraining the update to the same manifold. Therefore, at each of the K sampling steps the model estimates 2 (Line 7, Algorithm 1) which is decoded to fi. the denoised version of the current decoded sample 2t, using the learned flow model up.u- The likelihood is then evaluated at 91 to reflect the target data distribution. However, the gradient of the likelihood, which guides the generative process, is computed with respect to It, leading to backpropagation through va,t- Furthermore, as the predictor ds was trained in sequence space, our likeli- hood function changes to
Table 2. GFP and AAV data sets with number of data samples N. median normalized fitness scores and fitness range.
Task
N
Fitness f
Fitness Range
GFP Medium
2828
0.09
[0.01, 0.62]
GFP Hard
2426
0.01
[0.0, 0.1]
AAV Medium
2139
0.32
[0.29, 0.38
AAV Hard
3448
0.27
[0.0, 0.33]
For in-silico evaluation of the generated sequences with the oracle gv, we use the median normalized fitness, di- versity and novelty following (Jain et al., 2022), see Ap- pendix A.2. The oracle was trained on the complete DMS data with 56,086 mutants for GFP and 44,156 mutants for AAV. While diversity and novelty are reported in evaluation, no definitive higher or lower values are considered superior for a sequence. Note that Umin and ymax from the entire data set S* are used for both GFP and AAV to normalize fitness scores to [0, 1).
I V (D()))))())) v1
v log pogl())
(5)
where we additionally have to backpropagate through the decoder. The pseudocode of our VILGPO can be found in Algorithm 1, detailed visualization of the sampling scheme is reported in Figure 2. The hyperparameters J and ar denote the number of gradient descent steps on the likelihood and the guidance strength, respectively.
5

---

A Variational Perspective on Generative Protcin Fitness Optimization
()(()))) =v
g6 (b(()) =
At vas(21) (4) Vn a ((z))
At (2))
t= 0.0
t= 0.0
Va,
t At) v0,)(2)
1
== 0.5
t= 0.5
- 1.0
1 t= 1.0
Figure 2. Schematic depiction of classifier guidance, with J= and K = 6. Grey lines represent the latent manifolds at different time steps f, the blue line marks the trajectory of the maximum likelihood. Solid arrows indicate how the latent evolves over time. Left: Naive guidance with likelihood gradients v., computed directly pushes the sample off the manifold. This error accumulates, as indicated by the purple regions. Right: Guidance with manifold constraint, as employed in VLGPO (Algorithm 1), converges to valid sequence with fitness y. Solid arrows again denote the evolution of the latent, dashed arrows indicate the flow posterior sampling scheme that ensures the latent stays on the manifold when applying the likelihood gradient.
4.2. Implementation Details
each setting, used for classifier guidance and for ranking the samples, are trained only on the data sets listed in Table 2.
For training, we start by learning the VAE to compress the sequence token input space (d = 28 and d = 237 for AAV and GFP) to / = 16 and | = 32. A learning rate of 0,001 with convolutional architecture and B E {0.01, 0.001 for AAV and GFP is used for training the encoder E and decoder D in Equation (4), also see Appendix B.2. To learn the prior distribution of embedded latent sequences flow matching, the ID CNN commonly used for denoising diffusion probabilistic models (DDPMs)² is employed. A learning rate of 5e-5 and a batch size of 1024 were used to train ug for 1000 epochs. All VAE and flow matching models are always trained under the limited-data setting as listed in Table 2.
For evaluation of the generated samples, we use the or- acle Ie as in-silico fitness estimate, see Appendix A.1. The oracle is directly sourced from (Kirjner et al., 2023) and is the only model that was trained using the en- tire data S For sampling following Algorithm 1, we use Ie and compare VLGPO to GWG (which also uses the same trained predictor), and the identical smoothed predictor I3 in VLGPO to GGS (Kirjner et al., 2023). We benchmark against the respective baselines, namely GFlowNets (GFN-AL) (Jain et al., 2022), model-based adap- tive sampling (CbAS) (Brookes et al., 2019), greedy search (AdaLead) (Sinai et al., 2020), Bayesian optimization (BO- qei) (Wilson et al., 2017), conservative model- opti- mization (CoMS) (Trabucco et al., 2021) and proximal ex- ploration (PEX) (Ren et al., 2022). Morcover, we investigate the performance of the recently introduced gradient- -uided walk-jump sampling algorithm (gg-dWIS), which extends the walk-jump sampling framework originally developed for antibody sequence design (Ikram et al., 2024; Frey et al., 2023). Because the available source code was not directly compatible with the data sets in Table 2, we adapted our existing model architectures for implementation. We then performed grid search over sampling parameters, followed by top-k sampling for each task, to ensure a fair comparison.
{(x) using
At inference time, we follow the procedure outlined in Algo- rithm 1 to generate samples. The predictors Is and g3 (for details, see Appendix A. are re-used without any further training (Kirjner et al., 2023). We start from žu ~ (0, I) and use K = 32 ODE steps to integrate the learned flow until we obtain ø1. To optimize sequence fitness, the con- dition y
is selected for all samples. The parameters ae and J are determined via hyperparameter search, shown in Figure 3 and discussed in detail in Appendix B.4. Ultimately, they modulate the trade-off between increas- ing fitness and maintaining diversity. After generating 512 samples 21 to encourage sampling from the entire learned distribution, they are decoded using x D(21). Potential duplicates are then filtered out, and the top-k (k 128) samples, ranked by the predictor (go 93. respectively), are selected. Note that also the predictors 96 and 93 for
4.3. Results
We compare VLGPO as illustrated in Algorithm I for the four tasks in Table 2 by averaging over five seeds and com- puting the median normalized fitness, diversity and novelty (see Appendix A.2). The results reported in Table 3 and Ta-
'tttps:// com http://ttuub.ccm//uucccddaaaa lucidrains/ cfnoising-dlfffusoon--r--rr
6

---

A Variational Perspective on Generative Protcin Fitness Optimization
0042
4112
0.56
35
4.4
0344
40110
0343
0,344
0.66
0342
0,342
0001
0188
6039
balaniz parnece
(a) GFP medium
(b) GFP hard
(c) AAV medium
(d) AAV hard
Figure 3. Grid search for median fitness depending on sampling parameters de and J for the different tasks using the predictor Po. In general, higher values of au and J. corresponding strong classifier guidance, yield higher predicted fitness values.
ble 4 for GFP and AAV respectively, demonstrate the fitness improvement of VLGPO with both predictors (gs and 92) over all other benchmarked methods. In particular, VLGPO shows clear fitness improvement over GWG (which uses the same predictor 96) and GGS (which uses the same predictor gz). Moreover, the performance difference between our method and GWG (case of using non non-smoone -mooohed predictor go) further highlights the robustness of our approach in lim- ited data regimes and supports the advantage of the guided flow matching prior in the latent space.
(analogous to our predictor ga) is trained in the noisy one- hot-encoded space of sequences with single noise level, which limits its predictive capabilities.
4.4. Fitness Extrapolation
Table 2 illustrates the limited fitness range of the sequences in each task, whereas protein fitness optimization aims to sample sequences with higher fitness values. Because the available data set is small, extrapolating to these high-fitness sequences during sampling becomes especially challenging. The following experiment examines how much the oracle- evaluated fitness ygt. (provided by 9 deviates from the target fitness y. which is used as an input the sampling in Algorithm 1. Thereby, our variational method VL.GPO and directly learned poste- rior p(z|y) via fitness-ondditioned flow matching (z, v). The learned posterior does not need the predictor 96 for clas- sifier guidance, as it learns the conditional distribution in the latent space end-to-end. For fair comparison, the raw output without top-k sampling is applied. The results are depicted in Figure 4 for GFP and AAV hard.
When it comes to diversity and novelty, there is no clear definition of what is better - higher or lower scores (Kirjner et al., 2023). In general, we observe that reported values of VLGPO are in line with competing methods. Moreover, the variations between different methods in terms of diversity and novelty are significantly larger in GFP than in AAV. That discrepancy may arise due to GFP sequences being longer, thus representing a sparser and higher-dimensional search space that makes the protein harder to optimize. Inter- estingly, the sequences in the 99th fitness percentile, which are on the top end of figh-fittesss sequences, show a diversity of 4.73 for GFP and 5.23 for AAV, which are approximately comparable our results in Table and Table 4, except for AAV hard. In practice, the smoothed predictor 93 reduces the diversity within the generated sequences for both GGS and VLGPO. Additionally, more samples tend to collapse to the same decoded sequence, likely due to the smoother, less distinct gradients.
compare two approaches:
Due to the sparse data availability and the limited fitness range, the evaluated fitness Vgt cannot be expected to pre- cisely follow the required fitness Nevertheless, the results in Figure clearly demonstrate the advantage of classifier guidance in VLGPO, whereas the direct posterior fails to effectively exploit the given fitness condition. This gap is most evident in the GFP hard task, where the training data's s fitness range is only in [0.0, 0.1], making it difficult for the learned posterior to extrapolate to higher fitness values. In contrast, VI.GPO leverages the additional gradient infor- mation from the predictor Jø- thereby overcoming these limitations in higher-fitnes regions. This highlights the advantages of using separate classifier for guidance in domains where classifier-fre guidance struggles to produce high- fitness sequences.
While gg-dWJS is conceptually similar to VLGPO, its per- formance on both GFP and AAV (hard) does not fully align with findings. We hypothesize two main reasons for this discrepancy. First, the GFP data set (Table 2) is rel- atively small and limited to mutations of a single protein, posing a challenge for methods that rely on one-hot en- coding. Although gg-dWIS performs well on AAV, the one-hot-encoded space becomes extremely sparse for the longer GFP sequences (d 237), causing many generated samples to collapse onto identical sequences. Second, the discriminator model acting as the guidance for gg-dWJS

---

A Variational Perspective Generative Protcin Fitness Optimization
Table GFP optimization results. Best for fitness in bold, second-best underlined, and the results for method (VIGGP)
highlighted in grey.
Medium difficulty
Hard difficulty
Method
Fitness
Diversity
Novelty
Fitness 1
Diversity
Novelty
GFN-AL
0.09 = 0.1
25.1 + 0.5
213 + 2.2
0.1 + 0.2
23.6 + 1.0
214 + 4.2
CbAS
0.14 = 0.0
9.7 1.1
7.2 0.4
0.18 + 0.0
9.6 1.3
7.8 + 0.4
AdaLead
0.56 + 0.0
3.1
2.0 0.0
0.18 +: 0.0
5.6 0.5
2.8 + 0.4
BOqei
0.20 + 0.0
0.0
0.0
0.0 0.5
94.6
54.1 + 81
CoMS
0.00 = 0.1
25
192
0.0
1.5
201 3.0
PEX
0.47 = 0.0
3.0
1.2
0.0
0.0
1.3
gg-dWJS
0.55 + 0.1
5..3
16.3 5.7
0.61 0.1
44.8 + 47
GWG
0.10 = 0.0
+3.0
12.8 + 0.4
0.0
4.0
7.6 = 1.1
VLGPO, predictor 9s
0.87 = 0.0
4.31 0.1
6.0 - 0.0
0.75 0.0
3.1
6.0 = 0.0
GGS
0.76 = 0.0
3.7
5.0 = 0.0
0.74 0.74 0.0
3.6 0.1
8.0 = 0.0
VLGPO, smoothed O3
0.84 + 0.0
2.06 + 0.1
5.0 + 0.0
0.78 + 0.0
2.5 0.2
6.0 = 0.0
Table AAV optimization results. Best for fiteess in bold, second-best underlined, and the results for method (VI.GPO) highlighted in grey.
Medium difficulty
Hard difficulty
Method
Fitness
Diversity
Novelty
Fitness f
Diversity
Novelty
GFN-AL
0.20 + 0.1
9.60 + 1.2
19.4 + 1.1
0.10::0:
11.6 + 1.4
19.6 = 1.1
CbAS
0,43 = 0.0
12.7 + 0.7
7.2 + 0.4
0.36
14.4 + 0.7
8.6 = 0,5
Adaleead
0,46 + 0.0
8.50 0.8
2.8 0.4
0.40
8.53 + 0.1
3.4 0.5
BOqei
0.38 = 0.0
15,22 0.8
0.0
0.0
1.3
0.0
CoMS
0.37 0.1
10.1 5.9
8.2
0.2
10.7
10.0
PEX
0.40 0.0
2.80 +.0
1.2
0.3
0.0
1.3
gg-dWJS
0.48 + 0.0
9.48 + 0.3
4.2:0.44
0.33
14.3 0.7
5.3 0.4
GWG
0.43 0.1
6.60 6.3
7.7 0.8
0.33 +.0
12.0 0.4
12.2 - 0.4
VLGPO, predictor ga
0.58
5.58
5.0 + 0.0
0.51 0.0
8.44 0.2
7.8 + 0.4
GGS
0.51 + 0.0
4.0 + 0.2
5.4 + 0.5
0.60 + 0.0
4.5 + 0.5
7.0 + 0.0
VLGPO, smoothed 93.
0.53 + 0.0
4.96 + 0.2
5.0 0.0
0.61 - 0.0
4.29 + 0.1
6.2 0.4
4.5. Ablation Studies
Table 5. Influence of manifold constrained gradient in sampling for both predictors Jo and smoothed 93 and directly learned posterior for GFP medium and hard tasks.
We conduct an ablation study on the influence of manifold constrained gradients in sampling (Line 7, Algorithm 1). The results Table 5 and Table demonstrate the improve- ment gained by estimating Φ to compute the gradient of the likelihood term.
Medium Fitness
Hard
Method
Fitness t
VLGPO, predictor Is w/o manifold constraint
0.87 0.0
0.75 i 0.0
0.81 0.0
0.73 + 0.0
Further, the performance of the directly learned posterior p(rly) was investigated. While this approach still performs well for the medium tasks, it shows a larger performance drop on GFP (hard), indicating that it cannot match the explicit likelihood guidance provided by the fitness predictor in our variational method VLGPO.
learned posterior
0.83 + 0.1
0.44 : 0.1
VLGPO, smoothed 95 w/o manifold constraint
0.84 - 0.0
0.78 + 0.0
0.84 + 0.0
0.67 + 0.1
8

---

A Variational Perspective Generative Protcin Fitness Optimization
learned posterior p(()g)
leurned posterior p(a)p)
leamned posterior p(z(g)
0.6
VLGPO
VLGPO
VLGPO
0.8
0.8
0.5
0.6
0.4
Ver
0.4
0.4
0.3
0.2
0.2
0.2
0.0
0.0
0.1
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
V
(a) GFP medium
(b) GFP hard
(c) AAV hard
Figure 4. Comparing evaluated fitness yet from the oracle Os with required fitness y using the directly learned posterior model (in the same latent space) and our variational approach VLGPO.
which can be prone to overfitting given the limited size of employed data sets.
Table 6. Influence of manifold constrained gradient in sampling for both predictors Do and smoothed 93 and directly Iearned posterior for AAV medium and hard tasks.
A limiting factor of our method lies in its hyperparameter tuning requirements. We observe that hyperparameter selec- tion becomes more critical for challenging tasks such as GFP (hard), while it remains stable and robust for other tasks. Ad- ditionally, the restriction of the benchmark to only AAV and GFP is a limitation that should be addressed in future work. Conceptually, VI.GPO as well as competing approaches can be extended to other proteins from FLIP (Dallago et al., 2021) or ProteinGym (Notin al., 2023).
Medium Fitness f
Hard
Method
Fitness
VLGPO, predictor Jo
0.58 + 0.0
0.51 + 0.0
w/o manifold constraint
0.55 1 0.0
0.47 0.0
learned posterior
0.55 + 0.0
0.44 + 0.0
VLGPO, smoothed 93 w/o manifold constraint
0.53 + 0.0
0.61 0.0
0.52 = 0.0
0.58 = 0.0
Another important point is our reliance on in-silico eval- uation, where we follow (Kirjner al., 2023) in using trained oracle as the ground truth. This oracle was trained on significantly larger data set than the tasks presented in Table 2 and can therefore be expected to serve as good es- timator. Nevertheless, actual experimental validation could provide valuable insights into the applicability of VIGGPO. A complementary direction may to design more idealised, synthetic in-silico benchmarks that are nevertheless good proxies for protein design (Stanton et al., 2024). Morcover, additional metrics such as folding confidence or structural stability could be added to obtain a more complete perspec- tive beyond the fitness score (Johnson et al., 2023).
5. Discussion
We present VLGPO, variational approach for protein fitness optimization that enables posterior sampling of high-fitness sequences. Our method operates in learned smoothed latent space and learns a generative flow match- ing prior that imposes natural gradient regularization, re- moving the need for extra smoothing. Additionally, it in- corporates likelihood term through manifold-constrained gradients that helps guiding the sampling process towards high-fitness regions. VLGPO achieves clear fitness improve- ments, with respect to the fitness of training sequences, but especially when compared to all baseline and competing methods.
A Acknowledgements
The variational framework offers a versatile approach, as its modular design allows replacing any of its components as needed, such as the prior model, the predictor, the architectures. Future work could explore using embeddings from pretrained protein language models (pLMs) instead of the VAE, since such embeddings provide a more expressive latent representation. However, this would require finetun- ing the decoder to ensure faithful sequence reconstruction,
This work supported by the University Research Priority Program (URPP) Human Reproduction Reloaded of the University of Zurich.

---

A Variational Perspective Generative Protcin Fitness Optimization
Impact Statement
R., Cho, K., et al. Protein discovery with discrete walk- jump sampling. arXiv preprint arXiv:2306. /2360, 2023.
This paper presents a computational method for protein fitness optimization. There are many potential societal con- sequences of our work, none of which we feel must be specifically highlighted here.
Grathwohl, W., Swersky, K., Hashemi, M., Duvenaud, D., and Maddison, C. Oops took a gradient: Scalable sam- pling for discrete distributions. In International Con- ference on Machine Learning, pp. 3831-3841. PMIR, 2021.
References
Ben-Hamu, H., Puny, O., Gat, I., Karrer, B., Singer, U., and Lipman, Y. D-flow: Differentiating through flows for controlled generation. arXiv preprint arXiv:2402.14017, 2024.
Gruver, N., Stanton, S., Frey, N., Rudner, T. G., Hotzel, I., Lafrance-Vanasse, J., Rajpal, A., Cho, K., and Wil- son, A. G. Protein design with guided discrete diffusion. Advances in neural information processing systems, 36, 2024.
Brookes, D., Park, H., and Listgarten, J. Conditioning by adaptive sampling for robust design. In International conference on machine learning, pp- 773-782. PMLR, 2019.
Hermes, D., Blacklow, S. C., and Knowles, J. R. Search- ing sequence space by definably random mutagenesis: improving the catalytic potency of an enzyme. Proceed- ings of the National Academy of Sciences, 87(2):696-700, 1990.
Bryant, D. H., Bashir, A., Sinai, S., Jain, N. K., Ogden, P. J., Riley, P. F., Church, G. M., Colwell, L. J., and Kelsic, E. D. Deep diversification of an aav capsid protein by machine learning. Nature Biotechnology, (9(6)6911-666, 2021.
Higgins, I., Matthey, L.,, Pal, A., Burgess, C. P., Glorot, X.. Botvinick, M. M., Mohamed, S., and Lerchner, A. beta- vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
Chung, H., Sim, B., Ryu, D., and Ye, J. C. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35:25683-25666, 2022.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., and Ye, C. Diffusion posterior sampling for general noisy in- verse problems. In International Conference on Learning Representations, 2023, URL https: htttt::t: //ppnreveiw....c wwwooooommiionnne
Ikram, Z., Liu, D., and Rahman, M. S. Antibody sequence optimization with gradient-guides discrete walk-jump sampling. In ICLR 2024 Workshop on Generative and Ex- perimental Perspectives for Biomolecular Design, 2024.
Jain, M., Bengio, E., Hernanezz-Garcia, A., Rector- Rector-Brooks, J., Dossou, B. F., Ekbote, C. A., Fu, J., Zhang, T., Kil- gour, M., Zhang, D., al. Biological sequence design with gflownets. In International Conference on Machine Learning, pp. 9786-9801, PMLR, 2022.
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhat- tacharya, N., Goldman, S., Madani, A., and Yang, K. K. Flip: Benchmark tasks in fitness landscape inference for proteins. bioRxiv, pp. 2021-11, 2021.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, $4:8780-8794, 2021.
Johnson, R., Fu, X., Viknander, S., Goldin, C., Monaco, S., Zelezniak, A., and Yang. K. Computational scoring and experimental evaluation of enzymes generated by neural networks. biorxiv. preprint, 20231092023.03))044 531015, 2023.
Emami, P., Perreault, A., Law, J., Biagioni, D., and John, P. S. Plug & play directed evolution of proteins with gradient-based discrete meme. Machine Learning: Sci- ence and Technology, 4(2):025014, 2023.
Johnston, E., Fannjiang, C., Wittmann, B. J., Hie, B. L., Yang, K. K., and Wu, Z. Machine learning for protein engineering. In Machine Learning in Molecular Sciences, pp. 311- Springer, 2023.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for higo-resolution image synthesis. URL htts:://axxv. org/abs/2403.03326 2, 2024.
Kingma, and Welling, M. Auto-encoding varia- tional bayes, 2022. URL https://arziv.org/
Abs/1312.5l 6114.
Frey, N. C., Berenberg, D., Zadorozhny, K., Kleinhenz, J.,
Kirjner, A., Yim, J., Samusevich, R., Bracha, S., Jaakkola, T. S., Barzilay, R., and Fiete, I. R. Improving protein
Lafrance- Vanasse, J., Hotzel, I., Wu, Y., Ra, S., Bonneau,
10

---

A Variational Perspective Generative Protcin Fitness Optimization
optimization with smoothed fitness landscapes. In The Twelfth International Conference Learning Represen- tations, 2023.
Sinai, S. and Kelsic, E. D. A primer on model-guided exploration of fitness landscapes for biological sequence design. arXiv preprint 2rXiv:2010.100 2020.
Lec, M., Vecchietti, L. F., Jung, H., Ro, H. J., Cha, M., and Kim, H. M. Robust optimization in protein fitness landscapes using reinforcement learning in latent space. arXiv preprint arXiv:2405. 18986, 2024.
Sinai, S., Wang, R., Whatley, A., Slocum, S., Locane, E., and Kelsic, E. D. Adalead: A simple and robust adap- tive greedy search algorithm for sequence design. arXiv preprint 2rX1v:2010.02141. 2020.
Lin, Z. Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. Evolutionary-seee prediction of atomic-level protein structure with a language model. Science, 379(6637); 1123-1130, 2023.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, Deep unsupervised learning using nonequi- librium thermodynamics. In International Conference on Machine Learning, pp. 2256-2266 PMLR, 2015.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A.,
Lipman, Y., Chen, T.
T.Q.Q. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In International Conference Learning Representations, 2023. URL https: http://peeeeeeeeeeeeeeeeeeeeeeee id=PqvMRDCJT9t.
Ermon, S., and Poole, Score-based generative mod- eling through stochastic differential equations. In In- ternational Conference Learning Representations, 2021. URL https:/tt/........... pttttps://pperevvieee /openreview net/ forum? id Ad-PxTIG12RRES.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint XXiv:2209.0300 2022.
Stanton, S., Alberstein, R., Frey, N., Watkins, A., and Cho, K. Closed-form test functions for biophysi- cal sequence optimization algorithms. arXiv preprint urXhv:2407.00236 2024.
Notin, P., Dias, M. Frazer, J., Marchena-Hurtade J., Gomez, A. N., Marks, D., and Gal, Y. Tranception: pro- tein fitness prediction with autoregressive transformers and inference-time retrieval. In International Conference Machine Learning, pp. 16990-17017. PMLR, 2022.
Trabucco, B., Kumar, A., Geng, X., and Levine, Conser- vative objective models for effective offline model-based optimization. In International Conference Machine Learning, pp. 10358-10368, PMLR, 2021.
Notin, P., Kollasch, A., Ritter, D., Van Niekerk, L., Paul, S., Spinner, H., Rollins, N., Shaw, A., Orenbuch, R., Weitz- man, R., et al. Proteingym: Large-scale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing Systems, (6:64331-643777 2023.
Tran, T. V., Ngo, N. K., Nguyen, V. T. D., and Hy, T. Latentde: Latent-based directed evolution accelerated by gradient ascent for protein sequence design. In NeurIPS 2024 Workshop on Al for New Drug Modalities, 2024.
Van Cleve, J. and Weissman, D. B. Measuring ruggedness in fitness landscapes. Proceedings of the National Academy of Sciences. 112(24):7355-3445 2015.
Praljak, N., Lian, X., Ranganathan, R., and Ferguson, A. L. Protwave-vae: Integrating autoregressive sampling with latent-based inference for data-driven protein design. ACS synthetic biology, 1212))35544 3561, 2023.
Wilson, T., Moriconi, R., Hutter, F,, and Deisenroth, M. P. The reparameterization trick for acquisition functions.
Ren, 7., Li, J., Ding, F., Zhou, Y., Ma, J., and Peng, Proximal exploration for guided protein sequence design. In International Conference on Machine Learn- ing. pp. 15520-18536. PMLR, 2022.
arXiv preprint arXiv: 2017.
Yang, J., Lal, R. G., Bowden, J. C., Astudillo, R., Hameedi, M. A., Kaur, S., Hill, M., Yue, Y., and Arnold, H. Active learming-assiseee assisted directed evolution. Nature Com- munications, 16(1):7 14, 2025.
Rezende, D. J. and Viola, F. Taming vaes. arXiv preprint arXiv:1810.00597. 2018.
Romero, P. A. and Amold, F. H. Exploring protein fitness landscapes by directed evolution. Nature reviews Molec- ular cell biology, (M12))86668766 2009.
Zheng, Q., Le, M., Shaul, N., Lipman, Y., Grover, A., and Chen, R. T. Guided flows for generative modeling and decision making, arXiv preprint arXiv:2311.134433 2023.
Sarkisyan, K. S., Bolotin, D. A., Meer, M. V., Usmanova, D. R., Mishin, A. S., Sharonov, G. V., Ivankov, D. N., Bozhanova, N. G., Baranov, M. S., Soylemez, O., et al. Local fitness landscape of the green fluorescent protein. Nature, 533(7603):397-44) 2016.

---

A Variational Perspective on Generative Protcin Fitness Optimization
A. Additional Methods
additional task of AAV (medium) in Figure 5. supports the finding that the classifier guidance in combination with the generative prior in our variational approach VLGPO yields sequences whose predicted fitness yge follow the con- ditioned fitness y more closely than the directly learned posterior model.
A.I. Fitness predictor and oracle
The fitness predictors 96 and 93. as well as the oracle gv are directly re-used from (Kirjner et al., 2023) without any modifications. The authors use the identical ID CNN ar- chitecture with 157k learnable parameters for all networks, which maps one-hot encoded protein sequence to scalar value representing the predicted fitness. Our method VL- GPO allows to replace all of these components by re-trained models or alternative architectures, although it is known that simple CNNs can be competitive in such data-searce settings (Dallago al., 2021).
learned posterior p(æ) y)
0.6
VIGPO
0.5
0.4
Since (Kirjner et al., 2023) does not explicitly report the final performance of the in-silico oracle, we compute the Mean Squared Error (MSE) on subset of 512 randomly selected samples from the ground truth data set S" to estimate its reliability. The oracle's s predictions closely follow the target fitness values, resulting in MSE values of 0.012240 for GFP and 0.002758 for AAV.
0.3
0.2
0.1
0.2
0.4
0.6
y
A.2. Metrics
Figure 5. AAV medium
For the evaluation of the generated sequences, we use me- dian fitness, diversity and novelty as described in (Kirjner al., 2023; Jain et al., 2022). The sequences are assessed by the oracle Go that was trained on the full data set S* with minimum and maximum fitness values Ymin and Vmax- The median normalized fitness of sampled sequences E X is computed using
B.2. Variational Autoencoder
The VAE embeds sequences into a latent space that yields continuous gradients for sampling in Algorithm 1. Empiri- cally, we choose / 16 and / = 32 for AAV and GFP, since further compression reduces the decoder D's reconstruction accuracy and harms sequence generation. Moreover, the medium and hard tasks are defined by minimum number of mutations from the 99th fitness percentile S", a suffi- ciently large latent dimensionality is required to retain all relevant information.
median (1 (0(x)) Vmin
: : x
Umax - Ymin
Diversity is defined as the median similarity within the set of generated sequences by
We train VAE for each task achieve at least 80% re- construction accuracy validation subset, balancing the latent prior with B. Table 7 shows these results. The lower accuracy in AAV compared to GFP is due to larger effects of mutations on shorter proteins (sequence length d). Since the data sets (see Table 2) are very small, the Kullback- Leibler (KL) divergence in Equation (4) is crucial to ensure Gaussian latent distribution. We can also sample from the VAE and evaluate predicted fitness via gv (Table 7), although median normalized fitness is lower than in Table 2. This is expected, given the sparse latent space and the "hole" problem of VABS (Rezende & Viola, 2018).
median ((dist(z, ('),)) x, E X, x xx x
using the Levenshtein distance are evaluating discrete sequences. Finally, novelty considers the minimum distance of the generated sequences E X' to any of the sequences in the respective data set S that the generative flow matching prior was trained on (see Table 2). Therefore, it is computed using
median (dstn(dist(z,,)) E xx)x)x))
( a
B. Additional Results
B.3. Unconditional Sampling
In Algorithm I one can recover the unconditional sampling case by setting a; = 0 and J = O, thereby sampling exclu- sively from the learned generative flow matching prior. As a result, the classifier guidance in the likelihood term does not
B. Fitness Extrapolation
The experiments on fitness extrapolation of generated se- quences shown in Section 4.4 are complemented by the
12

---

A Variational Perspective Generative Protein Fitness Optimization
12
30
10
21
(a) GFP medium
(b) GFP hard
(c) AAV medium
(d) AAV hard
Figure 6. Grid search for diversity depending on sampling parameters oz and J different tasks.
ferent tasks. We keep a2 constant for all steps t. In addition to Figure 3, the resulting heatmaps for these two parameters on diversity are shown in Figure Overall, combining the findings on predicted fitness and diversity it seems to be heuristic tradeoff to balance these metrics, unlike for GFP (medium), which allows to choose both parameters high. Likewise, for AAV (hard) and (medium) there appears to be broad range from which suitable parameter choices for a, and can be made. Only GFP (hard) displays substantially compromised diversity very quickly (note the limited numer- ical range displayed in the colorbar), hence we choose sig- nificantly lower values a 0.02 and J 5. For the other tasks, we use the hyperparameter settings obtained from the grid search experiments, i.e., Or E {0.97, 1.2, 0.56} and E (39, 19, 37} for AAV (medium). AAV (hard) and GFP (medium). Nevertheless, we do not observe major differences if these hyperparameters are adjusted slightly.
Table 7. VAE reconstruction and sampling results.
Task
Reconstruction Accuracy t
Fitness f
GFP Medium
97.0%
-0.08
GFP Hard
96.8%
-0.21
AAV Medium
80.4%
0.28
AAV Hard
87.8%
0.23
impact the sampling procedure. The median fitness values of the generated samples, evaluated as usual by the oracle Po. shown in Table 8. Note that, in the absence of the predictor Io during VLGPO sampling, the flow matching model effectively serves only as prior without any fitness conditioning, so no post-proe post-processing with top-k sampling is applied for the sake of solely analyzing the prior model.
As shown in Table the median fitness is lower than in Table 3 and Table 4, both of which use VLGPO with classi- fier guidance 9g. However, it exceeds the median fitness of the data sets S from Table 2, likely due to limited training data and the lack of fitness information of the flow matching prior. Moreover, the model may exhibit a mode-seeking tendency, concentrating on modes that are easier to model. While the averaged novelty aligns with expectations, di- versity is much higher in the unconditional scenario, since no classifier guidance steers samples towards higher- fitnesss modes.
Finally, the influence of the number of ODE steps, which directly corresponds the sampling steps K, is examined. This is shown in Figure 7, with the resulting fitness on the left and diversity on the right. The fitness initially exhibits an overshooting behavior, accompanied by a decrease in diversity, but this effect stabilizes after approximately 10 sampling steps. Beyond this point, both metrics remain relatively constant and stable with respect the number of sampling steps K, which is generally the desired behavior.
2000
CFP mxtiom
17.5
had
Table Unconditional optimization results (au, 0, 0).
08:
AAY medium
AAV hard
Task
Fitness
Diversity
Novelty
0.6-
12.5
100
GFP Medium
0,22 + 0.1
18.9 + 2.5
6.2 + 0.4
0.4
GFP Hard
0.42 0.1
14.2 + 1.1
7.0 0.0
AAV Medium
0.38 0.0
1.18
6.0 0.0
0.
AAV medium
CEP hard
25
AAV Hard
0.28 + 0.0
15.6 +.2
8.0 +
0.0
AAV hord
0.0
O0E K
GDB
B.4. Sampling Parameters
Figure 7. Median fitness (left) and diversity (right) for all four tasks depending on employed ODE steps K in sampling.
The determination of the hyperparameters a and / in VL-
GPO sampling is performed through grid search across dif-
13


Reviews:
[{"review_id": "Ko1PMHZCYw", "reviewer": "Reviewer_3GwA", "summary": "This paper proposes a new in-silico method for generating novel high-fitness protein sequences. It first embeds sequences in a lower-dimensional space via a VAE, then fits a generative model to the embeddings by flow-matching. The sampling is guided by a pre-trained fitness predictor and manifold-constrained gradients. Empirical results show improved sampling of high-fitness variants, even in data-scarce settings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive feedback and the insightful comments, which help to improve the quality of the manuscript.\n\nWe fully agree with the reviewer that assessing fitness with a computational model is less reliable than direct wet-lab validation. Experimental validation is the final goal, in-silico evaluation is an important step along the way. We also appreciate the comment regarding the potential overestimation of the performance gap compared to baseline methods. This is indeed a challenge for the community, but hard to quantify precisely.\n\nBelow, we address the reviewer’s more specific questions directly:\n\n- **Can you clarify the connections between the proposed approach and variational inference?**\n\nThank you for this point. VLGPO uses variational inference through the initial embedding step to obtain continuous latent embeddings of the discrete protein sequence variants - the autoencoder is trained with variational inference. However, our approach extends this by introducing additional generative modeling components,  namely flow matching and classifier-guided sampling, to further refine and control the generation process. We will clarify this connection explicitly in the revised manuscript.\n\n- **Can you comment on the poor performance of GFN-AL?**\n\nTo ensure a fair evaluation, the results for all methods are taken from a comprehensive comparison done in recent published work [1]. Hence we can only speculate about the poor performance. GFN-AL might struggle in limited-data scenarios due to sparse reward signals from the few observed mutants, which could potentially limit effective exploration.\n\n- **Can you contrast your approach with classifier-free guidance?**\n\nThank you for this question — we find it very interesting, as it highlights the advantages of using a separate classifier for guidance, which we have addressed in Fig. 3. There, we compare our method to a conditional model that directly learns the posterior $p(x|y)$. In that setup we additionally tested classifier free guidance, but it did not make a noticeable difference. As shown in both the figure and the ablation tables, classifier guidance allows for  more effective steering of the generation process: in challenging scenarios like GFP (hard), classifier-free guidance struggles to produce higher-fitness sequences, while classifier guidance successfully targets high-reward regions.\n\n [1] Kirjner, Andrew, et al. \"Improving protein optimization with smoothed fitness landscapes.\" *ICLR,* 2024.", "claims_and_evidence": "* This paper argues that, for protein fitness optimisation, embedding sequences in a continuous latent space is more effective than token-based sequence representation. This claim is partially supported by the comparison with GWG, GGS, and gg-dWJS that rely on one-hot encoding.\n* This paper argues that using a fitness predictor is an effective approach to guide the optimisation towards high-fitness regions, particularly in limited data regimes. This claim is supported by the experimental study, notably with the comparison with a conditional diffusion model trained from scratch (on the labeled sequence-fitness pairs) without the fitness predictor guidance. The supplementary material reports the different evaluation metrics calculated on the samples obtained from unconditional diffusion model, for reference.", "methods_and_evaluation_criteria": "* The method is sound, well-motivated and each component is clearly described in Section 3. \n* The evaluation is robust. Employing two protein optimisation benchmarks of varying complexity, 9 baseline approaches, and 3 different metrics, it clearly demonstrates the advantages of the proposed approach. \n* However, as the authors note among the potential limitations, the assessment of the fitness relies on a neural fitness predictor which is probably less reliable than wet-lab validation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design and analyses are solid: \n* Results are averaged over five seeds.\n* A fair comparison with other baselines is achieved through grid search over hyperparameters and using the same networks when possible. \n* The ablation study (comparing guidance with and without manifold-constrained gradients, unconditional, and conditioning from scratch) is conducted for both cases, with and without graph-based smoothing.", "relation_to_broader_scientific_literature": "The method is a combination of ideas from the latest generative modelling literature (flow-matching in VAE latent space and guidance with manifold constrained gradient) applied to the real-world use case of protein fitness optimisation.", "essential_references_not_discussed": "The related work is comprehensive.", "other_strengths_and_weaknesses": "**Strengths**\n- The clarity of the paper.\n- The robust experimental design. \n- The novel combination of ideas applied to the real-world use case of protein fitness optimisation. \n\n**Weaknesses**\n- The use of the fitness predictor may overestimate the performance gap with some baseline approaches.", "other_comments_or_suggestions": "* Can you clarify the connections between the proposed approach and variational inference?", "questions_for_authors": "* Can you comment on the poor performance of GFN-AL? \n* Can you contrast your approach with classifier-free guidance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NxGo9sjXuy", "reviewer": "Reviewer_TQyg", "summary": "This paper presents a novel protein fitness optimization model called Variational Latent Generative Protein Optimization (VLGPO). VLGPO uses flow-matching to perform fitness optimization in the continuous latent space of the generative model, allowing efficient exploration of the fitness landscape. Guided by fitness predictors, VLGPO can effectively optimize high-fitness proteins.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their valuable comments and feedback. Below, we address each point individually:\n\n- **Since the evaluation of the methods relies on the predictor, it would be helpful to provide details on the accuracy of $g_{\\phi}$  as in-silico oracle.**\n\nThank you for highlighting this important aspect. We reused the oracle and predictor from [1] entirely out-of-the-box, without any modifications, aiming to present a method in which alternative predictors or oracles could easily be used as drop-in replacements. We will comment on this in more detail in the revised version.\n\nHowever, since [1] does not explicitly report the final performance of the in-silico oracle, we have now computed the Mean Squared Error (MSE) on a subset of 512 randomly selected samples from the ground truth dataset $\\mathcal{S}^*$. The oracle’s predictions closely follow the target fitness values, resulting in MSE values of 0.012240 for GFP and 0.002758 for AAV. \n\n- **In Tables 2 and 3, the variability in diversity and novelty among different methods is significantly greater in GFP than in AAV, which is another observation worth discussing.**\n\nWe appreciate the reviewer’s insightful observation. This aligns well with our findings, indicating that the GFP setting indeed appears more challenging compared to AAV. This discrepancy may arise due to GFP sequences being longer and thus representing sparser and higher-dimensional search spaces. We will discuss this observation in the revised version of the manuscript.\n\n- **It would be helpful if the author could also provide more analysis and statistics, such as diversity at different fitness, of the GFP and AAV dataset.**\n\nThank you for this valuable suggestion. The diversity within the top-performing (99th percentile) sequences of the entire dataset $\\mathcal{S}^*$ is 4.73 for GFP and 5.23 for AAV, as briefly discussed in Section 4.3. In contrast, the diversity of sequences in the four considered tasks is notably higher: for GFP (medium) it is 14.5, for GFP (hard) 16.3, for AAV (medium) 15.9, and for AAV (hard) 18.4. The training datasets naturally contain more diverse sequences, as it includes those with lower fitness levels. In contrast, the top-performing sequences exhibit lower diversity, aligning closely with the diversity observed in Tables 2 and 3. We will highlight this observation in the revised manuscript.\n\n [1] Kirjner, Andrew, et al. \"Improving protein optimization with smoothed fitness landscapes.\" *ICLR,* 2024.", "claims_and_evidence": "The author demonstrated the performance of VLGPO on two protein datasets, GFP and AAV, with medium and hard difficulty. VLGPO showed superior fitness optimization ability compared to previous methods.", "methods_and_evaluation_criteria": "The paper clearly presents the algorithmic and training details of flow-matching and classifier guidance. However, since the evaluation of the methods relies on the predictor $g_{\\phi}$, it would be helpful to provide details on the accuracy of $g_{\\phi}$ as in-silico oracle.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The author follows the experimental design in [Kirjner et al., 24], demonstrating that VLGPO can optimize sequences for both high fitness and high diversity and novelty. In Tables 2 and 3, the variability in diversity and novelty among different methods is significantly greater in GFP than in AAV, which is another observation worth discussing.", "relation_to_broader_scientific_literature": "Previous works mainly focus on directly optimizing protein sequences. This paper optimizes protein sequence in a continuous latent space by using flow matching, providing novel insights into exploring the protein fitness landscape.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "It would be helpful if the author could also provide more analysis and statistics, such as diversity at different fitness, of the GFP and AAV dataset.", "other_comments_or_suggestions": "NA", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X4N0avSTHo", "reviewer": "Reviewer_jm6F", "summary": "The paper proposes Variational Latent Generative Protein Optimization (VLGPO), a method for protein sequence optimization by training a VAE combined with a learned flow matching prior over mutations. A fitness predictor is used for guidance, and the method is evaluated on commonly used database lookups including the AAV and GFP datasets, as well as diversity and novelty metrics. Hyperparameter optimization is identified as a challenge for the approach, which is sensitive to these choices.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their careful evaluation of our manuscript and for the positive assessment of our work. Below, we address each point individually:\n\n- **The authors may be interested in this very recent in silico benchmark, which introduces synthetic test functions that can be used for benchmarking black-box optimization methods like VLGPO.**\n\nThank you for directing us toward this interesting recent work. We will examine it in more detail and consider its suitability for future applications, as synthetic test functions appear very promising for further assessing the generated sequences.\n\n- **The biggest weakness is the reliance on GFP and AAV datasets as the benchmarks, which is a general limitation that affects the field. Have the authors considered other FLIP tasks, or other related tasks?**\n\nThis is indeed an important point. While our method could, in principle, extend to other FLIP or ProteinGym tasks, we chose the GFP and AAV datasets as they currently are well-established benchmarks to allow for fair comparisons against our proposed approach. The main focus here was to investigate performance in limited initial datasets, reflecting common practical scenarios.\n\nNevertheless, exploring additional tasks beyond GFP and AAV is an exciting avenue for future research, and we fully agree on the importance of designing and adopting broader benchmarks in this context.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper is well-situated in terms of the broader literature, and detailed comparisons are made to prior work", "essential_references_not_discussed": "The authors may be interested in this very recent in silico benchmark, which introduces synthetic test functions that can be used for benchmarking black-box optimization methods like VLGPO. \n\nStanton, S., Alberstein, R., Frey, N., Watkins, A., & Cho, K. (2024). Closed-form test functions for biophysical sequence optimization algorithms. arXiv preprint arXiv:2407.00236.", "other_strengths_and_weaknesses": "The paper is clearly written, addresses an important problem in biology, contains relevant benchmarks and baselines, and introduces a novel approach. \n\nThe biggest weakness is the reliance on GFP and AAV datasets as the benchmarks, which is a general limitation that affects the field. Have the authors considered other FLIP tasks, or other related tasks?", "other_comments_or_suggestions": "See above", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "muyFC3Tkvy", "reviewer": "Reviewer_Lapo", "summary": "The paper focus on the problem of protein fitness optimization to find new variants with enhanced fitness, facing challenges such as a vast search space and discrete protein sequences. The introduced Variational Latent Generative Protein Optimization (VLGPO) is a variational framework that enables posterior sampling of protein sequences based on desired fitness levels by combining a learned prior and fitness predictor. They first train a VAE on the sequence space to compress sequence into continuous latent representation and then train a flow matching model to learn a generative prior for the latent space. The fitness predictor is used for classifier-guidance of flow matching generation. VLGPO demonstrates strong performance on AAV and GFP benchmarks in medium and high difficulty tasks with limited data, achieving clear fitness improvements over baselines.  The method has limitations in hyperparameter tuning, especially for challenging tasks like GFP (hard), and relies on in-silico evaluation with a trained oracle as ground truth, where experimental validation could provide more insights into its applicability.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing feedback on our manuscript and appreciate the comments. We will also discuss related work that you mentioned. \n\n- **The predictors and the oracle are sourced directly from previous works.**\n\nIndeed, we take this from recent work from ICLR’24 [1]. As mentioned to reviewer TQyg, we will further elaborate on this. \n\n- **Parameterization of the NN and time step sampling choices for flow matching.**\n\nOur experiments have shown to be very robust against the choice of time steps used for inference, as we use a flow-based approach that is much less sensitive to hyperparameter settings than standard time-discrete diffusion models. The choice of the architecture at hand is rather generic and used as such for many other related tasks. \n\nThe time steps are sampled uniformly between 0 and 1 in training, which is the standard for flow-based approaches, we saw no need to change it. \n\n- **Nowadays, transformer-based models, such as ESM, have shown superiority over CNNs.**\n\nTransformer-based models are standard in protein design, we show that guided flow matching performs well even with simpler architectures (although our network does include attention layers). More expressive models might yield further improvements. Note, previous work [1,2] has discussed that CNN-based models are competitive in limited-data settings.\n\nAs noted in Sec. 5, ESM embeddings are highly expressive, and we plan to explore their potential in future work.\n\n- **Regarding classifier guidance, it remains unclear why the objective is to optimize towards a specific fitness value rather than maximizing the fitness value.**\n\nMaximizing the fitness value is our ultimate goal; however, we intended to demonstrate the expressiveness of our approach by steering sequences toward specific fitness values. In practice, we compared both formulations which performed similarly. Given that our variational framework allows for fitness maximization in the likelihood term, we will explicitly incorporate this into Eq (3). If of interest, we can provide the results in the supplementary.\n\n- **The idea of applying flow matching on the latent space and guiding generation with a classifier has been proposed many times in different domains.**\n\nWe agree that classifier guidance is widely used in generative AI. However, it cannot be trivially applied to discrete sequences, and its application in latent spaces for protein fitness optimization has not been explored before, to the best of our knowledge.\n\n- **All evaluations are conducted on relatively small benchmarks which casts significant doubt on the reliability of the evaluation.**\n\nWe respectfully disagree with this concern. Our evaluation protocol and benchmarks align directly with established standards from previously published work [1]. Such dataset sizes reflect realistic scenarios. Moreover, the in-silico oracle was trained on the complete DMS datasets with 56,086 mutants for GFP and 44,156 for AAV. \n\n- **The dataset is split according to fitness values... a more appropriate split should be based on mutation depth**.\n\nBoth tasks *medium* and *hard* indeed are based on fitness values as well as mutation gaps. This is discussed in [1] (independent original work) and in our manuscript in Sec. 4.1 and in Tab. 6. \n\n- **There is a lack of clarity regarding the dataset used to train the flow matching and VAE models.**\n\nFor each of the four tasks, the flow matching and VAE models are trained exclusively on their respective limited datasets to emulate a realistic scenario. We will explicitly clarify this in the revised manuscript.\n\n- **The achievement of high fitness values comes at a substantial cost of sacrificing diversity and novelty.**\n\nWe agree that there is an inherent trade-off between achieving high fitness values and maintaining diversity, which is also reflected in our approach. Given that our method achieves a substantial improvement in fitness compared to alternative approaches, we believe the benefit outweighs the marginal drop in diversity compared to competing methods.\n\n- **How the authors perform the grid search for hyperparameters is unclear.**\n\nThe plots shown in the manuscript are generated by the oracle, which are an ablation and not the source of our hyperparameters, similar to [1]. We agree, this was not explained well enough. For our parameter estimation we combine grid searches over the diversity as well as the fitness given only by the predictor. We will add the diversity and predictor fitness plots (which resemble Fig. 5) in the appendix. Ultimately it is a heuristic tradeoff between fitness and diversity, which we will further elaborate upon. \n\n- **The authors should incorporate more realistic settings and additional baselines.**\n\nSee response to reviewer jm6F, last point.\n\n[1] Kirjner, Andrew, et al. \"Improving protein optimization with smoothed fitness landscapes.\" \n\n[2] Dallago, Christian, et al. \"FLIP: Benchmark tasks in fitness landscape inference for proteins.\"", "claims_and_evidence": "All contribution points are supported in the paper, except some concerns regarding evaluation. See below.", "methods_and_evaluation_criteria": "The proposed method suffers from a lack of clarity and fails to demonstrate sufficient novelty.\n1. The predictors $g_\\phi$, $g_\\tilde{\\phi}$, and the oracle $g_{\\psi}$ are sourced directly from previous works. However, the paper offers no insights into their design process, the training process on the benchmark, or their accuracy levels. This absence of information leaves readers uncertain about the reliability of these predictors.\n2. When training a flow matching model, the parameterization of the neural network and the choice of the time step sampling distribution significantly impact the final performance. These factors should be carefully considered as hyperparameters to be tuned. Unfortunately, the paper neglects to address this, leaving a gap in understanding how the model was optimized for optimal performance.\n3. Most of the models used in the evaluation are based on CNNs. Nowadays, transformer-based models, such as ESM, have shown superiority over CNNs. This raises doubts about the reliability of the evaluation. Using outdated models may lead to an inaccurate assessment of the proposed method's performance compared to the latest advancements in the field.\n4. Regarding classifier guidance, it remains unclear why the objective is to optimize towards a specific fitness value rather than maximizing the fitness value. In real-world scenarios, it is entirely possible to achieve fitness values higher than the highest ones present in the dataset. 5. The idea of applying flow matching on the latent space and guiding generation with a classifier has been proposed many times in different domains. This work seems to directly apply this method on the protein sequence design task.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I have serious doubts about the soundness of the experimental design and the validity of the results presented.\n1. All evaluations are conducted on relatively small benchmarks, containing fewer than 3,500 mutants. This narrow scope casts significant doubt on the reliability of the evaluation. Given that it only represents a small fraction of the vast possible mutation landscape, the generalizability of the findings is severely compromised. Moreover, the predictors and oracles, trained and evaluated on these limited datasets, are also suspect. In contrast, many recent datasets, such as those in [1] and [2], have been released with 10 to 100 times more mutants, highlighting the inadequacy of the current benchmark size.\n2. The dataset is split according to fitness values, with lower-value data used for training the fitness predictor and higher-value data as the generation objective. However, in real-world protein engineering, the typical approach is to design single-mutants first, followed by double-mutants and then triple-mutants. Therefore, a more appropriate split should be based on mutation depth. Additionally, accurately modeling the effects of multiple mutations remains a challenging problem, which poses a significant challenge to the accurate training of oracles and predictors.\n3. There is a lack of clarity regarding the dataset used to train the flow matching and VAE models. It is not specified whether they are trained on $S^*$ or if the training data contains all mutants. \n4. The achievement of high fitness values comes at a substantial cost of sacrificing diversity and novelty. This trade-off is highly undesirable.\n5. How the authors perform the grid search for hyperparameters such as $\\alpha_t$ and $J$ is unclear. It appears that the search is based on the final reported metrics. This raises concerns about the practical applicability of the method in real-world scenarios, as the hyperparameter tuning strategy may not be robust enough for different contexts.\n6. The authors should incorporate more realistic settings and additional baselines, similar to those presented in [1] and [2]. \n\n[1] Notin, Pascal, et al. \"Proteingym: Large-scale benchmarks for protein fitness prediction and design.\" Advances in Neural Information Processing Systems 36 (2023): 64331 - 64379.\n\n[2] Ouyang-Zhang, Jeffrey, et al. \"Predicting a protein's stability under a million mutations.\" Advances in Neural Information Processing Systems 36 (2023): 76229 - 76247.", "relation_to_broader_scientific_literature": "See the summary, method and experiment parts.", "essential_references_not_discussed": "[1] Notin, Pascal, et al. \"Proteingym: Large-scale benchmarks for protein fitness prediction and design.\" Advances in Neural Information Processing Systems 36 (2023): 64331 - 64379.\n\n[2] Ouyang-Zhang, Jeffrey, et al. \"Predicting a protein's stability under a million mutations.\" Advances in Neural Information Processing Systems 36 (2023): 76229 - 76247.", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes to optimize the fitness of proteins by first compressing the protein sequences into latent continuous space and then using flow matching models to sample in the latent space guided by a pretrained protein fitness predictor. Experimental results prove the competitiveness of the proposed approach on two protein benchmarks. 

The reviewers in general appreciate the novelty of the proposed approach and its significance over existing approaches. A reviewer raises big concerns over the evaluation set up, which the authors should address in revised version. The AC tends to vote for acceptance.
Decision: Accept (poster)