Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Tasks
Authors: Angelica Chen
Samuel Don Stanton
Frances Ding
Robert G Alberstein
Andrew Martin Watkins
Richard Bonneau
Vladimir Gligorijevic
Kyunghyun Cho
Nathan C. Frey
Venue: ICML 2025 poster (2025)
Keywords: biophysical optimization
large language models
discrete sequence optimization
Primary area: deep_learning->large_language_models

Abstract: Although large language models (LLMs) have shown promise in biomolecule optimization problems, they incur heavy computational costs and struggle to satisfy precise constraints. On the other hand, specialized solvers like LaMBO-2 offer efficiency and fine-grained control but require more domain expertise. 
Comparing these approaches is challenging due to expensive laboratory validation and inadequate synthetic benchmarks. 
We address this by introducing Ehrlich functions, a synthetic test suite that captures the geometric structure of biophysical sequence optimization problems. 
With prompting alone, off-the-shelf LLMs struggle to optimize Ehrlich functions. 
In response, we propose LLOME (Language Model Optimization with Margin Expectation), a bilevel optimization routine for online black-box optimization.
When combined with a novel preference learning loss, we find LLOME can not only learn to solve some Ehrlich functions, but 
can even perform as well as or better than LaMBO-2 on moderately difficult Ehrlich variants. 
However, LLMs also exhibit some likelihood-reward miscalibration and struggle without explicit rewards. 
Our results indicate LLMs can occasionally provide significant benefits, but specialized solvers are still competitive and incur less overhead.

Generalists vs. Specialists: Evaluating LLMs on Highly- Conayy-Cnnttaineed Biophysical Sequence Optimization Tasks
Angelica Chen 12 Samuel D. Stanton *2 Frances Ding² Robert G. Alberstein Andrew M. Watkins
Richard Bonneau Vladimir Gligorijevié? Kyunghyun Cho Nathan C. Frey?
Abstract
All possible polypeptide sequences
Forcable oequence space Unfordable sequence spoce
Although large language models (LLMs) have shown promise in biomolecule optimization prob- lems, they incur heavy computational costs and struggle to satisfy precise constraints. On the other hand, specialized solvers like LaMBO-2 offer efficiency and fine-grained control but re- quire more domain expertise. Comparing these approaches is challenging due to expensive labo- ratory validation and inadequate synthetic bench- marks. We address this by introducing Ehrlich functions, synthetic test suite that captures the geometric structure of biophysical sequence op- timization problems. With prompting alone, off- the-shelf LLMs struggle to optimize Ehrlich func- tions. In response, we propose LLOME (Lan- guage Model Optimization with Margin Expec- tation), bilevel optimization routine for on- line black-box optimization. When combined with novel preference learning loss, we find LLOME can not only learn to solve some Ehrlich functions, but can even perform as well as or better than LaMBO-2 on moderately difficult Ehrlich variants. However, LLMs also exhibit some likelihood-reward miscalibration and strug- gle without explicit rewards. Our results indicate I.LMs can occasionally provide significant ben- efits, but specialized solvers are still competitive and incur less overhead.
Alcho beta proteins
attand content
Aiçhka
MOuntiy Conserved kird
Alpha-holic content
The space of all possible polypeptide sequences is vast, but only tiny fraction forms stable, folded (i.e. fea- sible) proteins. Different protein families (alpha, beta, and mixed alpha/beta) generally occupy distinct regions of se- quence space, but disordered proteins and remote homologs with conserved structure and low sequence identity illustrate the complexity of the protein design landscape.
Figure 1:
1. Introduction
Despite their remarkable abilities, large language models (LLMs) often fail at tasks with fine- ene-grained constraints recent work has shown that even state-of-the-art models struggle to reliably generate text with fixed number of words or to incorporate specific keywords or constraints (Garbacea & Mei, 2022; Sun et al., 2023; Yuan et al., 2024; Chen et al., 2024c). This limitation becomes especially criti- cal black-box biomolecule optimization problems (Figure 1), where even minor violations of biophysical constraints like protein stability or solubility can render solution im- possible to synthesize, purify, and assay (Hie al., 2024; Ismail et al., 2024). While specialized solvers like LaMBO- 2 address these constraints through careful modeling choices and architectural design (Gruver et al., 2024), adapting such solvers to new domains requires significant domain exper- tise and engineering effort. Recent work suggests that the
Work done at Genentech 'Center for Data Science, New York University, New York City, U.S.A. "Prescient Design, Genentech, New York City, U.S.A. 'Prescient Design, Genen- tech, San Francisco, U.S.A.. Correspondence to: Angel- ica Chen sangelica.chenee www.chhnnnnnuuunr...ee Samuel Stanton <sttn- ton.samuel com..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Generalists vs, Specialists: Evaluating LLMs on Hlighly-Constranne Biophysical Sequence Optimization Tasks
LLM Outer Loop
Input Sequences
Oracle Scoring
LLM Training
LLM Inner Loop
Optimized Sequences
AAA AM
AM
0.77
Sequence pool
LUM ranking
LLM refinement
RAA
A&S ASN AM
AM AMA
AMA
0.85
AM
Figure 2: An overview of Large language model optimization with Margin Expectation (LLOME). LLOME alternates between two optimization loops: (1) the outer loop trains the LLM on oracle-labeled data, and (2) the inner loop generates candidate sequences through iterative refinement without oracle access. At each outer loop iteration, the highest-ranked candidates are evaluated by the oracle to generate training data for the next iteration. LLOME enables effective optimization while minimizing expensive oracle queries through its bi-level structure.
capabilities of LLMs may be attainable at far lower cost than previously thought (Zhu et al., 2024; Guo et al., 2025), and a deeper question remains: can improved preference learning methods help these models combine human-like flexibility with precise constraint satisfaction?
even when the prompt reveals the entire test function. We show that LLMs nevertheless can be taught to solve some Ehrlich functions when used to drive bilevel optimization loop with online feedback, and are particularly effective when paired with a novel preference learning loss. Finally, a non-trivial closed-form objective allows us to deeply study preference learning itself. leading to new insights. In sum- mary, our findings and contributions are as follows:
fundamental challenge obstructing the development of LLMs as black-box optimization (BBO) solvers evalu- ation. Unlike typical machine learning (ML) benchmarks for supervised and unsupervised models, optimization algo- rithms cannot be evaluated with a static dataset unless the search space is small enough to be exhaustively enumerated and annotated with the test function. Existing synthetic test functions commonly used in BBO research (Molga & Smutnicki, 2005) have very well-documented structure and solutions, making train-test leakage into pretrained LLM weights almost certain. Real-workd black-box objectives by definition do not have formally characterized structure or solutions and are usually expensive to query. For instance, biomolecule optimization tasks require wet lab experiments for verification and chatbot systems require online user feed- back, which is unsuitable for rapid development.
1. Novel Test Functions for BBO and Preference Learn- ing: Ehrlich functions are accessible to all researchers, difficult for ofate-of-the-art solvers, and well-motivated by real biomolecule optimization problems.
2. LLOME (Large Language Model Optimization with Margin Expectation) We propose a bilevel optimization algorithm that allows LLMs to explore and learn new capabilities from online feedback when prompting fails.
3. Margin-Alignedd Expection (MargE) Loss: Our ex- periments Ehrlich test functions motivate pro- pose MargE, novel training objective that maintains the simplicity of supervised fine-tuning (SFT) and direct preference optimization (DPO), yet outperforms them.
It is clear that test functions that are both more accessible and more difficult are needed for early-stage research and validation. We propose Ehrlich functions' as an idealized model of real biomolecule BBO tasks like antibody affinity maturation, building on principles from structural biology and biomolecular engineering experience. Ehrlich functions have adjustable difficulty and are always provably solvable; easy instances be solved quickly by genetic algorithm and used for debugging, but the same algorithm fails to solve harder instances after consuming over 500M function evaluations. These results can be reproduced in minutes on single GPU. Importantly, Ehrlich functions are procedurally generated and not yet compromised by train-test leakage to pretrained L.MMs. Sfate-of-thh-are I.I.M chatbots struggle to solve Ehrlich functions to optimality by prompting alone,
4. New Insights into Preference Learning: We find that preference-tume LLM likelihoods do not necessarily correlate with the true objective. Furthermore, iterative preference tuning with ground-truth rewards outperforms training on preference pairs alone, and DPO in particular suffers from mode collapse and over-optimization.
5. Comparisons between LLMs and specialized solvers: We compare LLOME to LAMBO-2, a solver purpose- built for constrained discrete BBO. We find that LLOME can outperform or perform well as LAMBO-2 on medium difficulty test functions, and is comparable on easier and harder variants, indicating that specialized models remain competitive after accounting for compute cost.
Named after Paul Ehrlich, an early pioneer of immunology.
2

---

Generalists vs. Specialists: Evaluating LLMs on Hlighly-onnstraneed Biophysical Sequence Optimization Tasks
2. Background
(Rafailov al., 2024a) and distribution mismatch between the training dataset and policy outputs (Chen et al., 2024b; Tang et al., 2024). Nevertheless, DPO has become one of the most prevalent algorithms for offline alignment of LLMs. We provide further background on past work related to LLMs for optimization in Sec. 3.
We focus on pre-trained autoregressive large language mod-
els (z) parameterized by 0. Th defines a probability dis- tribution over discrete tokens æ E V for vocabulary V. We can also define the likelihood of sequences
V" as | [ where V the set of all con- catenations of tokens in V.
(x) =
3. Related Work
Supervised Finetuning (SFT) After pre-training, LLMs are typically finetuned on some task-specific dataset DD = V+) consisting of pairs of input x and target y se- quences. During SFT, Ty is trained to minimize the negative conditional log likelihood of examples from D:
Our work combines insights from multiple areas of research, including discrete sequence black-box optimization, con- trollable text generation, and LLMs for optimization and scientific discovery. See Appendix A.1 for further discus- sion on related work.
LSFT :
E - log Te (y(x) xyyD
Discrete Sequence Black-Box Optimization Many algo- rithms for discrete sequence optimization take inspiration from directed evolution (Aroold, 1998), a combination of random mutagenesis (a means to generate variants of the cur- rent solution) and high throughput screening (discriminative selection pressure). Researchers have explored many types of variant generators, including genetic algorithms (Back, 1996; Sinai et al., 2020), reinforcement learning (Anger- mueller et al., 2020), denoising with explicit discriminative guidance (Stanton et al., 2022; Maus et al., 2022; Gruver et al., 2024), and denoising with implicit discriminative guidance (Tagasovska et al., 2024). While these algorithms are all very general in principle, in practice a substantial amount of effort is required to actually implement these al- gorithms for new tasks due to changes in the problem setup. Our work investigates whether L.MSs can provide more generalizable approach that extends readily to new problem domains while maintaining competitive performance.
Preference Learning In some settings, LLMs are further trained to align their output distributions to a reward distribu- tion, typically encoded with a reward model r(x): V* + R. This is frequently accomplished with reinforcement learn- ing, where the LLM is trained via a policy gradient method to maximize the expected rewards The reward model is trained from dataset of human prefer- ences consisting of triples (x.Yw> Y.). where x is prompt obtained from some offline dataset X' and yw and yı are sampled from the current policy. The initial model is re- ferred to as the reference policy Tiat and Yw: Yt are assigned such that Yu is more preferred by human raters than yt- More recently, practitioners have added a KI. regularizer to prevent the LLM from quickly over-optimizing, yield- ing a family of learning algorithms known reinforcemens learning from human feedback (RLHF; Ziegler et al., 2019; Stiennon et al., 2020), sharing a common objective:
y).
LLMs for Optimization and Scientific Discovery Prior work on LLMs for optimization has largely followed two approaches. The first uses LLMs to translate natural lan- guage descriptions into formal mathematical representations that can be solved by traditional optimizers (Ramamonji- son et al., 2022; Ahmed & Choudhury, 2024). The second approach leverages L.MSs directly as optimizers, often by embedding them within evolutionary algorithms (Romera- Paredes et al., 2023; Chen et al., 2024a) or using them for prompt-based based optimization (Yang et al., 2024). Most closely related to our work is Ma et al. (2024), which also employs LLMs in a bilevel optimization loop. However, while they assume access to gradients through differentiable simula- tions, our method operates in the more challenging setting where only black-box evaluations are available. We also provide novel insights into how LLMs can improve their optimization capabilities through specialized training objec- tives, even without access to ground-trum -tuth rewards during the inner optimization loop.
(R) =
E - r(xx)) + +KL((T ( TRa) x
(1(x)
RLHF is commonly trained using Proximal Policy Opti- mization (PPO; Schulman et al., 2017), which involves con- siderable engineering complexity due to the need to train and coordinate four models (Tg, TRee. reward model, and critic model). Furthermore, RLHF-PPO is particularly sensitive to hyperparameter values and prone to training instabilities (Zheng et al., 2023b). To address some of these issues, Rafailov et al. (2023) proposed an offline version of RLHF known as Direct Preference Optimization (DPO). DPO skips reward modeling and directly trains on the pref- erence triples with contrastive objective (0):
Te(Vw 3log (x(())x) Tyee(yw.l [x)
E logo ( Blog X.
T (yi
where a is the sigmoid function. DPO often produces mod- els with similar generative quality as RLHF-PPO, but in- volves notable tradeoffs such as faster over-optimization
Controllable Text Generation (CTG) CTG represents
3

---

Generalists vs. Specialists: Evaluating LLMs on Hlihhyy-onsstranne Biophysical Sequence Optimization Tasks
specialized case of optimization where the objective is to generate sequences with specific attribute values rather than maximizing a general objective function. While LLM prompting can effectively control high-level attributes (Brown et al., 2020), precise control remains challenging (Carlsson et al., 2022). Two primary approaches have emerged: control codes prepended during training (Keskar et al., 2019; Padmakumar al., 2023; Raffel et al., 2020: Madani et al., 2023) and inference-tine guidance using auxiliary models (Dathathri et al., 2020; Liu et al., 2021; Deng& & Raffel, 2023; Dekoninck et al., 2024). Our work bridges CTG and optimization by viewing sequence con- straints as part of the optimization problem, demonstrating that LLMs can learn to generate sequences satisfying precise constraints through our bilevel optimization framework.
Algorithm LLOME, an approach for bilevel optimization of highly constrained sequence optimization problems with LLMs. We use no 10. j = 2000, and T = 10.
Input: Scoring function f: pretrained L.MM no, parameter- ized by initial weights 8n: initial seed sequence X, E F; j number of test function evaluations per round; T number of LLOME rounds.
S + {(xu, (xx)))
Xo + GENETICALGORTTIIM
Seed with nu
rounds of evolution.
So + (()) | x Xo}
b Score the initial candidates.
S 2 SUS
0 while <
Outer Loop
do
D, < DATASTTFORMATTIGG(S)
0,11 + TRAIN(O,, D,)
X.+1 +- ITTRATIVERENNNENTTTL.L.. Si) D Inner Loop
X41 <- (ILTER(X..I.)) & Filter X441 down to j samples. & Oracle labeling.
4. Method
S.41 <- {(x, x &413
S < SUS$14
The goal of any optimization procedure is to find a max- imizer x² E arg = J (x) of an objective function f: + R over a feasible set F C. X. In the black-box optimization (BBO) setting, we only have access to zero- order information about f. We can query J different inputs, but we get no other information (e.g.. derivative in- formation). With an infinite query budget, we could find xx by brute force. In practice, limited resources usually con- strain both human and artificial intelligences to strategies that iteratively refine a current solution xG. This local, itera- tive approach aligns with the decision-theorett concept of satisficing - seeking a "good enough" improvement rather than exhaustive global optimum (Simon, 1956; Wilson, 2024). Furthermore, in many real high-dimensional search spaces, such iterative local optimization is surprisingly ef- fective (Wu et al., 2023). The success of gradient- based training for neural networks, which finds solutions through iterative local updates, itself underscores the power of local optimization strategies.
i i+1
end Output: arg ((0)) f(x
proach, Cuain the MargE loss (Sec. 4.2), however other preference-based or supervised training objec- tives can also be used.
2. Policy Execution (Inner Loop). Using the updated policy me, from the outer loop, the inner loop generates and selects new batch of candidate sequences for evaluation. This phase operates without direct calls to the black-box oracle f.
Input prompts x x(1) seeds are selected from the top- scoring historical sequences in D
The policy тT, generates new candidate output sequences {vi") .....y, multiple sampling from ne,- Heuristics such as ad- justing sampling temperature are used to manage diversity and avoid premature collapse.
(1) from these seeds via
4.1. The LLOME Algorithm
A subset ( S. [x(²) .i."....y? /0) is chosen by ranking with the policy likelihood Te, (vlx).
To adapt LLMs for the aforementioned BBO setting, we propose LLOME, which employs bi-level optimization strategy for iteratively refining an L.M policy * /x) to generate improved sequences. This process alternates be- tween improving the policy based on current data (the outer loop) and using the updated policy to generate and select new, high-potential candidates for black-box evaluation (the inner loop). The bi-level process unfolds as follows:
3. Data Collection:
The selected candidates (0) are evaluated by the oracle f to yield new batch of labeled data ((i+1) = {(y, f(y)) y E y(). This new data is added to the cumulative dataset: D(++1) = D(A) u ((++1) The process then returns to the outer loop (Step 1) with the augmented dataset.
1. Policy Improvement (Outer Loop). At each it- eration i, given the cumulative dataset DD()) (con- taining all previously evaluated sequences and their scores), the outer loop tunes the policy parameters 0, = arg ming Luain ID(1), Nee). In our primary ap-
Algorithm 1 provides high-level outline of LLOME We collect an initial data package (T)0)) from the history of a pre-solver (in our case, no iterations of a genetic algorithm, details in Appendix A.6.3). In real applications, this data package may be provided to the user from historical records.
4

---

Generalists vs. Specialists: Evaluating LLMs on Highly--onstrainen Biophysical Sequence Optimization Tasks
A key technique employed during policy improvement is dataset matching, augmentation strategy proposed by Tagasovska et al. (2024). For all X E IDC*), we search D( for other sequences y'm in its local neighborhood (up to distance cutoff match) ) that represent known improvements (i.e., f(y'm) ) > f(xx)). See Appendix A.6 for detailed sub- routine descriptions. The modularity of the policy training objective Lusin in the outer loop enables us benchmark dif- ferent LLM fine-tuning strategies within the LLOME frame- work. In our experiments, we compare ALOME-MARGE against variants such as LLOME-SFT and LLOME--DO..
in Appendix A.3). For the regularization term, we employ mass-covering reverse KL. divergence, ADRu(Ted || mo). This choice encourages To to retain broad coverage over re- gions where Tee assigns probability, which helps avoid pre- mature policy collapse by maintaining exploratory breadth. Furthermore, the reverse KL does not heavily penalize To for placing mass on optimal responses in the tails of Trref and naturally suited for training with off-policy data, as its expectation is taken with respect Tzet- In contrast, ob- jectives regularized by forward KL (e.g., (TL (ree) as in Steinberg et al. (2024)) may also satisfy SIC; however, they are prior mode-seeking and better suited for on-policy sample estimation. In the next section, we discuss a compu- tationally tractable form of Eq. (1) and our specific choice of improvement measure r(x.
4.2. Theoretical Motivation for MargE
Having described the higher-level overview of LLOME, we now motivate the design of our LLM training objective, MargE, from first principles.
4.3. Computing the FReKI. Loss
To emulate an iterative refinement strategy with an LLM, we pose the task as learning a policy <o(y|x) x) that responds with a refined solution y X' given a current prompt so- lution x E X, aiming to maximize some measure of local improvement X x X' + R. Given a finite dataset of triplets (x, process, we seek to learn the optimal policy for selecting improvements. Ideally, we would want to select y to deter- ministically maximize r(x.y), y). However, acknowledging our bounded compute resources (i.c., fixed model archi- tecture and training budget) and incomplete information (finite training data), we invoke the principle of maximum entropy (Jaynes, 1957) as our guide for choosing the form of the optimal stochastic policy π²' This principle leads us to the Boltzmann distribution π (ylx) x exp(3 . r(x, y)). where 8 > 0 is the rationality parameter controlling the explore-exploit tradeoff. The Boltzmann distribution is known to maximize policy entropy subject to the constraint of achieving certain expected reward (Ortega & Braun, 2013; Jeon et al., 2020), and is mathematically equivalent to the Bradley- preference model (Luce et al., 1959).
Direct optimization of Eq. (1) is challenging due to the KL terms involving the intractable partition function of x*. To arrive at a computationally tractable form suitable for training with off-policy data, we expand the KL divergence terms and apply importance sampling, using samples from the reference policy Tref yielding the following expression for (0):
generated through such an iterative
E X gruiTer(- (xxx)
Pg
A ly|
log - pp (r(x,y) (2)
where Txx is the distribution of input prompts, P7 = To (rrt)///Tr///))) is the importance weight likelihood ra- tio, and ly] normalizes the policy log- likeliiiddoood by the length of the response. The full derivation, along with a discussion of its design principles in comparison to DPO and RLHE, can be found in Appendix A.4.
4.4. The Margin Reward Function
Thus far we have worked with the FReKL. loss in general form for an arbitrary reward function r(x,y) y). For iterative refinement behavior, we propose a specific reward struc- ture that directly quantifies the notion of improvement. We define the margin reward as:
To ensure stable learning and ground our policy, we reg- ularize Ty towards a prior policy Tyee (e.g., a base pre- trained LLM). Our theoretical policy objective is an in- stance of the generalized variational Bayes (GVB) frame- work (Knoblauch et al., 2019). We call our specific instan- tiation the Forward-Reverse KL (FReKL) loss, which we define as follows:
7 T'margin (x. y) = ( f(x)) f(x) if f(y) > f(x),
(3)
otherwise,
LEREKL (0) == DKL(#O || m²) + XDRL (Tree To).
(1)
where f(vv) is the objective value of the prompt and f(y) is the value of the response. Typically if & F (the feasible
This objective is designed to achieve several key desiderata. First, the data term DKL. (To x²) pushes the learned policy Te towards the optimal target policy т.. Second, this for- mulation satisfies the Strong Interpolation Criteria (SIC; Hu et al., 2024), ensuring a smooth tradeoff between x² and the reference policy Teef as governed by X E (0, +00) (proof
set) [xx is defined to be -o. however we set f(x) 0 for all infeasible x to avoid infinite rewards.
The specific form of mmargin y), particularly the clipping at zero, is important. While simply using fly) - fxx as the reward might seem intuitive, it can lead to unintended
5

---

Generalists vs. Specialists: Evaluating LLMs on Hlighyy-Cnsstranne Biophysical Sequence Optimization Tasks
policy behavior due to the translation-invariance property of Boltzmann distributions. Specifically, unclipped re- ward implies <yy|x) = <y)x') for all pairs x, x,x' E X, meaning the preference for any response y is independent of the prompt and depends only on f(y) (see Appendix A. for details). Such prompt-independent responses would not satisfy our goal of local refinement. Clipping the reward at O when fy) S fx) resolves this issue and interestingly resembles margin loss. Hargin-based rewards are popular in the Bayesian Optimization literature; however, improve- ments are usually calculated against a fixed baseline, e.g- the best-observed value (Močkus, 1975; Jones et al., 1998).
Setting q k corresponds dense signal which incre- ments h2 m (()) whenever additional element of any motif has been fulfilled. We provide additional details in Appendix A.2 about how to ensure that all motifs are jointly satisfiable and that there exists at least one feasible solution that attains the optimal value of 1.0. We also provide further evidence in Appendix A.2 that f is difficult to optimize with a GA, even with small L, k. and e values.
5.2. Experimental Setup
We evaluate LLOME LAMBO-2, and the GA on four Ehrlich functions with varying parameters and difficulties. To avoid potential confusion between different test functions, we propose the following naming convention: Ehr( L)- c-k-q. The test functions we consider are as follows:
To avoid ambiguity, when the FReKL loss (Eq. 2) is combined with the margin reward function (Eq. 3), we will call the resulting policy objective the Margin-Aligned Expectation (MargE) loss in the remainder of this work.
f1: Ehr(8, 128)-8-8-8 (casy)
5. Evaluation
f2: Ehr(32, 32)-4-4-4 (medium)
fa: Ehr(32, 128)-4-4 (medium)
We now introduce Ehrlich functions, a class of closed-form test functions that capture the geometric structure of bio- physical sequence optimization problems while enabling fast evaluation and reproducible benchmarking. These func- tions maintain key characteristics of real fitness landscapes, non non-additive mutational effects, while avoiding the fidelity- latency tradeoffs inherent in simulation based approaches (Kellogg et al., 2011; Hummer et al., 2023).
f4: Ehr(32, 128)-8-8-8 (difficult)
We tune hyperparameters for only a single test function (f2) and carry the best configuration across to all functions (more details provided in A.7 and A.8). See the Appendix (Fig. 12) for results on a test function with g < k.
LLOME Details We compare the performance of three dif- ferent variants of LLOME (LLOME-SFT, LLOME- MARGE, LLOME-DPO) against that of GA. The details of the ge- netic algorithm are given in Appendix A.6.3 and the training details for SFT, DPO, and MargE are given in Appendix A.7. Each LLOME variant is seeded with data from 10 rounds of the GA (i.e., ng = 10). For CLLOM--MRER and LLOME- DPO, one round of SFT is trained before continuing with MargE and DPO training in future iterations. All three vari- ants use the pre-trained model PYTHIA 2.8B (Biderman et al., 2023) as the base model. During the TRAIN step of each iteration of LLOME (step 1 of Alg. 1), we train the current checkpoint for one epoch. Lastly, we use j = 2000 test function evaluations per iteration of LLOME.
5.1. Defining an Ehrlich Function
We provide a complete description of Ehrlich functions in Appendix A.2. In brief, given a token vocabulary V and the set of sequences 4 formed of concatenations of L tokens in V, we first define F C VL, the set of feasi- ble sequences. F is defined as the support of a discrete Markov process (DMP), more details of which are given in Appendix A.2. We also define a set of spaced motif's that represent biophysical constraints in specific regions of a sequence. These motifs are expressed as pairs of vectors ((m(),
(1), ,())),, ...., (m((s) {(())) where ( E Dk ((t) E 24, and k: < L//c. An Ehrlich function f then describes the degree to which a sequence x v is feasible and possesses all motifs, modulated by response function g. For q E [1, k]
LAMBO-2 Details We also compare the performance of LLOME against LAMBO-2. LaMBO-2 is black box op- timization algorithm tailored to protein sequence design with wet lab validation in real antibody lead optimization settings (Gruver et al., 2024). We instantiate LaMBO-2 to jointly train an encoder shared between generative discrete diffusion head and discriminative heads, which guide gener- ation via their predictions of the reward of sequence and whether it satisfies the problem constraints. We use CNN architecture and train the model from random initialization (for hyperparameter details see Appendix A.8). In the ex- periments presented below, the LaMBO-2 model has a total of 3 314K parameters. To compare directly with LLOME, we seed LaMBO-2 with the same 10 rounds of GA designs and
bits of precision, f is expressed as:
f(x) = M. m go (x,,,, m (() if E F,
(4)
-00
otherwise.
The function h, defines the degree to which x fulfills a given constraint (m, and is defined as follows:
h,,, m, s) = ( ((maxx) KZL
k 1 [1e+o] m/]////
Σ
1=1
6

---

Generalists vs. Specialists: Evaluating LLMs on Biophysical Sequence Optimization Tasks
[4] - 128
Gereti Aigattii
LICRESSF LaMEC 2
2.6
94
0.6
0.6
digitthr
Genetic Aggiitie
Genetic gerther
LUCEE (MD
LICMb UHD
LUCM UPO
02
LLCKE MAAEEE
LICKE MARCE
LLCME MAnCE
LICHE-ST
LLCKE
Norrbe Red Furation Evatuntirs
NATTER Rel Funsttin Evolosliors
Martar of Rod Paratton Evalcalion
Nambe of Tod Fanation Evaiuaaionn
(a) Ehr(8, 128)-8-8-8 Le. fi
(b) E3r 32)-4-4-4i.. Le. fz
(c) Ehr(32, 128)-4-4-4 i.e. i's
(d) Ehr(32, 128)-8-8-8 Le.
Figure 3: On medium- tasks, LLOME-MARGE and LAMBO-2 outperform other methods. On very easy or difficult tasks, all methods perform comparably. We show minimum simple regret achieved as function of the number of test function evaluations for Ehrlich functions f1. f2, fs. and ft. Some lines end early due to early convergence to an optimum or generator collapse. The first 10K test function evaluations of every method correspond to the solutions produced by the GA pre-solver.
of Unique Sequences
% Feasible Sequerices
1.0
0.0
Genatic Algarithm
0.6
LLOME-DPO
LLOME MARGE
04
LLOME-SFT
LaMBO-2
5000
10000
15000
20000
25000 30000 0
5000 10000
15000
20000
25000
30000
Number of Test Function Evalustions
Number Test Function Evalustions
Figure 4: LAMBO-2 produces the most diverse and feasible solutions but ALOME-MARGE balances diversity with accuracy. The percentage of generated sequences for fz that are unique or feasible. Although LAMBO-2 achieves the most diverse and feasible solutions, LLOME- MARGE balances diversity with regret (Fig. 3b). The lines for MARGE and LAMBO-2 end early because both discover the optimal solution early.
use j 2000 test function evaluations per iteration.
problem for the LLMs, even full problem specification. In the rest of our experiments, we provide models only with pairs of sequence x and score fxx).
Metrics We assess performance on Ehrlich function bench- marks with the simple regret metric: regret(r) = 1 - f(x). We primarily show results on minimum simple regret over all optimization iterates, i.e. min; regret (x4), and the aver- age regret of sequences in a given round. When applicable, we also evaluate each method's reward (Eq. 3), i.e., how much the model's 's output improves over the input.
LLOME-MARGE can perform better than or compara- ble to specialized models in designing sequences under oracle label budget constraints. In Figure 3, we plot the minimum regret achieved by each method as function of the number of test function evaluations, under four differ- ent Ehrlich test functions with varying difficulty. First, we find that test function difficulty is essential for highlighting statistically significant differences between methods. On easy (fi) or difficult (f4) tasks, all methods achieve compa- rable performance, either finding the optimal solution very quickly or making virtually no progress. In contrast, on the medium difficulty test functions fz and fs. performance is more differentiated. On f2 LLOME-MARGE performs the best, whereas LAMBO-2 performs the best on fa. The other methods - LLOOMFSFFT, LLOME--PO, LAMBO and the baseline GA- - lag behind. Among the LLOME variants, LLOME-SFT and LLOME-MARGE achieve signif- icantly higher rewards than LLOME-DPO (Figs. 14a, 14b). Test function f2 (Figure 3) is particularly interesting case study, since the different methods have well-separated
6. Results
6.1. Solver Benchmark Results
State-of-the-art LLMs struggle to solve Ehrlich func- tions even with full problem specification. As an initial test, we prompted OpenAT's ol (accessed on Jan. 28, 2025) and Google's Gemini 2 Flash Thinking (named 'Gemini 2.0 Flash Thinking Experimental 01-21' in Google AI Studio) models with description of f2. including the full transi- tion matrix, all constraints, and a few in-context examples. (See prompt in A.5.) With only samples, o1 and Gemini Flash achieve minimum regrets of 0.9375 and O, respec- tively. However, most outputs were either infeasible or had regret close to indicating that this is still a challenging
7

---

Generalists vs. Specialists: Evaluating LLMs on HlighlyConnstrineed Biophysical Sequence Optimization Tasks
regret curves. We thus focus on f2 for further analysis.
larger edit to the input sequence, the output is less likely to he feasible (Fig. 15). For both LLoOE--FF and LLOME- MARGE, edit larger than 0.3 Hamming distance away from the input is less than 20% likely to be feasible. Since the Ar threshold we set for the PropEn dataset formatting algorithm (Alg. 2) is 0.25, this is unsurprising. The LLM is never been trained on edits of larger than 0.25 Hamming dis- tance. Overall, LLOME- MAMM- exhibits the best tradeoff of all methods, producing the largest proportion of unique and feasible sequences for the smallest budget of test func- tion evaluations and for moderately sized edits of Hamming distance < 0.3 (Fig. 15).
How do specialized models compare with generalist mod- els? At first glance, Figure 3 seems to show that generalist models like LLMs can outperform specialized models like LAMBO-2. However, one factor hidden in these plots is the computational cost of different methods. For rela- tively easy optimization problems, since the performance of various methods is similar, using a specialized model with 0.01% of the parameters of an LLM may be more practical. In addition, LAMBO-2 offers precise control over the sequence design space (through specifying num- ber of tokens to mutate and the desired maximum edit dis- tance). This allows domain experts to tune the algorithm with appropriate hyperparameters. Nonetheless, LAMBO-2 is sensitive to hyperparameter choice and requires custom tuning for each test function, whereas LLOME performs well across multiple functions without custom tuning. In Figs. 4 and 13 we plot the diversity and the feasibility of each method's designs over the course of optimization. We see that LLOME-N EMAMGER strikes a careful balance of explo- ration and exploitation without custom tuning. However, LAMBO-2 proposes more diverse and feasible designs than LLOME-MARGE, but requires custom tuning for each test function.
6.2. Understanding/Ablating LLOME
Here we summarize various aspects of LLOME's perfor- mance, deferring sections on LLOME's test-time extrapola- tion abilities and sensitivity to changes in hyperparameters, evaluation budget, and presolver to App. A.9.
LLMs are moderately effective at ranking their own outputs. The iterative refinement process requires that the LLM rank and filter its own outputs (Alg. 4), but we have not yet considered how effective the LLM is at selecting the best candidates. When compared to oracle selection (i.e. using the ground-truth score to select candidates), we find that the likelihood method often selects high-scoring but not the highest scoring candidates (Fig. 18). To better understand this gap between likelihood and oracle selection, we plot the calibration curve of regret versus model likeli- hood in Figure 5a. A a steep linear trend. The LLOME- DPO calibration curve monotonically decreases with regret. but also fails to gen- cralize, producing the fewest sequences with regret outside of the training distribution. In contrast, LLOME-MAREE and LLOME-SFT assign higher likelihoods to lower regret sequences, but also exhibit some degree of overconfidence. Indeed, Fig. 17 shows that the likelihood and reward of SFT- and HargE-geneeated sequences are inversely corre- lated for high likelihoods exceeding 0.7. We hypothesize that LLMs may become increasingly miscalibrated as their outputs extend beyond the training distribution.
SFT- and MargE-trained LLMs generate unique and fea- sible sequences, but DPO-trained LLMs struggle. Sue- cessfully solving a highly-constrained optimization problem requires that the model be able to generate a diverse array of feasible sequences. Compared to the GA, Figures 4 and 13 show that LAMBO-2, MARGE, and LLOME- SFT produce significantly higher proportions of unique sequences. However, these three methods also experience an initial dip in the percentage of feasible outputs before learning to satisfy their constraints. In contrast, LLOME- DPO does not improve either the diversity or the feasibility of its outputs even when provided with more oracle labels. In some of our experiments, the LLOME-DPP pipeline ends prematurely due to producing an insufficient number of unique sequences to seed the next round. Like other past works (Pal et al., 2024; Rafailov et al., 2024b; Feng et al., 2024; Pang et al., 2024), we observe that the likelihood assigned by the DPO-trained LLM to both yor and y con- tinuously declines throughout training, implying that prob- ability mass is moved to sequences outside of the training distribution. Since the percentage of infeasible sequences generated by LLOME--DO increases over multiple itera- tions, it is likely that DPO moves some probability mass to infeasible regions of the sample space. As such, DPO may be ill-suited for training solvers for constrained problems.
model would show
When is explicit reward information required during training? Since we have observed that LLOME-SFT and LLOME-MARGE perform similarly up to a certain point (Fig. 3), we might hypothesize that incorporating explicit reward values into the training objective is only necessary once the LLM is closer to the optimum. Since MargE re- quires a larger memory footprint than SFT (due to the need to store and compute likelihoods with both ne and Teer and collecting ground truth rewards may be expensive, train- ing the LLM with further rounds of SFT before switching to MargE would be more efficient than relying mostly on MargE training. We test this hypothesis via a multi-stage
Although LLMS are capable of generating new feasible sequences, we also find that they suffer from the classic explore versus exploit tradeoff. When the LLM makes
8

---

Generalists vs. Specialists: Evaluating LLMs on HoighlyConsttainee Biophysical Sequence Optimization Tasks
1.0
1.0
Average Hegret
Minimum Regret
0.8
0.9
0.0
10.FT, o MargF
9 MaraE
0.4
7FT. 3 MargE
Mimimum
6.8
LLONE-OFO LLOME-MARGE
5:FT, 5 MargE
0.2
3 3... 7 Margi:
0.2
LOME MARGE
MargE
LOOME REINFORCE
04
0.0
5000
10000 15000 20000 25000 30000
5000
10000 15000 20000 25000 20000
Likelhoodd
Function Evaluations
Number Function Evaluntions
(a) Likelihood vs. regret.
(b) Combining SFT with MargE.
(c) MargE vs. REINFORCE.
Figure 5: We examine the design principles behind MargE, including calibration of likelihood vs. regret (5a), the importance of ground truth rewards (5b), and the difference between MARGE and policy gradient methods like REINFORCE (5c).
pipeline, where early rounds of LLOME use SFT training, and later rounds use MargE. We keep the total number of LLM training rounds constant at 10 (unless the LLM finds the optimal solution earlier), evaluate on test function f2. and vary the proportion of rounds that use SFT versus E, as shown in Fig. 5b. We refer to the pipeline with i rounds of SFT training followed by j rounds of MargE training as LLLOE-SFFTT--AEEE
optimization setting with finite data, where future rounds may require further exploration. We apply the same impor- tance sampling and nofaaoormaiiztioon from the MargE loss to the KL-regularized REINFORCE loss:
The (ylx) TRee (y/x) y(x,)) log (y(xxx)-> log no (y|xx)
E x- y-uTRel-
lyl
We find that LLOME- SFT:-MARGE, not only is the sole pipeline to find the optimal solution (Fig. 5b), but also achieves the best cal- ibration curve (Fig. 19). In contrast, switching from SFT to MargE training at an intermediate point results in the worst performance. It appears that SFT and MargE may have conflicting loss landscapes - if we first train for a few rounds with the SFT loss, subsequently switching to MargE training impedes the LLM's progress.
Fig. 5c shows that although creases the regret to some degree, it plateaus early and does not reach optimality, suggesting that the SIC criteria has meaningful influence on the LLM's optimization abilities.
de-
7. Conclusions
Our work is response to the lack of both non-trivial syn- thetic benchmarks for biophysical sequence optimizers and rigorous analyses of how LLMs perform on these highly- constrained optimization tasks in realistic settings. Our pro- posed test functions bear significant geometric similarities to real biophysical sequence optimization tasks and allow for rapid iteration cycles. In addition, although a wealth of work exists that adapts LLMs for biophysical tasks, few have studied the abilities of L.IMs to adhere to hard con- straints in realistic bi-level optimization settings. To that end, we propose and analyze LLOME, a framework for in- corporating LLMs in bilevel optimization algorithms for highly constrained discrete sequence optimization problems. We show that in some settings, LLOME discovers lower- regret solutions than LAMBO-2 and GA, even with very few test function evaluations. When combined with MargE training, LLOME is significantly more sample efficient than LLOME-SFT or LLOME-DPO, demonstrating its potential to be useful in data-sparse lab settings. However, in very easy or difficult tasks, specialized models have the advan- tage - they offer comparable regret for greater steerability and significantly lower computational cost. Our findings also highlight that LLMs can robustly extrapolate beyond their training data, but are occasionally miscalibrated and benefit from training with ground truth rewards.
Does fulfilling the Strong Interpolation Criteria (SIC) matter for LLM training? The drawbacks of SFT, DPO, and RLHF (as discussed in Sections 2 and 1) motivated the design of MargE as a training objective that (1) uses the ground truth reward, (2) is less complex than RLHF, (3) reinforces the probability of high-reward outputs, (4) does not continuously increase output lengths, and (5) fulfills SIC (Hu et al., 2024). While criteria (1)-(4) have been discussed in prior sections, it remains to be seen whether (5) is indeed integral for training an LLM to optimize well.
To evaluate the impact of the SIC criteria on LLM training, we compare MARGE with REINFORCE (Williams, 1992). a policy gradient method that directly optimizes for maximal reward but does not fulfill SIC. We choose REINFORCE because it is similar in principle to MargE and has been shown to perform comparably (Ahmadian et al., 2024) to RLHF (Stiennon et al., 2020; Ziegler et al., 2019) and RL- free variants such as DPO (Rafailov et al., 2023) and RAFT (Dong et al., 2023). The effect of SIC is such that MargE smoothly interpolates between x² and TRef whereas the KL- regularized REINFORCE algorithm interpolates between T (a degenerate
distribution) and TRef; thereby violating SIC. In theory, moving the policy towards n encourages collapse, which is undesirable in an online
9

---

Generalists vs. Specialists: Evaluating LLMs on HlighlyConnstraneed Biophysical Sequence Optimization Tasks
Impact Statement
sequence design. In International conferennce on machine learning, pp- 324-334, PMLR, 2ML.,
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Arkhipov, M., Trofimova, M., Kuratov, Y., and Sorokin, A.
Tuning multilingual transformers for language-specifge named entity recognition. In Erjavec, T., Marciñczuu, M., Nakov, P., Piskorski, J., Pivovarova, L.,, Snajder, J., Stein- berger, J., and Yangarber, R. (eds.), Proceedings the 7th Workshop on Balto-Slavic Natural Language Processing, pp. 89-93, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/119-333 URL wttpt:tt//// pestps://aclanttholo..
Acknowledgements
We thank Natasa Tagasovska and the Prescient Design LLM team for valuable discussions and feedback during the development of this project. We additionally thank Miguel González-Duque for support with incorporating the LaMBO-2 method into the poli poli-baseling tory and for continued maintenance of pol pooli-oe wal-baaalliee nies. The authors also thank the Prescient Design Engineering team for providing HPC and consultation resources that con- tributed to the research results reported within this paper.
arg/W11 9-119-31 3712.
Arnold, F. H. Design by directed evolution. Accounts of chemical research, 31(3):125-133) 1998.
reposi-
Back, T. Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms. Oxford university press, 1996.
References
Barlow, K. A., Conchuir, S., Thompson, S., Suresh, P.,
Lucas, E., Heinonen, M., and Kortemme, T. Flex ddg: Rosetta ensemble- estimation of changes in protein- protein binding affinity upon mutation. The Journal of Physical Chemistry B, (2((1)):33995399) 2018.
Ahmadian, A., Cremer, C., Gallé, M., Fadace, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties Volume I: Long Papers), pp. 12248-12267, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10.18653/v1/20044adlloo
Barrera, A., Vedenko, A., Kurland. J. V., Rogers, M.. Gisselbrecht, S. S., Rossin, E. J., Woodard, J., Mari- ani, L., Kock, K. H., Inukai, S., Siggers, T., Shokri, I.., Gordán, R., Sahni, N., Cotsapas, C., Hao, T. Yi, S.. Kellis, M., Daly, M. J., Vidal, M., Hill, D. E., and Bulyk, M. L. Survey of variation in human tran- scription factors reveals prevalent dna binding changes. Science, 351 36280))) 1450-1450441 1454, 2016a. doi: 10. 126/ science.aad2257. URL https: htpp:://wwww.. science. org/doi//n /10bs///1..o 1126 http://scennceeccec aad2257.
662. URL https////aaatttoog....o. wrg.://0..r.g//20..///// acl acl-long. 662.
AhmadiTeshnizi, A., Gao, W. and Udell, M. Optimus: Scalable optimization modeling with (mi) lp solvers and large language models. arXiv preprint 2ix::2442..1172 2024.
Barrera, L. A., Vedenko, A., Kurland, J. V., Rogers, J. M., Gisselbrecht, S. S., Rossin, F., J., Woodard, J., Mariani, L., Kock, K. H., Inukai, S., al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science, 351 (6280) 1450-14544 2016b.
Ahmed, T. and Choudhury, S. Lm4opt: Unveiling the po- tential of large language models in formulating math- ematical optimization problems, 2024. URL https: //arxivvoory abs /2403.4022 01342.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' Brien, K., Hallahan, E., Khan, M. A., Purohit, S.. Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Amin, S. and Neumann, G. T2NER:
Transformers based transfer learning framework for named entity recogni- tion. In Gkatzia, D. and Seddah, D. (eds.), Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguistics: System Demonstrations, pp. 212-220, Online, April 2021. Association for Computational Linguistics. doi: URL https:// aclant aclanthology org/ eacl demos. 25.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R. Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, F., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K. A., Davis, J., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy. J., Ethayarajh, K., Fei-Fei, I.., Finn, C., Gale, T., Gillespie, L. E.,, Goel, K.,
Angermueller, C., Belanger, D., Gane, A., Mariet, Z., Dohan, D., Murphy, K., Colwell, L., and Sculley, D.
Population-bast -base black-box optimization for biological
10

---

Generalists vs, Specialists: Evaluating LLMs on Highhy- Biophysical Sequence Optimization Tasks
Goodman, N. D., Grossman, S., Guha, N., Hashimoto,
Freire, P. Wang, T., Marks, S., Segerie, C.-R., Carroll, M., Peng, A., Christoffersen, P., Damani, M., Slocum,
T., Henderson, P., Hewitt, J., Ho, D. E., Hong, J., Hsu, K., Huang, J., Icard, T. F., Jain, S., Jurafsky, D., Kalluri, P.. Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M. S., Krishna, R., Kuditipudi, R., Kumar, A., Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li, X. L., Li, X., Ma, T., Malik, A., Man- ning. C. D., Mirchandani. S. P., Mitchell, E., Munyikwa, Z., Nair, S., Narayan, A., Narayanan, D., Newman, B., Nie, A., Niebles, J. C., Nilforoshan, H., Nyarko, J. F., Ogut, G., Orr, L.., Papadimitriou, I., Park, J. S., Piech, C., Portelance, E., Potts, C., Raghunathan, A., Reich, R., Ren, H., Rong, F., Roohani, Y. H., Ruiz, C.. Ryan, J., R'e, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K. P., Tamkin, A., Taori, R., Thomas, W., Tramèr, F., Wang, R. E., Wang, W. Wu, B., Wu, J., Wu, Y., Xie, S. M., Yasumaga, M., You, J., Zaharia, M. A., Zhang, M., Zhang, T., Zhang, X., Zhang, Y., Zheng, L., Zhou, K., and Liang. P. On the opportunities and risks of foundation models. ArXiv, 2021. URL https: //cr //erfm. at stanford. www.d//asttsseppporreeeee .df. pdf.
S., Anwar, U., Siththaranjan, A., Nadeau, M., Michaud,
E. J., Pfau, J., Krasheninnikov. D., Chen, X., Langosco, I.., Hase, P., Brysk, E., Dragan, A., Krueger, D., Sadigh, D., and Hadficid-Mendd D. Open problems and funda- mental limitations of reinforcement learning from human feedback, 2023. URL https: www.i.wrgg......aaa..ooo..oooott 2307. 1307.15217.
Chan, A., Madani, A., Krause, B., and Naik, N. Deep extrapolation for attribute-enhanced genera- tion. In Ranzato, M., Beygelzimer, A., Dauphin, Y.. Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, vol- ume 34, pp. 14084-14096. Curran Associates, Inc., 2021. URL (raaa0offs9eeeb33555993333223333 pdf.
Chaudhury, S., Lyskov, S., and Gray, J. J. Pyrosetta: a script- based interface for implementing molecular modeling algorithms using rosetta. Bioinformatics, 26(5):689-691 2010.
Brown, N., Fiscato, M., Segler. M. H., and Vaucher, A. C. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3):1096-1108 2019.
Chen, A., Dohan, D. M., and So, D. R. Evoprompting: language models for code-level neural architecture search. In Proceedings of the 37th International Conference on Neural Information P Processing Systems, NIPS '23, Red Hook, NY, USA, 2024a. Curran Associates Inc.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbeet-
A., Krueger, G.,
Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https::t////c...... htttps://prcceedding. neurips. cc cc/paper httppeeeiieeeeppppee/00/////////
Chen, A., Malladi, S., Zhang, L. H., Chen, X., Zhang, Q.. Ranganath, R., and Cho, K. Preference learning algo- rithms do not learn preference rankings, arXiv preprint arXiv:2405. 19534, 2024b. URL https: /arxiv. org /t///bb//24000 /200 19534.
Chen, M. Tworek, J., Jun, H., Yuan, Q.. Pinto, H. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint 2rXiv:2107.03333 2021.
pdf.
Chen, X., Liu, T., Zhao, H., Zhou, G., and Zhang, Y.-Q. Cerberus transformer: Joint semantic, affordance and attribute parsing. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 19649-19658, 2022a.
Carlsson, F., Öhman, J., Liu, F,, Verlinden, S., Nivre, J., and Sahlgren, M. Fine-grained controllable text gen- eration using non-residual prompting. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume I: Long Papers), pp. 6837- 6857, Dublin, Ireland, May 2022. Association for Com- putational Linguistics. doi: 10.186633/1122022.ac-oomo
Chen, Y.. Song, X., Lee. C., Wang. Z.. Zhang, R. Dohan, D.,, Kawakami, K., Kochanski, G., Doucet, A.. Ranzato, M. A., Perel, S., and de Freitas, N. Towards learning universal hyperparameter optimizers with transformers. In Koyejo, S., Mohamed, S., Agar- wal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 32053-33 -2068.. Curran Associates, Inc.,
471. URL https:/ acl bol-long.44 471.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T. Lindner, D.,

---

Generalists vs, Specialists: Evaluating LLMs on Hlighly-onsstraneed Biophysical Sequence Optimization Tasks
2022b. URL http:://proceeding..eerrrt ccc/ppper_fillesspppee/////////e #1651100fccc72ee6ccee211..113333 Contrr-Contg 2023. ISSN 2835-8856. URL https: pdf.
RAFT: Reward ranked finetuning for generative founda- tion model alignment. Transactions Machine Learn-
//openreview.r net forum? ftt.m/dmmmmmmmmmmm0000
Drozdov, A., Schärli, N., Akyürek, E., Scales, N., Song, X., Chen, X., Bousquet, O., and Zhou, D. Compositional semantic parsing with large language models. In The Eleventh International Conference on Learning Represen- lations, 2023. URL https: wow.reveeeeeeeeeeeeeeeeee net/ forum?id forumt:id-g.wehssBByst
Chen, Y., Xu, B., Wang, Q. Liu, Y., and Mao, Z. Bench- marking large language models controllable gener- ation under diversified instructions. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16): 17808-17816, Mar. 2024c. doi: (0.1609/aaii.v3811 v38i16. 29734. URL php/
orq/ index.
Du, L., Torroba Hennigen, L., Pimentel, T., Meister, C., Eisner, J., and Cotterell, R. A measure-theoreeie char- acterization of tight language models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61 Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pp. 9744 9770, Toronto, Canada, July 2023. Association for Com- putational Linguistics. doi: 10. (0.18653/w1/20233acluu 543. URL aattss://aclast hrw.clannhhllloyoor//22222222 acl- 2el-long-o 143.
Chinery, L., Hummer, A. M., Mehta, B. B., Akbar, R., Rawat, P., Slabodkin, A., Le Quy, K., Lund-Johansen, E,, Greiff, V., Jeliazkov, J. R., et al. Baselining the buzz. trastuzumab-hert affinity, and beyond! bioRxiv, pp. 2024 03, 2024.
Cieplinski, T., Danel, T., Podlewska, S., and Jastrzebski, S. Generative models should at least be able to design molecules that dock well: new benchmark. Journal of Chemical Information and Modeling, 63(11):3238-3247, 2023.
Feng, D., Qin, B., Huang, C., Zhang, Z., and Lei, W. To- wards analyzing and understanding the limitations of dpo: A theoretical perspective, 2024. URL https: /algg//////// abaaaass/2/2/ /2404 04626.
Dathathri, S., Madotto, A., Lan, J.. Hung. J., Frank, E., Molino, P., Yosinski, J., and Liu, R. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020. URL https::t////seeeee http:////eeeeeeeeee/mmmmmeeeeee. net/ fest/Eorumiien (im (e-.rummmmd-eHeeeBBBB0B0
//arxiv. org/
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00002 2020.
Dekoninck, J., Fischer, M., Keurer-Kellner, L., and Vechev, M. Controlled text generation via language model arithmetic. In The Twelfth International Conference on Leurning Representations, 2024. URL https: openreview. wwwpeeeewwwwweeeeeeeeeeeeeeeeeee tete/foormmii -ww.:::w--dWwWWwWWLWWS
Gao, I.., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang. Y., Callan, J., and Neubig, G. Pal: Program-aided language models. arXiv preprint arXiv:2211.0043 2022.
Garbacea, C. and Mei, Q. Why is constrained neural lan- guage generation particularly challenging?, 2022.
Deng, H. and Raffel, C. Reward-augmented decoding: Effi- cient controlled text generation with unidirectional re- ward model. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 1 1771111791, Singapore. December 2023. Association for Computa- tional Linguistics. doi: 10.18663/1112023emmmppp 721. URL https: / /aaaaclaaaa /ww.aaaahlllooyoo..oooocoooooooa org/2023. emnlp-man matommaiin.... 721.
Gillespie, J. H. Population genetics: concise guide. JHU press, 2004.
Gligorijević, V., Berenberg, D., Ra, S., Watkins, A., Kelow, S., Cho, K., and Bonncau, R. Function-guided protein design by deep manifold sampling. bioRxiv, pp. 2021-12, 2021.
Graff, D. E., Shakhnovich, E. I., and Coley, C. W. Accelerat- ing high-throughput virtual screening through molecular pool-based active learning. Chemical science, 12(22): 7866-7881, 2021.
Doan, X.-D. Viec-nlp at nl4opt competition subtask 1: ensemble pre-trained language models for named en- tity recognition, 2022. URL https: /aaxiv.vvvoo///// org/ fbas/221..077 07211.
Gruver, N., Stanton, S., Frey, N., Rudner, T. G., Hotzel, I. Lafrance- Vanasse, J., Rajpal, A., Cho, K., and Wil- son, A. G. Protein design with guided discrete diffusion. Advances in neural information processing systems, 36, 2024.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., SHUM, K., and Zhang, T.
12

---

Generalists vs, Specialists: Evaluating LLMs
Biophysical Sequence Optimization Tasks
Guo, D., Yang, D., Zhang, H., Song, J., Zhang. R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., al. Deepseck-rl: In- centivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint 2rXiv:2501. 12948, 2025.
Hummer, A. M., Schneider, C., Chinery, L., and Deane, C. M. Investigating the volume and diversity of data needed for generalizable antibody-antigen óōg prediction. bioRxiv, pp. 2023-05, 05, 2023.
Ismail, A. A., Oikarinen, T., Wang, A., Adebayo, J., Stanton, S., Joren, T., Kleinhenz, J., Goodman, A., Bravo, H. C., Cho, K., et al. Concept bottleneck language models for protein design. arXiv preprint 17Xiv:241.06090 2024.
Guo, P.F.,, Chen, Y.-H., Tsai, Y.-D., and Lin, S.- D. Towards optimizing with large language models. Learning,
In Fourth Workshop on Knowledge-infie 2024a. URL https://s///ot //per////ppeeeeeeeeeeeeeee wew. ew. net/forunt id=vIU8LUckb4.
Jang, S. Tag embedding and well-defined intermediate representation improve aute-formulation of problem de- scription. arXiv preprint arXiv:211..3355,20000 2022.
Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Confer- ence on Learning Representations, 2024b. URL https: / //openreview.oo net. /tweeeeoouuueeee forum? 1ee22iddd2c- 2addddd22II000.
Jaynes, E. T. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Jeon, H. J., Milli, S., and Dragan, A. Reward-rations (im- plicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33: 4415 4425-4426, 2020.
Guo, Z., Liu, M., Ji, Z Bai, J., Guo, Y., and Zuo, W. Two optimizers are better than one: Lim catalyst empowers gradient-based optimization for prompt tuning, 2024c. URL httphhtp///// /tt////ara arxiv. org/ abs/ 2405 19732.
Johnston, K. E., Almhjell, P. J., Watkins-Dulaney E. J., Liu, G., Porter, N. J., Yang, J., and Arnold, F. H. A combinatorially complete epistatic fitness landscape in an enzyme active site. Proceedings of the National Academy of Sciences, 1210G2):e2404431122 2024. doi: 10.1073/ pnas p400.3400439121. URL hitps: /tt/:////wwww/// /www pnas org/ doi /oi/abs/// / 1 / 2400449122.
Gupta, S., Shah, R., Mohit, M., Kumar, A., and Lewis, M. Semantic parsing for task oriented dialog using hierar- chical representations. arXiv preprint arXiv: 180.00810 2018.
He, J., N, M., Vignesh, S., Kumar, D., and Uppal, A. Linear programming word problems formulation using ensem- blecrf ner labeler and i5 text generator with data aug- mentations, 2022. URL httht: //tptiii:///xi....// org/ abs/ 2212.14651
Jones, D. R., Schonlau, M., and Welch, W. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13:455-499 1998.
Kellogg, E. H., Leaver-Fay, A., and Baker, D. Role of conformational sampling in computing mutation-induced changes in protein structure and stability. Proteins: Struc- ture, Function, and Bioinformatics, 79(3):830-82) 2011.
He- Yueya J., Poesia, G., Wang, R. E., and Goodman, N. D. Solving math word problems by combining lan- guage models with symbolic solvers. arXiv preprint arX1v:2304.09102, 2023.
Keskar, N. S., McCann, B., Varshney, L., Xiong, C., and Socher, R. CTRL - A Conditional Transformer Lan- guage Model for Controllable Generation. arXiv preprin! arXiv:1909.05e0 05858, 2019.
Hie, B. L., Shanker, V. R., Xu, D., Bruun, T. U., Weiden- bacher, P. A., Tang, S., Wu, W., Pak, J. E., and Kim, P. S. Efficient evolution of human antibodies from gen- eral protein language models. Nature Biotechnology, 42 (2):275-288) 283,,222.2220 2024.
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., San- thanam, K., A, S. V., Haq, S., Sharma, A., Joshi, T. T. Moazam, H. Miller, H., Zaharia, M., and Potts, C. DSPy: Compiling declarative language model calls into state-of- the-art pipelines. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / / openrevi ew net teree/forunee wtt.fformmmmoooooo (ocrummm.d==25502225555555
Hu, X., He, T., and Wipf, D. New desiderata for direct prefer- ence optimization. In ICML 2024 Workshop on Models of Human Feedback for Al Aligument, 2024. URL https: //openreview.oomm net forum? Fd-FFfdiioo222222
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Ham- bro, E.,, Grefenstette, E., and Raileanu, R. Understanding the effects of RLHF on LLM generalisation and diver- sity. In The Twelfth International Conference on Learning Representationss 2024. URL https: /etetss://oooe opw.revie...... net www.oooruunmeeooooo FFA..T.
Huang, K., Fu, T., Gao, W., Zhao, Y., Roohani, Y., Leskovec, J., Coley, C. W., Xiao, C., Sun, J., and Zitnik, M. Ther- apeutics data commons: Machine learning datasets and tasks for drug discovery and development. arXiv preprint arXiv:2 22..200209990 02544448 2021.
13

---

Generalists vs. Specialists: Evaluating LLMs HlighlyConnstranne Biophysical Sequence Optimization Tasks
Knoblauch, J., Jewson, J., and Damoulas, T. Generalized variational inference: Three arguments for deriving new posteriors. arXiv preprint arXiv:1904.000 2019.
Luce, R. D. et al. Individual choice behavior, volume 4. Wiley New York, 1959.
Ma, P., Wang, T.-H., Guo, M., Sun, Z., Tenenbaum, J. B., Rus, D., Gan, C., and Matusik, W. LLM and sim- ulation as bilevel optimizers:
Korbak, T., Elsahar, H., Kruszewski, G., and Dymetman, M. On reward maximization and distribution matching for fine-tuning language models, 2022a. URL https: //openreview wwt 8tt.:iddd--f9af.
new paradigm ad- vance physical scientific discovery. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 33940-33962, PMIR, 21-27 Jul 2024. URL https: : proceedings.oo mlr press/v2ss3 3m/335/mm2222 /aa 4m. html.
Korbak, T., Perez, E., and Buckley, C. L.. RI with penal- ties is better viewed as bayesian inference, 2022b, URL https: /arxiv..vvvv org/ abs/ 22022202.. 11275.
Lange, R., Tian, Y., and Tang, Y. Large language mod- els as evolution strategies. In Proceedings of the Ge- netic and Evolutionary Computation Conference Com- panion, GECCO '24 Companion, pp. 579-582, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704956.
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., Fraser, S., and Naik, N. Large language models generate functional protein se- quences across diverse families. Nature Biotechnology, 41(8):109--110 January 2023. ISSN 1546-1696, doi: 10. 00.100886415887-00--00161888 URL http: dx doi. 0r3/1D..033//44133--022--
doi: 10.1145/ 3638530.3654238. URL /tttp:://di...o 1145/3638530 http//363853..65523333
Lehman, J., Gordon, J., Jain, S., Ndousse, K., Yeh, C., and Stanley, K. O. Evolution through large models. In Handbook of Evolutionary Machine Learning, pp. 331- 366. Springer, 2023.
Malladi, S. The hidden infinity in preference learning, July 2024. URL https://www princeton.ed::/ -dd/eduudesssee sma 1asaalladd/blo/2222222 27/dpo- innooinfiiityy...
Li, H., Ma, Y., Zhang, Y., Ye, C., and Chen, J. Exploring mathematical extrapolation of large language models with synthetic data. In Ku, I..W.,, Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Lin- guistics ACL 2024, pp. 936 966,946, Bangkok, Thailand and virtual meeting, August 2024. Association for Computa- tional Linguistics. doi: 10. 10.18653/v1/2024.1mm 10118555///2244mmmmgggelll
Mason, D. M., Friedensohn, S., Weber, R., C., Wag- ner, B., Meng, S. M., Ehling, R. A., Bonati, L., Dahinden, J., Gainza, P., et al. Optimization of therapeutic antibodies by predicting antigen specificity from antibody sequence via deep learning. Nature Biomedical Engineering, 5(6): 600-612, 2021.
55. URL https: http///aaaaatltoyo...oeo wwtp::/oooloyy../222222.. Sindings-acl.555
Maus, N., Jones, H., Moore, J., Kusner, M. J., Bradshaw, J., and Gardner, J. Local latent space bayesian optimization over structured inputs. Advances neural information processing systems, 35:34505-34516 2022.
Liu, A., Sap, M., Lu, X., Swayamdipta. S., Bhagavatula, C., Smith, N. A., and Choi, Y. DExperts: Decoding- time controlled text generation with experts and anti- experts. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers). pp. 6691-6706, Online, August 2021. Association for Computational Linguis- tics. doi: 10. 10.116633/112002..acclcc 522. URL https: a/acl antho ogy. www.g.ooorg//ogo /20p//202112 acl lowecllllgggg.llo long..gg..... 522.
Meng. Y., Xia, M., and Chen, D. SimPO: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.1477: 2024.
Meyerson, E., Nelson, M. J., Bradley, H., Gaier, A., Moradi, A., Hoover, A. K., and Lehman, J. Language model crossover: Variation through few-shot prompt- ing. ACM Trans. Evol. Learn. Optim., September 2024. doi: 10. 10.1145/3694791 URL https: : /oi. or0 1145/3694791. Just Accepted.
Liu, S., Chen, C., Qu, X., Tang, K., and Ong, Y.-S. Large language models as evolutionary optimizers. In 2024 IEEE Congress on Evolutionary Computation (CEC), pp-
1-8, 2024a. doi: 10.1 00.10099CEC00001.2024
Mittal, C., Kartik, K., Mausam, and Singla, P. Puz- zlebench: Can llms solve challenging first-order com- binatorial reasoning problems?, 2024. URL https: //arxiv.org/abs/2222 02611.
Liu, T., Astorga, N., Seedat, N., and van der Schaar, M. Large language models to enhance bayesian optimization. arXiv preprint 2xxv:24002.3921 2024b.
14

---

Generalists vs. Specialists: Evaluating LLMs on Ilighly-Constrained Biophysical Sequence Optimization Tasks
Močkus, J. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference Novosibirsk, July 1-7, 1974 6, pp. 400-404. Springer, 1975.
Ortega, P. A. and Braun, D. A. Thermodynamies theory of decision-making with information-proceestti costs. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 469(21533)201266333 2013.
Molga, M. and Smutnicki. C. Test functions for optimization needs. Test functions for optimization needs, 101:48, 2005.
Owen, A. B. Monte Carlo theory, methods and examples.
https:/ / arrtowen. su. doma ins ww/.smmo //c//////// 2
Nasir, M. U., Earle, S., Togelius, J., James, S., and Cleghom, C. Llmatic: neural architecture search via large language models and quality diversity optimization. Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1110-118, 2024.
Padmakumar, V. and He, H. Does writing with language models reduce content diversity?, 2024. URL https:
/ //arxiv.org aagg///b////// 2309. 051 96.96.
Padmakumar, V., Pang. R. Y., He, H., and Parikh, A. P. Extrapolative controlled sequence generation via iterative refinement. In International Conference on Machine Learning, pp. 26792-26808. PMLR, 2023.
Neidhart, J., Szendro, I. G., and Krug. J. Adapta- tion in tunably rugged fitness landscapes: The rough mount fuji model. Genetics, 198(2):699-721, August 2014. ISSN 1943-2631. doi: 10. 10.1534/geneties.11 14. 167668. URL http://dxx don..doi..oo org/ genetics, 114. 167668.
Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024. URL https:// 22.00.112222
arxiv. rrxiv.org/abae
Nie, A., Cheng, C.-A., Kolobov, A., and Swaminathan, A. Importance of directional feedback for LLM-based optimizers. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023. URL https:// openreview net/ forum? 1---11eG5643.....
Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimiza- tion, 2024. URL https: /twii/ixvvv/v/////vv//e//////atl org/ abs 2404. 19733.
Papkou, A., Garcia-Pastor, L., Escudero, J. A., and Wag- ner, A. A rugged yet easily navigable fitness land- scape. Science, 382(6673)caah33600, 2023. doi: 10.1126/ science. http:/seeenc..aa.e....
Nijkamp, E., Hayashi, H., Xiong, C., Savarese, S., and Zhou, Y. Codegen2: Lessons for training Ilms on programming and natural languages. ICLR, 2023a.
sch3nce.adh3860. URL https: org/doi/abe abs /10.
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y. Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthe- sis. ICLR, 2023b.
Pascual, D., Egressy, B., Meister, C., Cotterell, R., and Wattenhofer, R. A plug-and-play method for con- trolled text generation. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3973-3997, Punta Cana, Dominican Repub- lic, November 2021. Association for Computational Linguistics. doi: 10. (18666//1/2021...nnnnng--mmm
Ning, Y., Liu, J., Qin, L., Xiao, T., Xue, S., Huang, Z., Liu, Q. Chen, E., and Wu, J. A novel approach for auto- formulation of optimization problems. arXiv preprint arXiv:2302.04643 2023.
334. URL https: http///aaaaaaahhce/e wwtpg/..rrr//// 2021. Finding Findings-emmlpp.33 334.
Notin, P., Kollasch, A., Ritter, D., van Niekerk, L., Paul, S., Spinner, H., Rollins, N., Shaw, A., Orenbuch, R., Weitz- man, R., Frazer, J., Dias, M., Franceschi, D., Gal, Y., and Marks, D. Proteingym: Large-scale benchmarks for pro- tein fitness prediction and design. In Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Sys- tems, volume 36, pp. 64331-64379. Curran Associates, Inc., 2023.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Poesia, G., Polozov, A., Le, V., Tiwari, A., Soares, G., Meek, C., and Gulwani, S. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations, 2022. URL https: /openreview net/ forum? Vd=KmttV97.J433
Ogden, P. J., Kelsic, E. D., Sinai, S., and Church, G. M. Comprehensive aav capsid fitness landscape reveals a viral gene and enables machine-guides guided design. Science, 336(6446) 143, 2019.
15

---

Generalists vs. Specialists: Evaluating LLMs on HlihhyyConsttrinee Biophysical Sequence Optimization Tasks
Press, O., Smith, N., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapo- lation. In International Conference Learning Represen- tations, 2022. URL https: http:/ppeeeeeeeeeet.eeee n forum? d=R8sQPpGCvO.
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen,
X., Canny, J., Abbeel, P., and Song, Y. S. Evaluating protein transfer learning with tape, In Advances in Neural Information Processing Systems, 2019.
Comera-Paredess B., Barekatain, M., Novikov, A., Ba- log, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., Kohli, P., and Fawzi, A. Mathematical discoveries from program search with large language models. Nature, 625(7995): 468 475, December 2023. ISSN 1476-4687. doi: 10. 10.10088441566--03--66662444444 URL / /tw.//xxx.oooi////////o/////.... doi. org 1ttp:/0..1////41110 023 06924-6.
Pryzant, R., Iter, D., Li, J., Lee, Y., Zhu, C., and Zeng, M. Automatic prompt optimization with "gradient de- scent" and beam search. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7957-7968, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.1865//11/2233. mmnlp-main.494. 494. URL https://////.. /aclaattcocooyy.... org/2023. emn 1 1-mmpppamii n.rii..4444
Rongali, S., Soldaini, L., Monti, E., and Hamza, W. Don't parse, generate! a sequence to sequence architecture for task-oriented semantic parsing. In Proceedings of the web conference 2020, pp. 2962-2968, 2020.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019.
Sarkisyan, K. S., Bolotin, D. A., Meer, M. V., Usmanova, D. R., Mishin, A. S., Sharonov, G. V., Ivankov, D. N., Bozhanova, N. G., Baranov, M. S., Soylemez, O., et al. Local fitness landscape of the green fluorescent protein. Nature, 533(7603):397-441 2016.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https: www.r//eeeeeeeeeeeeeeee/ee /eett/eeeeeeeee/eee//eee forum? -id- HHH.HPSSIIE IXJaa9.
Schulman, J., Wolski, F. Dhariwal, P., Radford, A.. and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL https: http:///axvvvvvo /bs/
Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment al- gorithms, 2024a. URL patttss:/arriiie org/ abs/ 2406. 02900.
1707.0634
Schymkowitz, J., Borg, J., Stricher, F., Nys, R., Rousseau, F., and Serrano, L. The foldx web server: online force field. Nucleic acids research, 2005.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From SrS to $q""$: Your language model is secretly q- function. In First Conference on Language Modeling, 2024b. URL https: http://pppeeeeeeeeeeeeeeeee htt.rviiiinnnnneeeeeeeeeeeeeeeee met/forunt id=kEVeNxt.GX..
Shao, B., Gong, Y., Qi, W., Cao, Ji, and Lin, X. Graph- based transformer with cross- cons-daad verification for semantic parsing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8807-8814, 2020.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), January 2020. ISSN 1532-4435.
Shaw, P., Massey, P., Chen, A., Piccinno, F., and Altun, Y. Generating logical forms from graph representations of text and entities. In Korhonen, A., Traum, D., and Mårquez, L. (eds.), Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pp. 95-106, Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10.18653/11/19-10000. URL https: httpaaaaaannnhooooooooooo orgg/p9-1111.
Ramamonjison, R., Yu, T., Li, R., H., Carenini, G., Ghad- dar, B., He, S., Mostajabdaveh, M., Banitalebi-Dehkordde A., Zhou, Z., and Zhang, Y. NI4opt competition: For- mulating optimization problems based on their natural language descriptions. In Ciccone, M., Stolovitzky, G., and Albrecht, J. (eds.), Proceedings of the NeurIPS 2022 Competitions Track, volume 220 of Proceedings of Ma- chine Learning Research, pp. 103. PMLR, 28 Nov- 09 Dec 2022. URL https: pecetps:/pprooe http:/peeccc.s...c/oo/p mml.. press/v220/ ramamon 233 html.
Shi, P., Ng. P., Wang, Z., Zhu, H., Li, A. H., Wang, J., dos Santos, C. N., and Xiang, B. Learning contextual representations for semantic parsing with generation- augmented pre-training. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pp. 13806 13814, 2021.
16

---

Generalists vs. Specialists: Evaluating LLMs on Hiihhy--onsstrinee Biophysical Sequence Optimization Tasks
Simon, H. A. Rational choice and the structure of the environment. Psychological review, 63(2):129, 1956.
Tagasovska, N., Gligorijević, V., Cho, K., and Loukas, A. Implicitly guided design with propen: Match your data to follow the gradient, 2024. URL https://arxiia org/ abs/2405 18075.
Sinai, S., Wang, R., Whatley, A., Slocum, S., Locane, E., and Kelsic, E. D. Adalead: simple and robust adap- tive greedy search algorithm for sequence design. arXiv preprint arXiv:2010.022 12.241,, 2020.
Tang, Y., Guo, D. Z., Zheng, Z.. Calandricllo, D., Cao, Y.. Tarassov, E.,, Munos, R., Ávila Pires, B., Valko, M., Cheng, Y., and Dabney, W. Understanding the per- formance gap between online and offline alignment al- gorithms, 2024. URL pettps://arxiv...o abs/
Singhal, P. Goyal, T., Xu, J., and Durrett, G. A long way to go: Investigating length correlations in RLHF, 2024. URL https: http::://pperrvviiiee www.neettuuuueee id=sNtDKdcT1f.
Tripp, A., Simm, G. N. C., and Hemáández-Lobtto, J. M. A fresh look at de novo molecular design bench- marks. In NearlPS 2021 Al for Science Workshop, 2021. URL https:/ wwwp///pwwweeeeeeee/eeeeeoo net/forum? id=gS3XMun4cl_
Song. X., Tian, Y., Lange, R. T., Lee, C., Tang, Y., and Chen, Y. Position: Leverage foundational models for black-box optimization. In Forty-first International Con- ference Machine Learning, 2024. URL https: //openreview. net wwweet//ooummaeee (eeormmmiiddea222gn3333
Ushio, A. and Camacho-Collados, J. T-NER: An all-round python library for transformer- based named entity recog- nition. In Gkatzia, D. and Seddah, D. (eds.), Proceed- ings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pp. 53-62, Online, April 2021. Associ- ation for Computational Linguistics. doi: 10.18653/v1/ 2021 eecl--emoos.7. URL https: ha/aacllanhooooyy. o org 2cg/2021.22 eacl demos..
Stanton, S., Maddox, W., Gruver, N., Maffettone, P., De- laney, E., Greenside, P., and Wilson, A. G. Accelerating bayesian optimization for biological sequence design with denoising autoencoders. In International Conference Machine Learning, pp. 10459-20477 PMLR, 2022.
Steinberg. D. M., Oliveira, R., Ong, S., and Bonilla, E. V. Variational search distributions. arXiv preprint arXiv:2409.06142,22 2024.
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. I., Milles, L. F. Dauparas. J., Baker, D., Ovchinnikov, S., Sercu, T., and Rives, A. Language models generalize beyond natural proteins. bioRxiv, pp. 2022-12, 2022.
Stengel-Eskin, E., Murray, K., Zhang, S., White, A. S., and Van Durme, B. Joint universal syntactic and semantic parsing. Transactions of the Association for Computa- tional Linguistics, 9:756-773, 2021.
Wang, B., Zheng, R., Chen, I.., Liu, Y., Dou, S., Huang, C., Shen, W., Jin, S., Zhou, E., Shi, C., Gao, S., Xu, N., Zhou, Y., Fan, X., Xi, Z., Zhao, J., Wang, X., Ji, T., Yan, H., Shen, L., Chen, Z., Gui, T., Zhang. Q. Qiu, X., Huang, X., Wu, Z., and Jiang, Y.-G. Secrets of rlhf in large language models part i:: Reward modeling, 2024. URL hww.htttp////.. /ttp//aaxvvv///////////////////a www.ivvorr orgg rrgg/b////// /401 06080.
Stiennon, N., Ouyang, I.., Wu, J., Ziegler, D., Lowe, R. Voss, C., Radford, A., Amodei, D., and Chris- tiano, P. F. Learning to summarize with human feedback. In Larochelle, H., Ranzato, M., Had- sell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, vol- ume 33, pp. 5008-3021. Curran Associates, Inc., 2020. URL http::::://pcccednnge.. http:rcceeeiing...eri.........ss neurips. co cc/paper_aam htwpeeeffilleeeppe////// (F898855556929 pdf.
Wang, J., Lisanza, S., Juergens, D., Tischer, D.,, Watson, J. L., Castro, K. M., Ragotte, R., Saragovi, A., Milles, L. F., Back, M., et al. Scaffolding protein functional sites using deep learning. Science, (7666)))) 2022.
Pappe..
Sun, J., Tian, Y., Zhou, W., Xu, N., Hu, Q., Gupta, R., Wieting, J., Peng, N., and Ma. X. Evaluating large lan- guage models on controlled generation tasks. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3155-3168, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/202. 112633//1/002..mmmllppp emnlp-main. main.190. URL https:// agl antho ogy. .rgg 2223. emnlp-ma 1ammal-mnnn i 190.
Wang. K., Chen, Z., and Zheng, J. Opd@n14oph An en- semble approach for the ner task of the optimization problem, 2023a. URL peattss:/aaxxv..oom 2301.02453
Wang, S., Sun, X., Li, X., Ouyang, R., Wu, F., Zhang, T., Li, J., and Wang, G. Gpt-ner: Named entity recognition via large language models, 2023b. URL https: //axiiv. org/ abs// 2304 10428.
17

---

Generalists vs, Specialists: Evaluating LLMs on Hlihhyy-onsstrance Biophysical Sequence Optimization Tasks
Welleck, S., Kulikov, I., Kim, J., Pang. R. Y., and Cho, K. Consistency of recurrent language model with respect to incomplete decoding. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 5553-5568, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ main.448. URL https:// aclanthology. org / 2020 emnlp-ma 1immmmi...... 448.
Zhang, S., Chen, Z.,, Shen, Y., Ding, M., Tenenbaum, J. B., and Gan, C. Planning with large language models for code generation. In The Eleventh International Conference on Learning Representations, 2023b. URL https:// open ew net / forum? ? i 8c00ttbb...
Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., Liu, Y.. Jin, S., Liu, Q., Zhou, Y., Xiong, L., Chen,
L., Xi, Z., Xu, N., Lai, W., Zhu, M., Chang, C., Yin,
Z. Weng, R., Cheng, W., Huang, H., Sun, T., Yan, H., Gui, T., Zhang, Q. Qiu, X., and Huang, X. Secrets of
Williams, R. J. Simple statistical truddent-following algo- rithms for connectionist reinforcement learning. Ma-
rlhf in large language models part Ppo, 2023a. URL https: www.aaaivvv..........a..a orgg abs 220../2307.00 04964.
chine Learning, 8(3-4):229-2555 May 1992. ISSN
1573-0565, doi: 10.1007//6100992666 URL http: //dx www.doo..co
Zheng, R., Dou, S., Gao, S., Shen, W., Wang. B., Liu, Y., Jin, S., Liu, Q. Xiong, L., Chen, L., Xi, Z., Zhou, Y., Xu,
N., Lai, W., Zhu, M., Weng, R., Cheng, W., Chang, C., Yin, Z., Hua, Y., Huang, H., Sun, T., Yan, H., Gui, T., Zhang, Q. Qiu, X., and Huang, X. Secrets of rlhf in large language models part i: Ppo. 2023b.
Wilson, J. Stopping bayesian optimization with probabilistic regret bounds. Advances in Neural Information Process- ing Systems, 37:92244-98296 2024.
Wu, K., Kim, K., Garnett, R., and Gardner, J. The be- havior and convergence of local bayesian optimization. Advances in neural information processing systems, 36: 73497-73523, 2023.
Zheng. X., Lin, H., Han, X., and Sun, L. Toward uni- fied controllable text generation via regular expression instruction. In Park, J. C., Arase, Y., Hu, B., Lu, W., Wijaya, D., Purwarianti, A., and Krisnadhi, A. A. (eds.), Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Confer- ence of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1-14, Nusa Dua, Bali, November 2023c. Associa- tion for Computational Linguistics. doi: 10.18653/v1/ 2023 1023.jjenlp-mate 1 URL https: http:///aaaaattorryy.. org/2023. 0ts/2023..jen..... m.in.
Wu, N. C., Dai, L., Olson, C. A., Lloyd-Smith, J. O., and Sun, R. Adaptation in protein fitness landscapes is facili- tated by indirect paths. Elife, 5:e16965, 2016.
Xu, M., Zhang, Z., Lu, J., Zhu, 7., Zhang, Y., Chang, M., Liu, R., and Tang, J. Peer: a comprehensive and multi- task benchmark for protein sequence understanding. Ad- vances in Neural Information Processing Systems, 35: 35156-35173, 2022.
Zhou, J., Ji, J.. Dai, J., and Yang. Y. Sequence to se- quence reward modeling: Improving rihf by language feedback, 2024. URL https:
Yan, H.. Deng, B., Li, X., and Qiu, X. Tener: Adapting transformer encoder for named entity recognition, 2019.
URL https: /ttp//aaxvvv///a///va///../..aaa org/ 11s/1 04474.
Zhou, W., Jiang, E.E. Wilcox, E.,, Cotterell, R., and Sachan, M. Controlled text generation with natural language instructions. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 42602-42613, PMLR, 23-29 Jul 2023. URL https:t/p//// httptttp:/pppooeeeeeeee v202 zhou23g html.
Yang, C., Wang, x., Lu, Y. Liu, H., Le. Q. V., Zhou, D., and Chen, X. Large language models as optimizers. In The Twelfth International Conference on Learning Represen- tations, 2024. URL hitps: //te:://pppeeeee http//ppppeeeeeett..tee/ forum? 1 d= 4 V.OWELLT.
Yuan, W., Kulikov, I., Yu, P., Cho, K., Sukhbaatar, S., We- ston, J., and Xu, J. Following length constraints in in- structions, 2024. URL https: /ttp::////vvvvvooeeee 2406.17749
mlr. mtt.presss.//r//m.mmmmmm.mm.
Zhu, X., Li, J., Liu, Y., Ma, C., and Wang. W. Distilling mathematical reasoning capabilities into small language models. Neural Networks, 179:106594, 2024.
Zhang, M., Desai, N., Bae, J., Lorraine, J., and Ba, J. Using large language models for hyperparameter optimization. In NeurlPS 2023 Foundation Models for Decision Mak- ing Workshop, 2023a. URL https:///// http///ppnee....eeeeeee net/ forum? i L-FUdZ6HEErr
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv: / /0t.//9990000 09593999........... 2019.
18

---

Generalists vs. Specialists: Evaluating LLMs on Hfighly-Constrained Biophysical Sequence Optimization Tasks
A. Appendix
A.1. Extended Related Work
LLMs for Optimization and Scientific Discovery Much of the work on LLMs for optimization has been inspired by the design of classic black-box optimizers (BBO) such as genetic algorithms, bayesian optimizers, and random search methods. BBOs are characterized by lack of information about the true objective function. Instead, only inputs and their corresponding objective values are provided the optimizer, with no gradient priors about the objective. Since these optimization problems are often expressible in formal mathematical, logical, or symbolic terms, many initial attempts at LLMs for optimization used the LLMs to first translate natural language description of the problem into code or a modeling language, prior to passing this formalization into an auxiliary solver (Ramamonjison et al., 2022; Ahmed & Choudhury, 2024; Mittal et al., 2024; AhmadiTeshnizi et al., 2024). This approach was often quite effective, as it utilized the wealth of past work on LLMs for named entity recognition (Amin & Neumann, 2021; Ushio & Camacho-Collados 2021; Wang et al., 2023b; Yan et al., 2019; Arkhipov al., 2019, NER), semantic parsing (Drozdov al., 2023; Gupta et al., 2018, Shaw et al., 2019; Shao et al., 2020; Rongali et al., 2020; Shi al., 2021; Chen al., 2022a; Stengel-Eskin et al., 2021), and code generation (Chen et al., 2021; Nijkamp et al., 2023b;a; Zhang et al., 2023b; Poesia et al., 2022). The NLAOPT competition (Ramamonjison et al., 2022), for example, decomposed L.LMs- BBO into two subtasks: (1) the translation of natural language description into a graph of entities and relations between the entities, and (2) a formalization of the optimization problem into a canonical logical reperesentation that could be solved by many commercial solvers. Winning approaches often employed ensemble learning (He et al., 2022; Wang et al., 2023a; Ning et al., 2023; Doan, 2022), adversarial learning (Wang et al., 2023a; Ning et al., 2023), and careful data pre-/post-proceeee processing and augmentation (He et al., 2022; Ning et al., 2023; Jang, 2022). Later work demonstrated that pre-trained LLMs like GPT-4 could achieve competitive performance on NLP4OPT without the NER stage (Ahmed & Choudhury, 2024), though the F1 score still trailed behind the state-of-the-art translation+NERR approaches from the winning NI entries. Outside of NLAOPT, Gao et al. (2022), Yueya et al. (2023), AhmadiTeshnizz et al. (2024), and Mittal et al. (2024) use an LLM to formalize math, mixed integer linear programming, and combinatorial reasoning problems from a natural language description before offloading the solving to a Python interpreter or symbolic solver. Each work notes that the LLM performs better in this decomposed framework than through prompting alone.
Yet other approaches tackle LLMS-for-optimizzaionn directly with the LLM, without additional solvers. As Song al. (2024) argues, LLMs offer both powerful in-context learning abilities and flexible natural-languaat interface capable of expressing a wide variety of problems. Many techniques embed the LLM in an evolutionary algorithm, using the LLM as a mutation or crossover operator (Chen et al., 2024a; Guo et al., 2024b; Meyerson et al., 2024; Lehman et al., 2023; Nasir et al., 2024; Liu et al., 2024a; Lange et al., 2024; Liu et al., 2024b; Romera-Paredes et al., 2023). In this setting, the LLM provides a diversity of samples while the evolutionary algorithm guides the search towards high-fitnes: regions. This strategy is a form of li-le optimization, in which two nested optimization problems (one nested within the other) solved in alternation. It is common for the outer loop to optimize the model's parameters, and for the inner loop to optimize the model's outputs (Chen et al., 2022b; Guo et al., 2024c). Ma et al. (2024) combine this approach with feedback from physical simulations in the inner optimization loop to enable LLMs to complete various scientific optimization tasks, such as constitutive law discovery and designing molecules with specific quantum mechanical properties. Although their use of an LLM in a bi-level optimization loop is similar to ours, they directly train their parameters using differentiable simulations whereas we do not assume access to the gradients of the ground truth rewards. Our work also explores various aspects of LLM training that allow the LLM to improve its optimization abilities despite not having access to ground -ruth rewards in the inner loop. Lastly, other approaches avoid gradient optimization altogether and focus purely on grompt-based optimization, demonstrating success on diverse tasks such as hill-climbing (Guo et al., 2024a), Newton's 's method (Nie et al., 2023), hyperparameter search (Zhang et al., 2023a), and prompt engineering (Khattab et al., 2024; Pryzant et al., 2023; Yang et al., 2024).
Controllable Text Generation Controllable text generation (CTG) is a special case of optimization. Rather than searching for sequences that maximize the objective function, the goal is to produce a sequence with particular attribute value (e.g. having certain number of words, a more positive sentiment than the input, or a particular biological motif). In some aspects, this may be an easier task- - the optimizer need not generate an optimal sequence that is likely outside the distribution of its training data. In others, this may also be more difficult. Targeting particular attribute values requires precise knowledge of the shape of the entire objective function, rather than only the neighborhood of the optima. Although LLM prompting is in itself considered a form of CTG (Radford et al., 2019; Brown et al., 2020), prompting alone tends to offer better control for
19

---

Generalists vs. Specialists: Evaluating LLMs Hiighly-onnstraneed Biophysical Sequence Optimization Tasks
more open-ended, higher level instructions (e.g. "write a story in the style of Shakespeare") than for fine-grained constraints (e.g. "rewrite this sentence using only 10 words") (Carlsson al., 2022).
One common approach is to use control codes, or unique strings pre-pended in front of a training example that indicates which attribute is represented in the example's 's target (Keskar et al., 2019; Padmakumar et al., 2023; Raffel et al., 2020; Madani et al., 2023). This approach is typically less generalizable to new attributes or instructions, due to the need to re-train the model. However, more recent work has shown that LLMs are capable of learning to use these control codes in-context (Zheng et al.. 2023c; Zhou et al., 2023), simplifying the process by which new attributes can be introduced. popular alternative technique inference-time guidance, which uses auxiliary tools (Pascual et al.. 2021) or models (Dathathri et al., 2020; Liu et al., 2021; Deng & Raffel, 2023; Dekoninck et al., 2024) to guide the LLM decoding process.
Existing Benchmarks in Biophysical Domains There are few notable efforts to improve the state of sequence optimization benchmarks for biophysical domains.
Small Molecules In the small molecule domain, GuacaMol (Brown et al., 2019) and the Therapeutics Data Commons (TDC) (Huang et al., 2021) include simulation-baset test functions for small molecule generation/opiimization benchmarking. As we discussed in the main text, simulation-basee test functions have significant barriers entry ranging from computational resource requirements to software engineering concerns such dependency management. If these simulations were in fact well-characterrzed, high-fidelity proxies for real molecule design objectives then these objections could be resolved, however at the time of writing it is difficult to determine 1) when a simulated task is "solved" and 2) what constraints are required to prevent ML methods from "hacking" the simulation and 3) to what degree simulation scores correspond at all to actual experimental feedback. Indeed, one could argue that real molecule design objectives were sufficiently well-understood to characterize via simulation then the most effective approach to ML--aumeented molecule design would be simply approximate and accelerate those simulations rather than directly model experimental feedback.
Large Molecules In the large molecule domain, ProteinGym (Notin et al., 2023) assembles a collection of protein datasets and model baselines but is primarily targeted at evaluating offline generalization with a fixed dataset. The models from this benchmark could be used as "deep fitness landscapes" (i.e. an empirical function approximation optimization benchmark), with the corresponding limitations we discussed in the main text. Our work is most closely related to the FLEXS benchmarking suite (Sinai et al., 2020).² To our knowledge, FLEXS is the most comprehensive attempt to date to assemble a robust suite of benchmarks for large molecule sequence optimization, with benchmarks for DNA, RNA, and protein sequences from an array of combinatorially complete database lookups, empirical function approximators. and physics simulators. The Rough Mt. Fuji model (see below) is the only closed-form test function in the FLEXS suite. It is additive and hence trivial to model and solve. Hence our contribution can be seen augmenting existing benchmark suites with test functions that are geometrically similar to real sequence optimization problems and also easy to install and cheap to evaluate.
Models of Sequence Fitness in Theoretical Biology
Geneticists have proposed theoretical models of biophysical sequence fitness and the geometry induced by random mutation and selection pressure, notably the mutational landscape model from Gillespie (2004), with more recent variants including the Rough Mt. Fuji model from Neidhart et al. (2014). These models are interesting objects of study, however those models assume mutational effects are either independent or additive, which disagrees with the correlated non-additive structure we observe empirically. These models also do not account for "fitness cliffs" (i.e. expression constraints that are highly sensitive to local perturbation and determine whether function is possible to observe experimentally). We implemented the Rough Mt. Fuji model as an additional test function and verified that genetic algorithm can easily optimize Ehrlich functions can be seen as a constrained, non-additive mutational fitness landscape, and may be interesting objects for further theoretical analysis.
Phttps://gtt thub. www.hhh..ooom.eoooooooooc.oo.eho sams nai htt.//innii/ss/
/FLEXS
20

---

Generalists vs. Specialists: Evaluating LLMs on Hfighly-Constrained Biophysical Sequence Optimization Tasks
A.2. Ehrlich Functions
A.2.1. MOTIVATION FOR EHRLICH FUNCTIONS
Rigorous benchmarking is an essential element of good practice in science and engineering, allowing developers to evaluate new ideas rapidly in low-stakes environment and thoroughly understand the strengths and weaknesses of their methods before applying them in costly, consequential settings.
While there has been a surge of investment into ML algorithms for applications like drug discovery, good benchmarks for those algorithms have proven elusive (Tripp et al., 2021; Stanton et al., 2022). Experimental feedback cycles in the life and physical sciences require expensive equipment, trained lab technicians, and can take months or even years, ML researchers require rapid feedback cycles, typically measured in minutes, necessitating proxy measures of success.
This need particularly acute when evaluating black-box sequence optimization algorithms, which must produce sequence of discrete states that optimizes ground truth function. Unlike typical ML benchmarks for supervised and unsupervised models, optimization algorithms cannot be evaluated with static dataset unless the search space is small enough to be exhaustively enumerated and annotated with the test function. Many researchers turn to simulation or empirical function approximation to provide test functions for larger, more realistic search spaces, however there is always a compromise between the highest possible fidelity (which may still be quite imperfect) and acceptable latency for rapid development.
We posit that well-designed synthetic (i.e. closed-form) test functions present many advantages as biomolecular design benchmarks, compared to simulations empirical function approximations. First, we note that synthetic test functions have been universally used to test continuous optimization algorithms for decades (Molga & Smutnicki, 2005). Second, we note that optimization algorithms are designed to solve any test function belonging to certain class. For example, gradient descent provably converges (under suitable assumptions) to the global optimum of any differentiable convex function. The key observation is that a test function need not correlate at all with downstream applications as long as there is shared structure (i.e. geometry). Thus the design principles we adopted to create Ehrlich functions were:
Low costs/barriers to entry -a a good benchmark should be inexpensive and easy to use.
Well-characterized solutions -It should be easy to tell if a benchmark is "solved".. Incremental progress towards better solutions should be reflected in the benchmark score.
Non-trivial difficulty- -a a good benchmark should be challenging enough to motivate and validate algorithmic improve- ments. should not be possible to solve with naive baselines on a tiny resource budget.
Similarity to real applications whhile benchmarks inevitably require some simplification, a good benchmark should retain key characteristics of the desired application in a stylized, abstracted sense, otherwise the benchmark will not be relevant to the research community.
A.2.2. LIMITATIONS OF EXISTING SEQUENCE OPTIMIZATION BENCHMARKS
With these criteria in mind, we categorize existing types of biophysical sequence optimization benchmarks and describe their limitations.
Database Lookups Database lookup test functions are constructed at substantial cost by exhaustively enumerating a search space and associating each element with measurement of some objective. sometimes requiring large interdisciplinary teams of experimentalists and computationalists (Barrera et al., 2016b; Wu et al., 2016; Ogden et al., 2019; Mason et al., 2021; Chinery et al., 2024). Unfortunately this approach necessarily restricts the search space, and the correctness of the database itself cannot completely verified without repeating the entire experiment.
Empirical Function Approximation Empirical function approximation benchmarks are related to database lookups since they start from an (incomplete) database of inputs and corresponding measurements. This type of test function returns an estimate from a statistical model trained to approximate the function that produced the available data (e.g. hidden Markov model sequence likelihoods, protein structure models, or "deep fitness landscapes") (Sarkisyan et al., 2016; Rao et al., 2019; Angermueller et al., 2020; Wang et al., 2022; Verkuil et al., 2022; Xu et al., 2022; Notin et al., 2023; Hie et al., 2024). Unfortunately empirical approximation is only reliable locally around points in the underlying dataset, and is difficult to characterize exactly over which region of the search space the estimates can be trusted. As a result, blindly optimizing
21

---

Generalists vs. Specialists: Evaluating LLMs on Biophysical Sequence Optimization Tasks
Moleculer interaction (selt bridge)
Epilope
Epllopa-apaellic binders
Shared mobil
Antibodly
Antibody
Antibody 2
HINI spike protein
H5N1 spike protein
(binding
FSA
F64
Angin (C eprape))
1 V
Glixmate pooogpoot
Target domain
Epllope
Target chain 1: GMVDGWWYYY.CCOOO.OOOO.COMMMMMM.. SEBSSTKKIDSGTTTNNNSINORRO
AKHMO YOVRETWDY CDRHB: ARHONYYYYSOMD
CDRH2 QIPIFGTTK CDRH2 GISPIFOSTA
loopi ADDFAGTV
Target chain AQDILEK,
VLATOLL
ADIFBNTA
Figure 6: (a) Arginine and glutamate are complementary amino acids because they have strong sall bridge interaction. (b- c) Antibodies that bind to a specific region of a target protein (the epitope) have many therapeutic and diagnostic uses. (d) Antibodies with different sequences can bind to the same epitope on two homologous proteins because they are structurally similar, which manifests as shared motifs in sequence space. Structures shown have RCSB codes 3gbn and 4fqi.
empirical function approximators often reveals an abundance of spurious optima that are easy to find but not reflective of the solutions we want for the actual problem (Tripp et al., 2021; Stanton et al., 2022; Gruver et al., 2024).
Physics-Based Simulations Simulations are a very popular style of benchmark, but current options all violate different requirements of a good benchmark. Most simulations slow to evaluate, many are difficult to install, some require expert knowledge to run correctly, and yet still in the end simulations can admit trivial solutions that score well but are not actually desirable. For example, docking models have been proposed as test functions (Cieplinski et al., 2023), but they do not have well- cell-characterized solutions and are easy to fit with deep networks (Graff et al., 2021). The primary appeal of simulations is a resemblance to real applications, however the resemblance can be superficial. AAG simulations (Schymkowitz et al., 2005; Chaudhury et al., 2010) do not have a low barrier to entry, and yet the correlation of AÁG with real objectives (e.g.. experimental binding affinity) generally modest or unproven (Kellogg et al., 201 1; Barlow et al., 2018; Hummer et al., 2023). Despite their difficulties, simulation benchmarks can be an important source of validation for mature methods for which we can justify the effort. However the limitations of simulations makes them especially unsuited for rapid method development, leading us to explore other alternatives.
Test Functions Closed-form functions have many appealing characteristics, including low cost, arbitrarily large search spaces, and amenability to analysis, however existing test functions for sequence optimization are so easy to solve that they are mostly used to catch major bugs. Simply put, designing a functioning protein is much, much harder than maximizing the count of beta sheet motifs (just one of many types of locally folded secondary structure elements in proteins) (Gligorijević al., 2021; Gruver al., 2024). The beta sheet test function highlights the main difficulty of defining closed-form benchmarks, namely not oversimplifying the problem to the point the benchmark becomes too detached from real problems.
Closed-Form
A.2.3. EHRLICH FUNCTION DEFINITION AND CONNECTION TO REAL BIOPHYSICAL SEQUENCE DESIGN
Here we introduce Ehrlich functions and explain which specific aspects of real biophysical sequence design problems are captured by this function class, using antibody affinity maturation as a running example.
Uniform random draws in sequence space are unlikely to satisfy constraints. One of the first challenges encountered in bbx biophysical sequence optimization is constraint on which sequences can be successfully measured. For example, chemical assays first require the reagents to be synthesized, and protein assays require the reagents be expressed by some expression system such as mammalian ovary cells. Popular algorithms like Bayesian optimization often assume the search space can be queried uniformly at random to learn the general shape of the function. Sequences of uniformly random amino acids generally do not fold into a well-defined structure and cannot be purified, meaning that we cannot measure anything about the objective function (e.g. binding affinity). Unfortunately constraints like protein expression cannot currently be characterized as a closed-fome constraint on the sequence. We simplify and abstract this feature of biophysical sequences with the notion of feasible set of sequences J with non-zero probability under a discrete Markov process (DMP) with transition matrix A E R x RT.
F = x E X A[x xz] > AA 2 2}.
(5)
where X' = 2 is the set of all sequences of length I, > 2 that can be encoded with states V, with | = v.
22

---

Generalists vs. Specialists: Evaluating LLMs on Highhy-Connstraiee Biophysical Sequence Optimization Tasks
Native profeln-protein interface
b
Mutation (noninterface)
form interne
compeiant conformtion, incresiing
Mutation (Interfcc)
d
targel, can term shonger leraccionn conformaioo,, reducting uffeniy
Double stabilier bending cenformation huo stronger irtercctoons the mazaiedyy tncresenng affivety
banding
Figure 7: Illustration of an epistatic second-order interaction.
Note that if sequences are drawn uniformly at random, then assuming at least one entry of A is zero (i.c. at least one state transition infeasible), we have
1.12
8
1
P $ F
02
t=1
1 1//2
=
2
where // denotes integer division. If we choose L large enough we will see uniform random draws fall outside F with high probability. See Appendix A.2.4 for further details procedure generate random ergodic transition matrices with infeasible transitions.
Non-additive, pesition-dependent sensitivity to perturbation. By construction, any sequence optimization problem can be written as minimizing the minimum edit distance to some set of optimal solutions X In the antibody engineering context X" is not a singleton but a set of solutions that all satisfy a notion of complementarity with the target antigen of interest (more specifically the target epitope). As a simple example, the epitope has an arginine residue, then placing a glutamate residue on the antibody creates the possibility of salt bridge (see Fig. 6). Furthermore, the formation of salt bridge in this example requires that we place the glutamate at specific positions the antibody sequence that are in contact with the epitope (i.e. on the paratope). One of the reasons there are many possible solutions to the antibody-antigen binding problem is the absolute position of an amino acid in sequence space can vary as long as the resulting structure is more or less the same (i.c. there are two or more structural homologs). The functional effect of changes to the antibody sequence are not only non-additive, but can exhibit dependee higher-order interactions, a phenomenon known as epistasis (Fig. 7).
We abstract these features of biophysical sequence optimization by specifying the objective as the satisfaction of collection of e spaced motifs (mm(1, s()),, ...,, (m(*), (c), s(E))), where ( y* and s()) 2 for some < L//c. Given sequence x, we can represent the degree to which x satisfies a particular (m))), s0) with q E [1, k] bits of precision as follows:
Σ Ho m, ) // (*) /9-
(g ml (0) s())
max E<L
(6)
The quantization parameter q allows us to control the sparsity of the objective signal (note that q must evenly divide k). Taking q = l corresponds to a dense signal which increments whenever one additional element of the motif satisfied. Taking g = corresponds to a sparse signal that only increments when the whole motif is satisfied. For example, if k
23

---

Generalists vs, Specialists: Evaluating LLMs on Highhy--onsttaince Biophysical Sequence Optimization Tasks
Sequence Lenghh
Moor Count
Moif Lengh
Olpectow Coantiation
12b
250
512
1,0
1.0
Siagl Ragnes
0.8
Single Ragie
0.8
0.5
0.6
0.
0.5
0.6
0.4
0.4
Smmpl
Simple
0.4
0.2
0.2
0.0
0.0
0.0
60
200
400
200
400
Function Evelustions
Function Evaluations (M)
Function Evalustions (M)
Function Evalustions
Figure 9: Here we show how the difficulty of the test problem can be controlled by varying Ehrlich function parameters, keeping the optimizer fixed to the robust GA baseline defined in See. A.6.3. Starting from a fixed set of reference parameters we vary each parameter individually. For this optimizer, the problem difficulty depends most strongly on the quantization parameter q. The x-axis is defined in millions (M) of Ehrlich function evaluations, demonstrating the difficulty of these Ehrlich functions, even for small number of short motifs.
and q = 2 then Eq. equation 6 can assume the values 0, 0.5, or 1. If k = 2 and q = then Eq. equation 6 can only assume the values 0 or 1.
The defining characteristic of epistasis is not merely non-additivity (which we can model via quantization and product-of ootl-oof-oooo-ooooooooeoooooooe- motif-checks parameterization). but also non-monotoniciyy as a function of motif satisfaction. In simple terms, simply mean that a beneficial mutation in one sequence context be deleterious in another. We model non-monotonic mutational effects through the introduction of a response function g: [o, 1] + R. In particular we propose a cubic function g(h)- = ahh ah? + h, where a E R2 is the epistasis factor (see Fig. 8). When a = 0, g(h) = h, corresponding to a linear response. When a = 4, g(0) = (0.55, which means satisfying half of the motif scores the same as satisfying none of it. While a > 4 can be used, one must be careful guarantee that the optimal value still holds, i.e. f = 1, and f will no longer be strictly non-negative. Note this reponse function is symmetric with respect to all motif elements, as there is no notion of some motif elements being more "robust" to epistasis than others. While this modeling decision does not entirely agree with empirical evidence, it greatly simplifies the implementation, and captures the key non-monotonic behavior we desire. In the experiments for for all test functions, as we found the resulting optimization problems already sufficiently difficult
1.0
4-0
0.8
Resporue
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Motif Satisfaction
Figure 8: Motif satisfaction response
function g(h) = ah3 ah2 + h.
this paper we took to induce interesting differences in the optimizers, leaving investigation of the effect of taking for future work.
We are now ready to define an Ehrlichfuunction f yL + (-o0, 1J. which quantifies with precision the degree to which simultaneously satifies all (m (m())s)) s())
) if x is feasible, and is negative infinity otherwise.
Ii g o (x s
if F else
f(x)
Note that we must take some care to ensure that 1) the spaced motifs are jointly satisfuble (i.e. are not mutually exclusive) and 2) at least one feasible solution under the DMP constraint in Eq. equation 5 attains the global optimal value of 1. See Appendix A.2.5 for details.
One major advantage of procedurally generating specific instances of Ehrlich functions is we can generate as many distinct instances of this test problem as we like. In fact it creates the possibility of "train" functions for algorithm development and hyperparameter tuning and "test" functions for evaluation simply by varying the random seed. These functions can also be defined with arbitrary levels of difficulty, as shown in Fig. 9. However, defining a random instance that is nevertheless provably solvable takes some care in the problem setup, which we now explain.
24

---

Generalists vs. Specialists: Evaluating LLMs Hlihhy--Cnsstrance Biophysical Sequence Optimization Tasks
A.2.4. CONSTRUCTING THE TRANSITION MATRIX
Here we describe algorithm procedurally generate random ergodic transition matrices with infeasible transitions. A finite Markov chain is ergodic if it is aperiodic and irreducible (since every irreducible finite Markov chain is positive recurrent). Irreuuciiility means every state be reached with non-zero probability from every other state by some sequence of transitions with non-zero probability. We will ensure aperiodicity and irreuucibility by requiring the zero entries of A to have a banded structure. For intuition, consider the transition matrix
0.4
0.3
0.3
0.3
0.4
0.3
0
0
0.3
0.4
0.3
0.3
0.3
0.4
Recalling that is the number of states, we can see that every state a communicates with every other state x' by the sequence + (x + 1) mod v + - (x' - 1) mod v - x'. We also see that the chain aperiodic since every state æ has a non-zero chance of being repeated.
To make things little more interesting will shuffle (i.e. permute) the rows of banded structured matrix (with bands that wrap around), but ensure that the diagonal entries are still non-zero. Note that permuting the bands does not break irreducibility because valid paths between states can be found by applying the same permutation action on valid paths from the unpermuted matrix. We will also choose the non-zero values randomly, using the shuffled banded matrix only as a binary mask as follows:
Thanded matrix)
1
0
1
1
0
1
1
1
0
1
shutlie
1
1
0 1 1 1 1
1
0
0
1
1
1
O
1 1
0
diag=1
1
B.
Now we draw the transition matrix starting with random matrix with IID random normal entries, softmaxing with temperature 0 to make the rows to applying the mask B, and renormalizing the rows by dividing by the sum of the columns after masking.
(randn matrix)
+1.41
+1.67
-1.52
+0.63
0.35
+0.45
+0.86
-0.49
+1.42
-1.31
-0.31
+1.43
0.02
+1.55
-0.26
+1.13
T0.36
0.46
0.02
0.16
0.13
0.30
0.45
0.12
softmax
0.44
0.03
0.08
0.45
0.10
0.49
0.08
0.33
0.36
0.02
0.167
0.13
0.30
0.45
OB
0.44
0.03
0.08
0.45
0.49
0.08
0.33
0.66
0
0.04
0.30
0.15
0.34
0.51
0
A.
0.44
0.03
0.08
0.45
norm
0.55
0.09
0.36
25

---

Generalists vs. Specialists: Evaluating LLMs HlihhyyConsttrinee Biophysical Sequence Optimization Tasks
We also verify that A is ergodic numerically by checking the Perron-Frobenius condition,
(A )5j > O, Vi
(8)
where m = - (v-1)² +1, 1, A = =A, and A = AAA-1 for all b > 1. In our example, if u = 4 then m = -0 and we verify on a computer that
0.33
0.23
0.17
0.27
o
0.33
0.23
0.17
0.27
0.33
0.23
0.17
0.27
0.33
0.23
0.17
0.27
A.2.5. CONSTRUCTING JOINTLY SATISFIABLE SPACED MOTIFS
Here describe how to procedurally generate spaced motifs of length & such that the existence of optimal solution x" with length L with non-zero probability under transition matrix A generated by the procedure in Appendix A.2.4 can be verified by construction. If we simply sampled motifs completely at random from y* we cannot be sure that solution attaining a global optimal value of is actually feasible under the DMP constraint.
First we require that L 2 c k. Next to define the motifs, we draw a single sequence length c x from the DMP defined by A (the first element can be chosen arbitrarily). Then we chunk the sequence into e segments of length which defines the motif elements m() This ensures that any motif elements immediately next each other are feasible, and ensures that one motif can transition to the next if they placed side by side.
Next we draw random offset vectors (() The intuition here is we want to ensure that an optimal solution can be constructed by placing the spaced motifs end-to-end. If we fix e x k positions to satisfy the motifs, there are L - c X k "slack" positions that we evenly distribute (in expectation) between the spaces between the elements of each motif. We set the first element of every spacing vector s(") to 0, then set the remaining elements the partial sums of random draw from discrete simplex as follows:
(() ~u -u(( E RK-1 Σ w; = =1}) (0) j+1 =s +1+
(9)
L00 x (L-c k)//cl-
(10)
Finally, recall that in Appendix A.2.4 we ensured that self-transitions x - x always have non-zero probability. This fact allows us to construct a feasible solution that attains the optimal value by filling in the spaces in the motifs with the previous motif elements.
As concrete example, suppose L 8, c 2, and (hence Smax
3) and we draw the following set of spaced motifs:
m (1)
[o
1
2]
(11)
[
m (2)
[((1)
0
3
(22)
=
0
3
(12)
(13)
An optimal solution can then be constructed as follows:
Note that this solution only used to verify that the problem be solved. In practice solutions found by optimizers like genetic algorithm will look different. Additionally if L >> A then the spaced motifs can often be feasibly interleaved without clashes.
A.2.6. DEFINING THE INITIAL SOLUTION
Optimizer performance is generally quite sensitive to the choice of initial solution. our experiments we fixed the initial solution to a single sequence of length T. drawn from the DMP.
26

---

Generalists vs. Specialists: Evaluating LLMs Hiihhy--onsstrance Biophysical Sequence Optimization Tasks
A.2.7. EHRLICH BENCHMARK RANK CORRELATION WITH REAL-WORLD DATA BENCHMARKS
To validate the real-world applicability of Ehrlich functions an optimizer benchmark, we compared the performance of 64 variants of our genetic algorithm Ehrlich functions versus three well-established lookup- based biological test functions: TFBind8 (Barrera et al., 2016a), TrpB (Papkou et al., 2023), and DhfR (Johnston et al., 2024). For each benchmark, evaluate the median cumulative regret (estimated from 8 trials) of 64 variants of our GA with different hyperparameter settings, Each hyperparameter variant (pm, Pr. (t) was an element of the Cartesian product {0.0625, 0. 0.125, 0.25, 0.5} x {0.0625, 0.125, 0.25, 0.5) x (0.0625, 0.125, 0.25, 0.5}. We then computed rank correlations between algorithm performance on these biological benchmarks vs. comparable Ehrlich functions. As shown in Table I, the strong rank correlations are evidence supporting the claim that Ehrlich functions effectively capture the structure of real biological sequence optimization problems.
DhfR
TFBind8
TrpB
Ehr(4, 4)-2-2-2
0.75
0.75
0.61
Ehr(20, 8)-2-2-2
0.86
0.89
0.73
Table I: Spearman correlation coefficients of median regret achieved by our genetic algorithm variants on Ehrlich functions versus lookup-based biological test functions.
27

---

Generalists vs. Specialists: Evaluating LLMs Hiihhy--onsstranee Biophysical Sequence Optimization Tasks
A.3. Proof of Strong Interpolation Criteria for the Policy Objective
The policy objective defined
[( = (To || m²) + XDRI. (Frer To);
(14)
where Ho is the learned policy, n² is the optimal target policy (Boltzmann distribution based rewards), "ee is the reference prior policy, and À > 0 is the interpolation hyperparameter. We assume that Te, TT,, and Tref are probability distributions over the same action space } (conditioned on a state/prompt 2, which we omit for brevity in the distributions). We also assume that the family of policies parameterized by is sufficiently flexible to represent both x and Tref-
The Strong Interpolation Criteria (SIC) requires:
1. limx arg ming, L())
2. limx-xo arg ming, [(0) Tref
3. For À E (0, xo), the optimal (denoted )T²) interpolates between x² and Tref-
A.3.1. PROOF OF SIC CONDITION 1 (A - 0+)
As Å + 0+, the objective function in Eq. (14) becomes:
lim L(() lim (DKL(TO || m²) ADKL (Frer || mo))
A x
x-50+
(KI((T |l π 0- (TIe || To)
DKL(TO || TT))
The Kullback-Leibler divergence (To || x)) is non-negative, i.e., DKL(TO || m²) 2 0. It achieves its minimum value of O if and only if To(y)= = n" (y) for almost all E ). Therefore,
lim arg min L(0) n²" x
xy
This satisfies the first condition of SIC.
A.3.2. PROOF OF SIC CONDITION 2 (A + 00)
As + oo, we analyze the objective function L(0) = DKL (T*) + AKL((Te TT).
Consider the case where Te Nyef- this scenario. DKL (Tee Te) DKL(Tee Tret) 0. The objective function evaluates to:
((0) = DKL π²) + À 0 DKL (Trel TT).
This value is a finite constant (assuming Tref and x² have overlapping support such that the KL divergence is well-defined and finite).
Now, consider any policy ne * &eef such that DKL(TeC TT)) > 0. As + co, the term (DrL || ля) will tend to co, because DKL (Tee πo) a positive constant for this πe. Thus, for To # Wref (where DKL (KL((ee (Tret TT)) > 0):
lim C(0) DKL( || [1')== X
lim ADRL(Teet Te) = 0c. X
To minimize L(0) as X 0o, the policy Fy must be chosen such that the term growing with minimized, which means (KL(Tee TO) must be This occurs and only if Ty(Y) Tre((y) for almost all y V. Therefore,
lim arg min L()) = Trof X
This satisfies the second condition of SIC.
28

---

Generalists vs. Specialists: Evaluating LLMs on Hlighly-Constrannee Biophysical Sequence Optimization Tasks
A.3.3. PROOF OF SIC CONDITION 3 (INTERPOLATIIO FOR E (0)
(A) For any finite, positive X, the objective function ((Ta)) == (Te || m²) + XDrr. (Tref || mo) is minimized by policy "o We established that x² (y/ær) xc exp(3 r(z,)) 3)) is a Boltzmann distribution, and Tet is the reference policy.
The objective (=o)) is a sum of two non-negative divergence terms. The first term, (KL(TO || π²), pulls me towards x².
The second term, (Tie || mo), pulls vp towards Fref (in a mass-covering sense).
If operate directly in the space of probability distributions (assuming our model family Il = { J is rich enough to contain the solutions). L(xz) is strictly convex function of the distribution π E Ie for any À 2 0 (as both KL terms are convex in x, and the first is strictly convex if x² is fixed, and the second is convex in RT. Thus, for any À 2 0, there exists a unique minimizing distribution π: (AA) in a suitable convex space of distributions.
The first-order optimality condition for the unconstrained problem (in the space of distributions, subject to E, a(y) = 1) for ((x) ) found by setting the functional derivative with respect to rty) to zero:
8((()) = (1og TVV (t) (T)
+1) + A ( Tref (v)
+ μ =
(y)
where j is the Lagrange multiplier for the normalization constraint. This implies that the optimal distribution π (z) satisfies:
(y) (()
log (2) (y) log mT(())
+ +1+
x Trer(u) 7()(s))
a (1) x T (V) exp
This equation implicitly defines the unique optimal distribution T (v). The solution x(A) to the implicit equation (+) varies continuously with A. This is because the objective function [(T) changes continuously with X, and the minimizer of a strictly convex function typically depends continuously such parameters. Thus, the optimal policy To (A) (which is the T
in our model family that best approximates n (A)) smoothly interpolates from x² to Teef as À varies from 0f to co. This satisfies the third condition of SIC.
A.4. Derivations
Although DPO has recently become popular preference learning method due to its relative simplicity and competitive results, its offline contrastive objective suffers from a number of drawbacks, Firstly, since DPO optimizes for an off-policy objective, a improvements in offline metrics (Rafailov et al., 2024a; Chen et al., 2024b). DPO models also fail at ranking text according to human preferences (Chen et al., 2024b) and tend to decrease the probability mass assigned to preferred outputs (Pal et al., 2024; Rafailov al., 2024b; Feng al., 2024; Pang al., 2024). As training continues, DPO generations also increase in length, even if the quality of the generations does not necessarily improve (Singhal et al., 2024). Additionally, when the reference model already performs well on a particular subset of the input domain, DPO cannot achieve the optimal policy without deteriorating performance on that subset (Hu et al., 2024). Lastly, DPO does not make use of absolute reward values - instead, it simply assumes that r(c, Mww > r(2,90 31) for all (I, Mee 31) in the training dataset, but does not use information about how much better yo is than yt-
model rapidly over-optimizes, resulting in generations that decline in quality despite continued
RLHF, on the other hand, involves steep technical complexity and frequently exhibits training instabilities (Zheng et al., 2023a; Wang et al., 2024; Casper et al., 2023). Hu et al. (2024); Korbak et al. (2022b) additionally show that RLHF's objective interpolates between ARet and a degenerate distribution x that places all probability mass single reward- maximizing sequence, thereby promoting generator collapse. Indeed, much past work (Kirk et al., 2024; Zhou et al., 2024; Padmakumar & He, 2024) has illustrated the low diversity of RLHF- text.
Derivation of the MargE Objective To derive a training objective that fulfills SIC, we follow Hu et al. (2024) and propose an objective that takes the following general form:
arg min KL [1= + [ [t)
(15)
#
where xt is the target reward distribution and do is the -orth-nnormliized version of no.
29

---

Generalists vs. Specialists: Evaluating LLMs Hlihhy--onsstrance Biophysical Sequence Optimization Tasks
Since past work has indicated that preference-tuned models often spuriously favor longer generations, follow Malladi (2024); Meng al. (2024) by instead using length-normalized likelihoods. Additionally, it is standard in past literature (Ziegler et al., 2019; Korbak et al., 2022a) to model x² as a Boltzmann function of the reward. That is,
x (y/ x))
(x) exp(r(x,)))
where (x) is the partition function. Although this partition function is guaranteed converge due the observation that LLMs often place non-zero probability mass on infinite-length sequences (Du et al., 2023; Welleck et al., 2020), we make the assumption for now that this effect is negligible. This results in a formulation of the optimal policy that is Bradley-Terry with respect to the rewards. L.e.,
log x. (yn (x) - log x (12l2> = r(x, 91) r(x, 322
Then, the first term of Equation 15 can be expanded as:
(1)laa)) ा (1/(x)
arg min [+tll=//=)) = arg min
log
a youRa (-lw)
Since we cannot directly sample from neo we approximate it via an expectation over the un-normalized policy:
log <o(y)o) r(x, 5) + log (xx)
arg min
DrDDa g~ar (1(z)
lel
log No (vlx) r(z,,) v)
arg min
zo gramt
Rather than re-sampling from Ty after every step of training, we approximate this step using importance sampling:
To(y)æ) T2)((y/2z) [2)((
log no (yyla (r)
no [T) x
r(x, v)
3/
DeD m
To (2(3(x))) [
log no (v(x)) r(ar r(,,1)
(16)
w>D y~
For the second term of Equation 15, we remove terms not depending on 0, resulting in the standard token-averaged cross-entropy loss:
log गe (ylx) ar)
min 0
KL TRee ( ) = min
(17)
De you
|y]
Plugging Equations 16 and 17 back into 15, we obtain
(log (vll)) r(x, v))
log no (y(x))
MMargt To. TRefi Dr.)) =
B g (ar)
TRee (yl (Re((y(2z)
lyl
l
Since importance sampling often leads to high variance of the gradient estimates in practice (Owen, 2013), we instead use the self- normalizd version of this objective (Eq. 30).
Lemma A.I. Given Bradley-Terry model (yy[z) ziz exp(r(x, y)) with partition function 7 if reward function
r(a, y) = f(y) f()) for some real-valued scoring function f(x) V + R, then Ttylz) = (g]22) for any pair x, E dfm
30

---

Generalists vs. Specialists: Evaluating LLMs Hiihhy--onsstrinee Biophysical Sequence Optimization Tasks
Proof. Since ((x) is the normalizing constant for (ylo:), we can write
Z(x)
Σ
exp(r(z,))))
(18)
y'EV'
Σ exp(f(1/) - f()))
(19)
y'EV"
Σ exp(J(v/)).
(20)
exp(f(x)) V'EV"
It follows that
exp(S(x))
Σ Ever exp(f(1/)) exp(r(x,1))
(21)
exp(f(x) + 11y) f(x))
(22)
Eyeri exp(s(t'))
exp(f(v))
(23)
Ever. exp(f(1/))
exp(S(y) [(z) f())
(24)
Evev- oxp(f(y))
(25)
Even exp(f(y))
exp(r(z,
(26)
((v)
TT)22)
(27)
31

---

Generalists vs. Specialists: Evaluating LLMs Hlighly-Constranned Biophysical Sequence Optimization Tasks
A.5. Prompt
We use this prompt only in our initial prompting experiment with ol and Gemini Flash (see Section 6.1), with <TRANSITION MATRIX HERE> replaced by the transition matrix corresponding to f2-
You are given a challenging discrete optimization problem to solve. The problem consists generating a discrete integer sequence (with vocabulary consisting integers from to 31, inclusive) that lies in the support of a discrete Markov process (DMP) and contains specific spaced motifs, The transition matrix for the DMP is as follows:
<TRANSITION MATRIX HERE>
The motifs are 3, 16, 15, 11], [24. 3. 16, 15], [11, 14, 8, 10), [22, 27, 7,
7, 20] and the respective spacings for these motifs are [0, 2, 4, 5), [0, 3,5, 6]. 10, 1, 4, 6]. and [0, 2, 4, 15]. Your solution will be scored based both whether it is feasible (i.e. lies in the support of the DMP) and how many motifs it fulfills. That is, if the motifs and their then we score whether sequence æ fulfills motif i with 32 in this problem). If the sequence x is feasible, then the total score is I M(x, m ,,))))))))))))))))))))))))) ()) Otherwise, the score is O.
spacings are denoted as ((m(1), (")),,.., ,,,
))))). (m(()))
h(2, (() ) m
1 [ (4 = =m) 1)/4 where the length of
Here are some example sequences and their respective scores:
21 = 31, 2, 4, 15, 7, 14, 15, 32, 11, 29, 25, 1, 15, 11, 19, 24, 22, 5, 17 27, 14, 28 16, 15, 11, 14, 4, 16, 10) Score of : (1/4) (1/4) (1/2) (1/2) 1/64
T2 31, 15, 14, 15, 12,31, 10, 25, 1, 15, 11, 19, 24 5,, 34, 28, 16, 15, 11, 14, 10, 15]
Score of 22: 0
I3 = [3 16, 15, 11, 24, 2424,2244 24, 15, 11, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22,22.
22]
Score of z3: (1/2) (1/4) (1/4) (1/4) = 1/128
I1 33,3, 16, 16, 15, 11, 24, 24, 24, 24, 15, 11, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22,22, 22]
Score of 24: * (1/4) * (1/4) (1/4) = 1/64
is often a good idea to start with a random sequence, perturb it bit, score it and stay with this perturbed version scores better. You can repeat this process as many times as you want until the score converges the maximum value. It is extremely important check your progress be good idea backtrack what you have done (make sure mark important steps with special markers that you know where to backtrack to) and resume from there
in while. If you haven't ttt made good progress, it would
Since there could be many isolated solutions, it would he a good idea to try this process from a few different random initial sequences. Make sure to score the solution of each these random initial sequences and revisit them at the end to find the very best sequence.
Everyone's s life depends finding the best sequence, as this sequence would be key finding medicine for the future pandemic.
It is important for you to solve this problem directly yourself by thinking about it carefully and iteratively, as you are not allowed to use the computer, due to safety measures. Please directly output a sequence and not code, as you do not have access to an environment to execute the code in.
32

---

Generalists vs. Specialists: Evaluating LLMs Highly- Biophysical Sequence Optimization Tasks
A.6. Algorithms
Algorithm DATASETFORMATTIN(S)) An algorithm that adapts PropEn (Tagasovska et al., 2024) to create pairs or triples of data for LLM training. We use kn = 30, Ar = 0.25, and the fractional Hamming distance as d.
Input: Dataset S {(x, y)} of particles x and scores yr 2,, number of nearest neighbors to find; distance function d; threshold Ar; dataset type type ("binary") "triple") if type- "binary" then
2y (Z))2z)
D
( f(x)))((.))
else
( (zi,
D
end
Output: D
Algorithm 3 [TERATIVERRETEEMENT (mp; S). v/ represents with temperature / scaling. We use 200, 10. n, = 10, and T" = [0.6, 0.8, 1.0, 1.2, 1.4, 1.6). {(x, y)) of sequences x and scores y; n, number of seed examples; n, rounds of iteration per example; n,, outputs per iteration; set of temperatures to sample with.
Input: Pretrained L.LM my: dataset
S' (5<)
X'
(x. E TopK(S,
> Obtain n, seed examples from the top training examples score.
for
do
Xn f x
i - 0
for m do
X (T
S' U
i6i+1
end for t E T do
i i
for
< n. do
X. {xj ( x2
Sample n., sequences using temperature
S' - S U.
X:41 arg m na(x') xx)
Select the highest-likelihood sample the input for the next iteration.
i i i
end
end
end
S' DEDUPLICATE(S")
Output: S"
Algorithm 4 FILTER(X j). An algorithm for filtering a dataset X' of sequences down only sequences.
Input: LLM To that generated the sequences; dataset X' {(x y)} of sequences x and scores y; likelihood threshold Pminn: maximum proportion of infeasible sequences Pmax-infeas: final output size k.
X + {(xx y), Te (x)) (x, 8) X}
>> Compute likelihoods.
X' + {(x, y) no
Pmin: ((x, v). ma (x)) E )
Xiessible t- {xx, v)
0o. x) E X)
Kintezsible (x. 1)
(x,)) X}
Titasible Xtesiitel
Yemaa-innfcu. f- Tifesstble Pruc-infras
Pras-intess
X Xisasible SAMPLE Xitessibtee Plmax-intess)
Downsample infeasible examples. > Subsample j examples.
X + SAMPLE(2', min(j, (2()))
Output: X
33

---

Generalists vs. Specialists: Evaluating LLMs Hfighyy-onsstrannce Biophysical Sequence Optimization Tasks
A. DATASET FORMATTING
To format our data, we adapt PropEn (Tagasovska et al., 2024), a technique for matching pairs of examples implicitly guide the model towards generating sequences that are close by the input but still improve a particular property. Since PropEn originally designed for pairs of data (z, y), we also adapt PropEn to create triples of data (z, Yuu 31) for DPO training. In short, given a dataset D = { 84) b of input sequences and their scores yi, PropEn creates the following paired dataset:
PrropEn = {Tx 23) (Ia, Mi), (z), E D, D,d(z,, S Az. f(z)) f(z,) E 1yl
(28)
for thresholds Ar and Ag and distance function d. In all our experiments, we use the Hamming distance as d and modify the constraint f(x') - f(x) E (0, An) to f(x') > f(x) since we observed that this looser constraint was more effective in our experiments. We use ProopEn for both LLOME-SFF SFT and LLOME-MARGE.
To additionally adapt the PropEn dataset creation process for preference tuning (e.g. DPO), we create preference triples
using the following constraints:
D ProopEnTTiipe Triples
(3i, wxk)
A.
(29)
In preference tuning terms, the H; is the prompt or input sequence, and I.j and Fy are yo and yt, respectively. We use Prippein/rripls in LLOME-DPO, We formalize this algorithm in Alg. 2.
For all training algorithms, we format the input as "<inc> [3, 1, to indicate to the model that should edit the sequence to increase the score. Outputs are formatted similarly, but without the control code.
5] where "<inc>" is control code meant
A.6.2. ITERATIVE REFINEMENT
Our iterative refinement is formalized in Alg. 3. Loosely, we select the best n, training examples from the last round of the LLOME outer loop, and provide them as seed inputs to the L.LM to refine. For each seed input, we use 10 rounds of iterative generation where the best (highest-likelhhood) generation from the previous round is provided as input to the next round. We repeat this process with both greedy decoding and sampling at various temperatures. In the case of greedy decoding, only one generation obtained per iteration. When we sample, we sample n, = 10 outputs at once.
Since LLM outputs tend become less diverse with more rounds of training, we also implement automatic temperature adjustments after each round of LLOME. The default temperature range is T [0.6, 0.8, 1.0, 1.2, 1.4, 1.6). but if the average Hamming distance of generations from the last LLOME round was < 0.075, then instead use T + 0.6. For average Hamming distance between 0.075 and 0.1, we use T + 0.4. For averages between 0.1 and 0.1, we use T + 0.2.
A.6.3. GENETIC ALGORITHM
In Algorithms 5, 6, and 7, we provide pseudo-code for our genetic algorithm baseline, which we implement pure PyTorch (Paszke et al., 2019), using the torch optioniii API.
The GA baseline has only four hyperparameters, the total number of particles n, the survival quantile E (2/n, 1), the mutation probability Pm. and the recombination probability pr- Generally speaking for best performance one should use the largest possible, and tune (which determines the greediness of the optimizer), Pm, and p,.
Our genetic algorithm uses mutation probability pm = 0.005, n 1000 particles per iteration, survival quantile a = 0.1, and recombination probability p.- = 0.0882.
34

---

Generalists vs. Specialists: Evaluating LLMs Hlihhyy-onsstrince Biophysical Sequence Optimization Tasks
Algorithm 5 Genetic algorithm pseudo-code
Input: initial solution x", ft, mutation probability Pm, recombination probability pr, survival quantile a, # particles n
Ppp f motate( (x* (aaatee((")))x)""""" Pm):
for f I....,TTo do
< popp)
max 0/ f then x - argmax i
f max
end
T f -uaaniieeeyleeeeeee a a) Moo + fx Kpop f(x) } n' f'tn- Nool
Xpop Meop U Weecombine((roppp Pr: n²)
Xpop mutatet Pm, 1)
end
Returns: Estimated maximizer x" l
Algorithm 6 mutate function
Input: initial set X, mutation probability Pm, number of mutants
X' = 0
for x E X' do
for i = 1, I, n do
mask iiee)) < Pm
sub r0n 1, (x))))) x' thereemask,,ub.
X' = X' (x')
end
end
Returns: X'
Algorithm 7 recombing ne function
Input: initial set X, recombine probability number of recombinations n.
X' =
p(1) (raw...replaccemenn((.) n)
((2) draw.w.repplacement
for i 1,..., do
(!) = (())
x2) 2!
mask rind_like(x!!! < pr
x' where x (1), x(2)
X' = X'U
end
Returns: X'
35

---

Generalists vs. Specialists: Evaluating LLMs Hlihhyy-Cnsstranee Biophysical Sequence Optimization Tasks
A.7. LLM Training Details
We train every model for epoch with PyTorch DDP on A100 GPUs, using training loops implemented with the Huggingface datasets, trans Formers, and trl libraries. We conducted hyperparameter searches for each LLM training method, using the validation loss from the dataset of the first iteration of LLOME select the best hyperparameters, We also check whether the generated outputs are parsable and conform to the correct format (i.e., list of the correct length with values in the correct range). If the hyperparameter set-up with the lowest validation loss does not output sequences with the correct format > 90% of the time, then we select the set-up with the next best validation loss that meets these constraints. Notably, these format checks were the most important for DPO. Many DPO-trained models achieve low validation loss despite generating sequences with incorrect format. We use the best hyperparameters tuned from the validation dataset created during the first iteration of LLOME but do repeat hyperparameter tuning future iterations. All search ranges and final hyperparameter values (in bold) are listed below.
SFT
We train the SFT models with the Adam W optimizer, with B1 = 0.9, B2 = 0.999, À = 0.01, and e = x 10-8, We also search the following hyperparameter ranges:
Learning rate E {1 x 10-?,1 x 10 10-6, 11xx 10-5}
Batch size E {16, 32, 64, }
DPO We train the DPO models with the RMSprop optimizer, with a = 0.99, À 0, and E = 1 x 10-8, Due to computational constraints, we train with 6. We also search the following hyperparameter ranges:
Learning rate E {1 x 10-7, 1, 10-6}
Batch size E {64, 128}
3 E (0.1, 0.2, 0.4, 0.8}
MargE We train the MargE models with the AdamW optimizer, with 31 = 0.9, 82 = 0.999, A = 0.01, and = 1 x 10-8, We also search the following hyperparameter ranges:
Learning rate E {1 10-7, x 10 10-8)
Batch size E (64, 128)
E {0.2, 0.4, 0.8, 1.0, 10.0)
REINFORCE We trained REINFORCE with the same best hyperparameters as MargE.
Self-Normalization For both MargE and REINFORCE, applied self-normalization to the importance weights. That is, if B(x, y) is the batch of examples that particular example (æ, v) belongs to, then the self-normalized MargE and REINFORCE objectives are as follows:
w(x, v) (lg (log TT(U)=)) (oggT(ulz) r(xx,)) log (T(()))
Maugt (To TRet' D2))
E anD y
(30)
[y]
w(x, v) (-r(x, y) log T(((=)) log To (yla²)
ČREINOORCE TRofi D2))
E xo youter((-lw))
(31)
where
w(x, v) = To(y(z)/ (y)())
(32)
w(x, y)
w y) (a' (a' (BB().)))) w(x'v1)
(33)
Updating Ret During Iterative Training In Algorithm 1, we iteratively train the LLM using outputs generated by the last round of iterative refinement. In this iterative training process, at iteration of the outer loop, we always update नee for DPO, MargE, and REINFORCE, such that Reea == no.-
36

---

Generalists vs. Specialists: Evaluating LLMs Ilighly-Constrained Biophysical Sequence Optimization Tasks
A.8. LaMBO-2 Model and Training Details
We built upon the implementations of LaMBO-2 in cortex and poli-baselines. We use a LaMBO-2 architecture with an encoder that is shared between different output modules: one generative discrete diffusion head, one discriminative head predicting whether sequence satisfies the constraints of the problem, and one discriminative head predicting the reward of sequence.
Each of these modules is made up of ID CNN residual blocks, with layer normalization and sinusoidal position embeddings (except for the generative head, which is a linear layer directly from the shared embeddings to output logits over the vocabulary). The encoder is composed of residual blocks and the two discriminative heads are composed of residual block each, all with kernel width Each residual block has 128 channel dimensions and 128 embed dimensions. We applied diffusion noise and guidance the encoder embedding, and for each of the discriminative tasks, we trained ensemble of 8 independently randomly initialized heads.
A. 1. TRAINING SET REBALANCING
We employ a data rebalancing strategy during the training of LaMBO-2 to enhance continual learning. The objective is to enable the model to adapt effectively to newly acquired data while preserving knowledge learned from historical data. This achieved by sampling mechanism that effectively gives exponentially greater importance to more recent data during training.
Partitioning Scheme: Training data is partitioned based the iteration in which it was collected. This partitioning follows geometric scheme:
Partition 0 contains data from the most recent iteration.
Partition contains data from the two iterations immediately preceding those in Partition 0.
Partition contains data from the four iterations immediately preceding those in Partition 1.
This pattern continues, with Partition & generally containing data from 2 distinct iterations that are older than the data in Partition k - 1.
Formally, let T be the index of the most recent data collection iteration. Data collected during iteration (where < 7) is assigned to partition p(i) according to:
0 if = T,
p(i)
k
if [T 2 2,T - - +1] for t: 2 1.
Each partition l: (for 2 1) thus groups data from 2* consecutive iterations.
Round-Robin Partition Rebalancing Sampling Mechanism: During training, we construct minibatches by sampling one datapoint from each active partition. Since more recent data is grouped into smaller partitions (e.g., Partition contains data from only one iteration, while Partition & contains data from 2* iterations), datapoints from recent partitions have higher probability of being selected. This sampling strategy naturally creates the desired exponential weighting, prioritizing recent information.
To prepare data for training:
Datapoints within each individual partition are randomly shuffled.
The order in which the partitions themselves are processed for sampling also randomized. This shuffled order defines cycle for drawing samples.
To form training minibatch of predefined size:
We iterate through the partitions according to their current shuffled order.
37

---

Generalists vs. Specialists: Evaluating LLMs Hiihhy--onsstrince Biophysical Sequence Optimization Tasks
From each partition encountered in this eycle, one (randomly selected) datapoint is drawn and added to the current minibatch.
This process continues, drawing one datapoint from each subsequent partition in the shuffled order, until the minibatch reaches its target size.
If all partitions have been sampled from (i.e., one cycle through the shuffled partition order is complete) and the minibatch is not yet full, the order of partitions is re-shuffled, and sampling continues from the newly ordered partitions until the minibatch is filled.
When any partition is exhausted, the examples in that partition are shuffled and sampling continues.
A.8.2. CONTROLLING DIVERSITY THROUGH CANDIDATE SELECTION
We also introduce new hyperparameter to LaMBO-2, the "farthest first traversal (FFT) expansion factor", which provides a lever control the diversity of solutions used starting points for guided diffusion. For applications with similar number of ground truth evaluations n per evaluation iteration compared to the total number of iterations (e.g. a 24-well plate of samples for each of 10 iterations), the best FFT expansion factor is usually greater than one. In this setting, we retrieve the top an solutions seen over the optimizer's history and use farthest first traversal to select the subset of solutions which capture as much sequence diversity as possible among the original on solutions. The FFT procedure iteratively adds a solution to the selected subset it is maximally distant from all the currently selected solutions (See Algorithm 9 for pseudo-code). In this small n setting, a prevents generator collapse and trades exploitation for more exploration.
In contrast, when is very large compared to the number of iterations (such in the experiments for this paper, with 2000 and only 10 iterations), it is best to set < 1. In this case, select the top solutions from the optimizer's history to seed the next round, repeating them needed to have exactly n seeds. Since the guided diffusion is not deterministic, despite seeding multiple generative trajectories from the same starting sequence, the model still generates solution pools with high diversity. See Algorithm for pseudo-code of our candidate starting point selection strategy, covering both the a < 1 and a > settings.
Algorithm 8 Get Candidate Starting Points from LaMBO-2 Solution History
Input: History solutions x, scores y. FFT expansion factor a, number of samples to evaluate this iteration n, edit distance function edit dist
sorted_indices <- argsort(y, descending-True))
K + min(len(z))
candidate points +- æ/sorted indiieef K]]
a > then
indices Patttssfrirtaeeeemccccaaaattttte editt,s,eddiid cen.daaddit.ccidddddeesssee
else
repeat_factor + [1/a]+] +1
candidate points + repeat date points, factorr) n)
indices + aangelen
end
Returns: candi pnndiddaateeoiinstnndiiee
38

---

end Generalists vs. Specialists: Evaluating LLMs Hlihhyy-Constrannct Biophysical Sequence Optimization Tasks
Algorithm 9 Farthest First Traversal
Input: Library L of elements, distance function d, ranking scores s, number of elements to select
R argsort(()) 8) // Sort indi ces by scores
S + [M[O]] // Initialize selected indices with first element
R - R[1 // Remove first element from remaining
PQ < // Initialize priority queue
for i = 0, O.....,R|
do dist + -d(L L[S( // Negative distance for max max-heap PQ.push((cist, s
end
for i +- 1t n do
while True do
(dist, score, ida, checked) < PQ-pop() if checked < |5| then
min_dist + min(d(L(jdr)) (((bi)) for j E [checked, (S])
min_dist + min(min dist)
PQ push((-minn dist, score, idæ, (S1))
else
(apppend(idz)
break
end
end
Returns: S
A.8.3. TRAINING AND GENERATION HYPERPARAMETER SWEEPS
We trained LaMBO-2 with batch size of 128, using the Adam optimizer with 31 = 0.9, 82 0.999 -..055 and no weight decay. We also searched over the following hyperparameter ranges:
Number of design steps per iteration E (8. 16,32)
Number of mutations per design step {2, 4, 16}
Diffusion guidance step size E {0.01, 0.05, 0.1, 0.2)
Training epochs {8, 16, 50, 100)
Farthest first traversal (FFT) expansion factor E {0.1, 0.25, 1.0)
39

---

Generalists vs. Specialists: Evaluating LLMs HlihhyyConsstrance Biophysical Sequence Optimization Tasks
A.9. Additional Results
How sensitive is each method hyperparameters, test function diffieulty, and seed dataset? An important aspect of an optimization algorithm is how robust it is to hyperparameter settings, problem difficulty, and choice of seed examples. To explore robustness, present the Pareto frontiers and corresponding hypervolumes of all methods actoss all test functions for evaluation budgets ranging from 1K to 30K in Fig. 10. All three LLOME variants exhibit significantly lower average hypervolume than both the GA and LAMBO-2.
LLOME-DPO (avg. hypervolume=77074001
Genetic Algorithm (avg. hypervolume=11122641
LLOME-MARGE (avg. hypervolume=88221.9)
1.0
0.8
Min. Regret
0.6
AAAA
0.4
0.2
0.0
0
10000
20000
30000
LLOME-SFT (avg. 1y9ervolume=8868.99
LaMBO-2 (avg. hypervolume=10011..)
Num. Test Function Evals
1.0
0.8
Min. Regret
0.6
36 36
x
0.4
30
30 30 x
X
x x
0.2
x
0.0
10000
20000
30000
10000 Num. Test Function Evals
20000
30000
Num. Test Function Evals
Figure 10: Pareto frontiers of evaluation budget vs. minimum regret for variety of LLOME and LAMBO- 2 hyperparameter settings, Ehrlich functions, and seed datasets. The average hypervolume refers to the average (number of test function evaluations x minimum regret). Lower hypervolume is better. However, the average hypervolume of LLOME-DPO is artificially deflated due to many DPO experiments ending prematurely as result of generator collapse.
How sensitive is LLOME to model size? We compare LAMBO-2 against LLOME--MARGH with smaller LLM (226K params, based on the LLaMA architecture), trained from scratch, since the Pythia model used in main results (Fig. 3) is much larger in model size (2.8B parameters versus 314K) and has been pre-trained on the Pile (Gao al., 2020). Although pre-training should not offer any additional advantages due to the lack of overlap between Ehrlich functions and the pre-training data, we choose this setting to be similar in model size and training to LaMBO-2. We evaluate on the f2 function (i.e. Ehr(32, 32)-4-4-4), Results are shown in Fig. 11. Although LLOME-MARGE (LLAMA 226K) performs comparably to LLOME-MARGE (PYTHIA 2.8B) and LAMBO-2 up to 22K test function evaluations, performance eventually plateaus, perhaps owing to limited model capacity. Additionally, the 226K model often exhibits significant training instability, including numerous spikes in both loss and gradient norm. This instability may be due the lack of pre-training.
At inference time, LLMs can iteratively extrapolate beyond their training distributions. Extrapolating beyond the training distribution well-known machine learning problem (Bommasani et al., 2021; Press al., 2022; Li et al., 2024),
40

---

Generalists vs. Specialists: Evaluating LLMs Hlighyy-Constrainn Biophysical Sequence Optimization Tasks
M = 32, L = 32, c=
1.0
0.8
Minimum Regret
0.6
0.4
0.2
LLOME-MARGE (Pythia 2.8B)
LLOME-MARGE (LLaMA 226K)
LaMBO-2 (314K)
0.0
5000
10000
15000
20000
25000
Number Test Function Evaluations
Figure 11: Minimum regret achieved by L.L.ME-MARGE with two LLMs of different sizes, compared against LAMBO-2 on test function f2-
especially without explicit guidance provided at inference time. Although some prior work has shown LLMs to be effective at iteratively generating sequences that monotonically increase particular attribute to values beyond the training distribution (Chan et al., 2021; Padmakumar et al., 2023), much of this work focuses simpler tasks such as increasing the positive sentiment of text or decreasing the AAG of a well-studied protein. Since optimizing an Ehrlich function requires satisfying multiple constraints in addition to generating sequences that lie within a small feasible region, we posit that this evaluation a more challenging assessment of LLMs' inference-time extrapolation capabilities.
We display the iterative refinement results of LLOME'S inner loop in Fig. 16, which suggest that LLMs iteratively produce edits that significantly reduce regret. However, the first few edits frequently improve the sequence whereas later edits less likely to be helpful. This suggests that LLM'' inference-time extrapolative capabilities are limited - without further training explicit guidance, LLMS may be unable to continuously improve given sequence beyond certain threshold. By alternating between optimizing the model's parameters and optimizing the model's outputs, we provide a sample-efficient method for iteratively bootstrapping the model's 's extrapolative abilities using its own generations.
V
32, 32
1.0
0.8
Minimum Regret
0.6
0.4
Genetic Algorithm LLOME-DPO
0.2
LLOME-MARGE
LLOME-SFT
LaMBO-2
0.0
5000 10000 15000 20000 25000 30000
Number of Test Function Evaluations
Figure 12: Minimum regret achieved as a function of the number of test function evaluations, on test function similar to f2 but with half quantization.

---

Generalists vs. Specialists: Evaluating LLMs Conttly-Cn Biophysical Sequence Optimization Tasks
% of Unique Sequences
% of Feasible Sequences
Algorithm
0.6
LLOME-DPO
LLOME-MARGE
LLOME-SFT
JaMRO-2
0.0
5000
10000 150000 20000
25000
30000
5000
10000
150000
20000
25000
30000
Number lest Function Evaluations
Number Test Function Evaluations
Figure 13: The percentage of generated sequences for fs that unique feasible. The line for LLOME-DPO ends early due to degeneration of solutions.
LLOME-DPO
LLOME-DPA LLOME- MARGE
0.8
LLOME-MARGE ----
Average Reward
0.8
Max Reward
LLOME-SFT
LLOME-SFT
0.6
0.4
0.4
0.2
0.0
0.0
12000
16000
20000
24000
28000
12000
16000
20000
24000
28000
Number of Test Function Evaluations
Number Test Funition Evaluations
(a) Maximum reward achieved.
(b) Average reward.
Figure 14: Average and maximum reward achieved by methods that rely upon editing the original sequence. The shaded end early because LLOME-MARGE
regions in (14b) represent the 95% confidence interval. The lines for discovers the optimal solution early.
In Figure 14, we show the average and maximum reward achieved by each LLOME variant on f2. While all three variants achieve similar maximum reward throughout all iterations, LLOME-SFT and ALLME-MARGE achieve significantly higher average reward than -.O.--DP..
% of Feasible Sequences
80%
LLOME-DPO
LLOME-MARGE LLOME-SFT
60%
40%
20%
0%
0.2
0.6
Average Ha Hamming Distance
Figure 15: The percentage of feasible LLM-generated sequences, binned by the average Hamming distance (normalized by length) between the input and output. Shaded regions indicate the 95% confidence interval.
42

---

Generalists vs. Specialists: Evaluating LLMs Highly- Constraindd Biophysical Sequence Optimization Tasks
After 14000 Test Function Evailuations
After 16000 Test Function Evaluations
After 16000 Test Functionn Evaluations
0.0
Regret
0.5
0.0
After
Function
Regge:
0.2
0.0
After 26000 Test Runction Evolliations
After 28000 Test Function Evaluations
After 30000 Test Function Evoluations
- LLOME-DPO
Miniuum Regres
0.8
LLOME MARGE
0.4
0.2
10
10
Iteration X
Iteration
Iberation
Figure 16: Minimum regret of sequences generated during the LLM inner loop, at each iteration of the iterative refinement process. The titles reflect the number of oracle labels that each LLM has been trained on. These plots account for all generations sampled from the LLM inner loop, and not just the samples selected via likelihood selection, in Alg. 3.
43

---

Generalists vs. Specialists: Evaluating LLMs Highly- Biophysical Sequence Optimization Tasks
0.20
LLOME-DPO
Average Reward
LLOME-MARGE
0.15
LLOME-SI SFF
0.10
0.05
0.00
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Likelihood
Figure 17: Likelihood vs. reward.
LLOME-DPO
LLOME-MARGE
LLOME-SFT
Minimum Regret
0.8
0.4
0.2
0.0
10K
15K
20K
25K 10K
15K
20K
25K 10K
15K Number of Test Function Evaluations
20K
25K
Number Test Function Evaluations
Number Test Function Evaluations
Candidate Selection Method Likelihood Selected
- Srecle-Selected
Figure 18: Minimum regret of candidates selected by either L.M likelihood or the oracle on the f2 test function. Since the first 10K test function evaluations are seeds derived from the genetic algorithm, we show only candidates generated after the first 10K.
1.0
Average Regret
0.9
10 SFT, MargE
SFT, 1 MargE
SFT, 3 MargE
0.8
SFT. 5 MargE
3 SFT, 7 MargE
SFT, 9 MargE
0.7
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Likelihood
Figure 19: Calibration of likelihood vs. regret for multi-stage MFr+ training.
44


Reviews:
[{"review_id": "NygpwbI8Y2", "reviewer": "Reviewer_b3Qx", "summary": "This paper introduces a new synthetic test suite (Ehrlich functions) that captures the geometric structure of biophysical sequence optimization problems, proposes a framework LLOME (Language Model Optimization with Margin Expectation), a bilevel optimization routine for online black-box optimization, and uses a preference learning loss called MargE. To evaluate LLMs on biophysical sequence optimization, the paper conducts comparative evaluation of LLMs against specialized solvers like LaMBO-2.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful assessment of our work. We appreciate your recognition of our technical contributions and would like to address your concerns.\n\n## On the choice of Ehrlich functions over existing benchmarks\n\nTo validate the real-world applicability of Ehrlich functions, we conducted new experiments comparing optimizer performance on Ehrlich functions vs. 3 established lookup-based test functions:\n- [TFBind8](https://www.science.org/doi/10.1126/science.aad2257): DNA transcription factor binding optimization (8-base sequence)\n- [TrpB](https://www.science.org/doi/10.1126/science.adh3860): Tryptophan synthase β-subunit protein optimization (4-amino acid sequence)\n- [DhfR](https://www.pnas.org/doi/10.1073/pnas.2400439121): Dihydrofolate reductase DNA binding optimization (9-base sequence)\n\nFor each benchmark, we evaluated the median cumulative regret (estimated from 8 trials) of 64 variants of our GA with different hyperparameter settings. We then computed rank correlations between algorithm performance on these biological benchmarks vs. comparable Ehrlich functions. The strong rank correlations confirm that Ehrlich functions effectively capture the structure of real biological optimization problems:\n| Spearman Corr. with → | DhfR | TFBind8 | TrpB |\n|--------------------|------|---------|------|\n| Ehr(4, 4)-2-2-2    | 0.75 | 0.75    | 0.61 |\n| Ehr(20, 8)-2-2-2   | 0.86 | 0.89    | 0.73 |\n\n## On the design of Ehrlich functions\nWe developed Ehrlich functions after carefully analyzing the limitations of existing benchmarks. As we explain in Section A.2.2, current benchmarks fall into several categories, each with significant drawbacks. **Database lookup** benchmarks are costly to construct and unnecessarily restrictive of the search space. **Empirical function approximations** often have spurious optima that are easy to find but not reflective of real solutions for the biological problem. **Physics-based simulations** are slow to evaluate, difficult to install/run correctly, and admit trivial solutions that score well but are not desirable.\n\nInstead, we designed the Ehrlich suite to have the following criteria: (1) low compute cost, (2) well-characterized solutions, (3) non-trivial difficulty, (4) similarity to real-life applications, and (5) not already seen in training data. To the best of our knowledge, this is currently the only biosequence optimization benchmark to possess all 5 attributes.\n\n## On the relationship between Ehrlich functions and biological sequence optimization\nEhrlich functions capture four key properties of real biophysical sequence optimization:\n1. **Feasibility constraints**: The vast majority of random sequences are non-viable/non-expressible\n2. **Epistasis**: Non-additive effects between sequence positions\n3. **Position-dependent sensitivity**: The importance of specific residues at specific positions\n4. **Motif constraints**: The need for functional motifs to appear with proper spacing\nAs we illustrate in Fig. 6, these properties are directly related to antibody-antigen binding.\n\n## On the choice of comparisons\nWe have added a comparison against LLOME-MargE with a smaller LLM (~226K params), trained from scratch. Although pre-training should not offer any additional advantages due to the lack of overlap between Ehrlich functions and the pre-training data, we choose this setting to be similar in model size and training to LaMBO-2. We evaluate on the f2 function (i.e. Ehr(32, 32)-4-4-4). Despite being ~2/3 the size of LaMBO-2, this model achieves the same min. regret using several thousand fewer test function evaluations (please compare to LaMBO-2 results in Fig. 3b):\n| # Test Function Evals |   10000 |   12000 |   14000 |   16000 |   18000 |   20000 |   22000 |   24000 |   26000 |   28000 |\n|:-----------|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|\n| Min. Regret |   0.906 |   0.859 |   0.789 |   0.719 |   0.625 |     0.5 |   0.438 |   0.438 |   0.438 |   0.438 |\n\nThis illustrates the strength of LLOME-MargE, even in small models without any pre-training.\n\n## On trade-offs\nYou raise an important question about the trade-offs between LLMs and specialized models. We explicitly address this in Section 6.1:\n> \"For relatively easy optimization problems, since the performance of various methods is similar, using a specialized model with 0.01% of the parameters of an LLM may be more practical.\"\n\nOur results show that the choice between generalist and specialist models depends - for medium-difficulty problems, the performance improvements of LLMs may justify their cost. For very easy or difficult problems, specialized models offer comparable performance with much less compute.\n\n## On essential references\nWe have an extended Related Work section in A.1, which discusses many LLM + evolution works. Also, ProteinGym is not a sequence generation benchmark.", "claims_and_evidence": "Claims: \n1. Off-the-shelf LLMs struggle to optimize Ehrlich functions with prompting alone (Yes)\n2. LLOME with MargE can learn to solve some Ehrlich functions (Yes)\n3. LLOME can outperform LaMBO-2 on moderately difficult Ehrlich variants (Yes)\n4. LLMs show limited extrapolative capabilities without further training (Yes)\n\nThe evidence could be strengthened by including comparisons with more baselines.", "methods_and_evaluation_criteria": "The proposed methods are reasonably sound for the problem as defined. Ehrlich functions provide a controlled environment for testing optimization capabilities with well-defined constraints. The evaluation criteria (regret metrics, feasibility, and diversity) are appropriate for measuring optimization performance.\nHowever, the paper only compares against two baselines (genetic algorithm and LaMBO-2) and there is no evaluation on real biophysical optimization tasks to validate transferability.", "theoretical_claims": "The derivation of MargE in Appendix A.3 appears mathematically sound, though relatively straightforward compared to prior work in preference learning. The proof of Lemma A.1 establishing properties of Bradley-Terry models with reward functions helps justify the design choices in the reward function but is not especially novel.", "experimental_designs_or_analyses": "Only two baselines are considered; it is not clear to me whether it is fair to compare the proposed LLOME on pretrained data with LAMBO-2 that is trained from random initialization.", "relation_to_broader_scientific_literature": "The paper builds on directed evolution and genetic algorithms for black-box optimization, adding LLM-based approaches to this domain.", "essential_references_not_discussed": "More baseline works regarding LLMs and evolutionary algorithms can be considered. Here is a survey paper for reference: Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap (https://arxiv.org/pdf/2401.10034)\n\nThere are other standard sequence optimization benchmarks like proteingym that may need to be considered too.", "other_strengths_and_weaknesses": "Pros:\n1. The paper identifies an important challenge in comparing general-purpose and specialized models\n2. The analysis of preference learning calibration provides useful insights\n\nCons:\n1. The significance of Ehrlich functions is not convincingly established; it's unclear why existing benchmarks weren't sufficient. It would be better to motivate why Ehrlich functions are specifically representative of biophysical sequence optimization rather than generic constrained optimization\n2. The design of Ehrlich functions is not well motivated and not clearly explained in relation to biological sequence optimization tasks. It's unclear if the sequence nature of the problem is essential or just incidental to the optimization task\n3. Only two baselines are compared and it seems the comparison is not fair with LAMBO-2", "other_comments_or_suggestions": "See above", "questions_for_authors": "1. How do the authors justify that performance on Ehrlich functions would translate to real biophysical sequence optimization tasks? Could the authors provide evidence of correlation between performance on Ehrlich functions and established benchmarks?\n2. Why develop a new synthetic benchmark rather than using established ones like those in ProteinGym? What specific limitations of existing benchmarks necessitated this approach?\n3. Given that LLMs require significantly more computational resources, how would the authors characterize the trade-offs between performance and efficiency when choosing between generalist and specialist approaches? At what point would the performance improvements justify the increased computational costs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jreKg1Ij1s", "reviewer": "Reviewer_mMvu", "summary": "The authors introduce Ehrlich functions, a novel synthetic function suite designed to simulate the properties of biological sequences and to facilitate benchmarking of generative algorithms for sequence optimization. They also propose a bilevel LLM-based solver, LLOME, which leverages a new preference loss called MargE. Experimental results, benchmarked against LAMBO-2 and GA, suggest that LLOME holds promise for biological sequence optimization tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment and thoughtful questions. We're pleased you recognize the value of our contributions and appreciate your suggestions for strengthening our work.\n\n## On success rates and feasibility\n\nYou raised an important point about success rates in real-world settings with costly wet-lab validation. We directly address this through our measurements of feasibility rates over time in Figures 4 and 10, which show how each method improves in generating valid sequences as optimization progresses.\n\nFigure 12 provides additional insight by analyzing the relationship between edit distance and feasibility, showing that LLOME-MargE achieves the best balance between exploration (making meaningful changes) and constraint satisfaction.\n\nOur simple regret plots (Figure 3) also demonstrate sample efficiency, which is critical when optimization is constrained by laboratory resources. LLOME-MargE consistently finds high-quality solutions with fewer function evaluations than other methods, particularly on medium-difficulty problems.\n\n## On Ehrlich function applicability to real biology\n\nWe share your interest in validating Ehrlich functions against real biological data. To address this, we conducted new experiments comparing optimizer performance on Ehrlich functions versus established biological test functions (TFBind8, DhfR, TrpB -- see the response to Reviewer b3Qx). We found strong rank correlations (0.61-0.89) between algorithm performance on these benchmarks, confirming that Ehrlich functions effectively capture the structure of real biological optimization problems.\n\nThis validation approach allows us to demonstrate biological relevance while avoiding contamination in LLM training data, which could compromise benchmark integrity.\n\n## On unified vs. separate models for scoring and generation\n\nYou asked about the advantages of our unified approach compared to the traditional setup with separate scoring and generation models. The main benefits are:\n\n1. **Improved ranking and guidance**: When a model is jointly trained on both tasks, its internal representations develop a better understanding of the relationship between sequence features and objective values. This improves the model's ability to generate and rank candidates effectively.\n\n2. **Computational efficiency and search depth**: By unifying generation and evaluation in one model, we can perform deeper, more focused exploration with the same computational budget.\n\n3. **Overcoming distributional limitations**: Traditional \"generate-and-filter\" approaches assume the desired outputs already exist in a mode of the generative model's training distribution. Scientific discovery inherently requires finding solutions in low-density regions or even outside the training distribution entirely. Our unified approach enables the model to progressively learn to generate such solutions.\n\n4. **Simplified training and deployment**: Using a single model reduces engineering complexity and maintenance overhead in real-world applications.\n\nThank you for your valuable suggestions. We agree that further validation on real antibody maturation datasets would strengthen our work's impact, and we're actively pursuing this direction for future research. We believe our current contributions provide a solid foundation for advancing the application of LLMs to biological sequence optimization problems.", "claims_and_evidence": "The authors’ claims regarding LLOME, Ehrlich functions, and the MargE preference loss are well substantiated by the experimental results, which clearly demonstrate the efficacy and potential of these methods for biological sequence optimization.", "methods_and_evaluation_criteria": "One key contribution of this work is the introduction of Ehrlich functions for evaluating generative algorithms in biological sequence optimization tasks. While these functions may oversimplify the complexity of real biological sequences, they provide a practical starting point, enabling rapid approximation and assessment of model performance.", "theoretical_claims": "Both the MargE and the construction of Ehrlich functions is sound and correct.", "experimental_designs_or_analyses": "One crucial consideration missing from the experimental design is the success rate. With unlimited time and resources, nearly any optimization method could eventually produce a sequence meeting the required criteria. However, in real-world settings—especially those involving costly wet-lab validation—it’s important to ensure a high success rate in the final (or a limited number of) round(s) of experiments.", "relation_to_broader_scientific_literature": "This work holds significant value for a broad range of scientific fields, including protein optimization, mRNA design, and antibody/CAR T-cell engineering. The design of Ehrlich functions is especially noteworthy for accelerating the development of advanced optimization algorithms. However, to strengthen its relevance for biologists, it would be helpful to validate these functions against existing antibody maturation datasets—demonstrating that, with proper parameter settings and initial seed sequences, they can closely simulate real biological processes. Additionally, expanding the discussion to address broader applications in other biological contexts could further underscore the method’s versatility.", "essential_references_not_discussed": "There are several other attempts in using protein language models in simulating the antibody maturation and experimentally validated the effectiveness with wet-lab experiments. The authors may discuss the links between the proposed method and the ones that biologiest are interested in. See https://doi.org/10.1038/s41587-023-01763-2 and DOI: 10.1126/science.adk8946", "other_strengths_and_weaknesses": "It appears that the authors rely on a single model for both scoring and generation. What are the advantages of using a unified approach compared to the more traditional setup in protein engineering, where one model serves as the ‘oracle’ (scoring function) and a separate LLM is responsible for sequence generation?", "other_comments_or_suggestions": "See comments above.", "questions_for_authors": "Success rates: It would be highly informative to report the success rate of each optimization algorithm, as these metrics are critical for gauging practical utility—particularly when transitioning to costly wet-lab validation.\n\nEhrlich function applicability: Demonstrating whether Ehrlich functions can simulate real antibody maturation datasets would help validate their biological relevance. Such an evaluation could illustrate how well the functions capture key evolutionary or selection pressures inherent in the maturation process.\n\nProtein language models in antibody maturation: A deeper discussion of how protein language models (PLMs) apply to antibody maturation would strengthen the manuscript’s impact. \n\nOne model vs 2 model: Explain the benefit in using one model for joint scoring and optimizing.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "63dz9hY3ab", "reviewer": "Reviewer_2Up6", "summary": "This paper investigates the use of large language models (LLMs) as black-box sequence optimizers for biophysical sequence design and optimization. The authors compare generalist LLM-based approaches with specialized optimization methods, such as LaMBO-2, to determine whether LLMs can efficiently optimize under strict biophysical constraints. The study introduces new benchmarks, novel training objectives, and a bilevel optimization framework to enhance LLM-based sequence optimization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment and constructive feedback. We appreciate your recognition of our thorough experiments and well-defined benchmarks.\n\n## On testing with real-world biological datasets\n\nYou noted that our evaluation relies on synthetic Ehrlich functions rather than real-world biological datasets. While direct evaluation on real biological data would indeed be valuable, there are two key challenges:\n\n1. **Training data contamination**: Using widely available biological datasets risks contamination with LLM training data, which would invalidate our assessment of LLMs as black-box optimizers.\n\n2. **Accessibility**: Real biological datasets and simulators often require specialized software, significant computational or experimental resources, or domain expertise, creating barriers to reproduction and wider adoption.\n\nTo address these concerns while maintaining biological relevance, we conducted new experiments comparing optimizer performance on Ehrlich functions versus established lookup-based biological test functions (TFBind8, DhfR, TrpB -- see our response to reviewer b3Qx). We found strong rank correlations (0.61-0.89) between algorithm performance across these benchmarks, confirming that Ehrlich functions effectively capture the structure of real biological optimization problems.\n\nThis validation approach allows us to demonstrate biological relevance while avoiding contamination in LLM training data. It also preserves the computational accessibility that makes Ehrlich functions valuable for algorithm development.\n\n## On additional references\n\nThank you for suggesting additional relevant references. We will include them in the revised version to better situate our work within the growing literature on protein sequence optimization using LLMs.\n\nWe believe our work provides several unique contributions to this field:\n\n1. A systematically designed benchmark that balances realism, computational accessibility, and difficulty\n2. A bilevel optimization framework that effectively leverages LLMs for constrained optimization\n3. A novel preference learning objective that outperforms SFT, DPO, and REINFORCE when rewards are observed.\n\nThese contributions advance both the theoretical understanding and practical application of LLMs for biological sequence optimization.\n\nThank you again for your thoughtful review and suggestions for improvement.", "claims_and_evidence": "The majority of claims in this paper are supported by thorough experiments, well-defined benchmarks, and ablation studies. However, some claims would benefit from additional validation.\n\nThe only benchmark used is the Ehrlich functions, which are synthetic test functions, no real-world biological datasets (e.g., protein sequences, DNA regulatory elements) are tested.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally well-designed for assessing LLMs in biophysical sequence optimization.", "theoretical_claims": "All correct.", "experimental_designs_or_analyses": "No issues.", "relation_to_broader_scientific_literature": "This paper contributes to LLM-based sequence optimization, preference learning for biophysical design, and comparisons between generalist (LLM-based) and specialist (model-based) solvers.", "essential_references_not_discussed": "No significant reference are missing.", "other_strengths_and_weaknesses": "See above", "other_comments_or_suggestions": "Please considering reference some prior work on the protein sequence optimization:\n\nChen, A., Stanton, S. D., Alberstein, R. G., Watkins, A. M., Bonneau, R., Gligorijević, V., ... & Frey, N. C. (2024). LLMs are highly-constrained biophysical sequence optimizers. arXiv preprint arXiv:2410.22296.\n\nGomez-Uribe, C. A., Gado, J., & Islamov, M. (2024). Designing diverse and high-performance proteins with a large language model in the loop. bioRxiv, 2024-10.\n\nSubramanian, J., Sujit, S., Irtisam, N., Sain, U., Islam, R., Nowrouzezahrai, D., & Kahou, S. E. (2024). Reinforcement Learning for Sequence Design Leveraging Protein Language Models. arXiv preprint arXiv:2407.03154.\n\nWang, Y., He, J., Du, Y., Chen, X., Li, J. C., Liu, L. P., ... & Hassoun, S. (2025). Large Language Model is Secretly a Protein Sequence Optimizer. arXiv preprint arXiv:2501.09274.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ejBIuJ9DI3", "reviewer": "Reviewer_xgMH", "summary": "This paper tackles the problem of biophysical sequence optimization - a task where even small deviations from stringent constraints (e.g., protein stability or solubility) can render a solution unusable. To bridge the gap between generalist LLM-based methods and specialist solvers, the authors introduce a synthetic test suite and a optimization framework that continuously optimize protein sequence using LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment of our work. We appreciate your recognition that our approach is well-motivated and technically sound.\n\n## On the applicability of synthetic benchmarks to real-world cases\n\nYou raised an important question about whether our synthetic benchmarks apply to real-world cases. To address this directly, we conducted new experiments comparing optimizer performance on Ehrlich functions versus established lookup-based biological test functions (TFBind8, DhfR, TrpB -- see our response to reviewer b3Qx). We found strong rank correlations (0.61-0.89) between algorithm performance across these benchmarks, confirming that Ehrlich functions effectively capture the structure of real biological optimization problems.\n\nThis validation supports our benchmark design principles, which were carefully crafted to capture key properties of real biophysical sequence optimization:\n\n1. **Feasibility constraints**: The vast majority of random sequences fail to express or fold properly\n2. **Epistasis**: Non-additive effects between sequence positions\n3. **Position-dependent sensitivity**: The importance of specific residues at specific positions\n4. **Motif constraints**: The need for functional motifs to appear with proper spacing\n\nBy deliberately incorporating these properties, Ehrlich functions provide meaningful insights into algorithm performance on real-world biological optimization tasks while maintaining computational accessibility.\n\n## On the computational cost of LLMs\n\nYou also asked about the computational cost of using LLMs. We address this important practical consideration in Section 6.1:\n\n> \"For relatively easy optimization problems, since the performance of various methods is similar, using a specialized model with 0.01% of the parameters of an LLM may be more practical.\"\n\nOur experiments revealed an interesting nuance: the optimal choice between generalist LLMs and specialist models depends on problem difficulty. For medium-difficulty problems, LLMs with appropriate training can significantly outperform specialized models, potentially justifying their higher computational cost. For very easy or very difficult problems, however, specialized models offer comparable performance with substantially lower computational requirements. This insight provides practical guidance for practitioners choosing between approaches based on their specific constraints and objectives.\n\nWe have additionally conducted new LLOME-MargE experiments with a very small LLM (~226K params) trained from scratch (see our response to reviewer b3Qx). We found that even for a very small LLM with no pre-training, LLOME-MargE is significantly more sample efficient than LaMBO-2, despite having fewer model parameters than LaMBO-2. As such, the computational costs of LLOME need not always be a concern.\n\n\n\nThank you again for your thoughtful review. We believe our work contributes valuable insights to both the machine learning and biological sequence design communities, and we appreciate your recognition of its potential impact.", "claims_and_evidence": "Yes, the proposed methods seem to be well-motivated and well supported.", "methods_and_evaluation_criteria": "Strengths:\n- The paper presents a well-motivated and technically sound approach. The whole framework is self-contained, borrowing insights from discrete optimization, LLM fine-tuning, and preference learning.\n- The benchmark (test suites) design is novel\n\nWeakness/Question:\n- Does the synthetic benchmarks generally applicable to real-world case?\n- How is the computation cost of the LLMs?", "theoretical_claims": "The paper does not contain any theoretical claims. The preference learning objectives are not new (in the appendix) so it doesn't really require further examination.", "experimental_designs_or_analyses": "The overall experimental design is comprehensive and verify the effectiveness of  the proposed method", "relation_to_broader_scientific_literature": "The key contribution can be seen as a application of LLM on the tasks of biophysical sequence optimization, it would be interesting to researcher who works on this field", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See Methods And Evaluation Criteria", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See Methods And Evaluation Criteria", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Other expertise"]}}]

Author rebuttal:

Meta-review: The authors introduce a synthetic benchmark suite and LLM-based optimization algorithm that does well on this suite. The reviewers overall appreciate the contributions of the paper, and the synthetic benchmark suite in particular. I do think that there are some lingering weaknesses -- namely that it would be great to validate the synthetic benchmark suite against empirical data to show that performance on the synthetic benchmark suite correlates with some real world task of interest. To this end I do appreciate the authors' inclusion of rank correlations against TFBind8, DhfR, TrpB (and am ultimately recommending acceptance), although I do think as much validation here would be great.
Decision: Accept (poster)