Steering Protein Language Models
Authors: Long-Kai Huang
Rongyi Zhu
Bing He
Jianhua Yao
Venue: ICML 2025 poster (2025)
Keywords: Protein Language Model
Steering
Protein Engineering
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: An activation editing method in Protein Language Model to steer the sequence generation toward desired properties and its downstream application to protein optimizaiton.

Abstract: Protein Language Models (PLMs), pre-trained on extensive evolutionary data from natural proteins, have emerged as indispensable tools for protein design. While powerful, PLMs often struggle to produce proteins with precisely specified functionalities or properties due to inherent challenges in controlling their outputs. 
In this work, we investigate the potential of Activation Steering, a technique originally developed for controlling text generation in Large Language Models (LLMs), to direct PLMs toward generating protein sequences with targeted properties. We propose a simple yet effective method that employs activation editing to steer PLM outputs, and extend this approach to protein optimization through a novel editing site identification module. Through comprehensive experiments on lysozyme-like sequence generation and optimization, we demonstrate that our methods can be seamlessly integrated into both auto-encoding and autoregressive PLMs without requiring additional training. These results highlight a promising direction for precise protein engineering using foundation models.
Code is available at <https://github.com/Long-Kai/Steering-PLMs>.

Steering Protein Language Models
Long-Kai Huang' Rongyi Zhu
Bing He
Jianhua Yao
Abstract
mance in predicting diverse protein properties (Mardiko- raem & Woldring 2023; Notin et al., 2022: Zhang et al., 2024; Chu et al., 2024; Gordon et al., 2024). However, their ability to generate proteins with precisely specified proper- ties remains limited, typically requiring massive sequence generation followed by resource-intensive screening.
Protein Language Models (PLMs), pre-trained on extensive evolutionary data from natural proteins, have emerged as indispensable tools for protein design. While powerful, PLMs often struggle to produce proteins with precisely specified func- tionalities properties due inherent challenges in controlling their outputs. In this work, we in- vestigate the potential of Activation Steering, a technique originally developed for controlling text generation in Large Language Models (LLMs), to direct PLMs toward generating protein sequences with targeted properties. We propose a simple yet effective method that employs activation editing to steer PLM outputs, and extend this approach protein optimization through novel editing site identification module. Through comprehensive experiments on lysozyme-Iie sequence genera- tion and optimization, we demonstrate that our methods can be seamlessly integrated into both auto-encoding and autoregressive PLMs without requiring additional training. These results high- light a promising direction for precise protein engineering using foundation models. Code is available at Github!.
To address this limitation, several strategies have been explored to control the generation. One straightforward method involves fine-tuning PLMs using dataset consist- ing of proteins that exhibit desired characteristics, thereby converting a general-purpose model into specialized gen- erator ( (Nijkamp et al., 2023). This approach, however, de- mands hundreds or even thousands of high-quality data and substantial computational resources. Besides, risks dilut- ing the general knowledge encapsulated during the initial pre-training phase. Another strategy incorporates special keyword tags indicating functionalities properties during the pre- pre-training, aiming to guide the generation process akin to prompting in large language models (LLMs) for natural language processing (Madani et al., 2020; Lv et al., 2024). Yet, this approach lacks flexibility, as any control requires alignment with the tags used during the pre-training phase, limiting its adaptability new controls. These challenges motivate the exploration of inference-time control meth- ods that preserve model knowledge while enabling precise steering.
Inference-ting intervention methods, known as activation steering or activation editing, have been introduced guide the generated texts of LLMs toward desired behaviors (Sub- ramani et al., 2022; Turner et al., 2023; Panickssery et al., 2023; Wang & Shu, 2023; Liu al., 2023; Li et al., 2024; Zou et al., 2023; Cao et al., 2024; Qiu et al., 2024; Lee et al., 2024a). These methods presuppose that the models inherently possess the knowledge required to generate the desired output in the internal representations but may not always actualize this potential in its outputs. By modifying the internal activations, we can steer the model's behavior to produce the desired texts. Despite their success in LLMs, these techniques remain largely unexplored in the context of PLMs, where the unique characteristics of protein languages present additional challenges.
1. Introduction
Protein Language Models (PLMs) (Madani et al., 2020; Ni- jkamp et al., 2023; Lin et al., 2022; Hayes et al., 2024; Lv et al., 2024) have emerged as transformative tools for under- standing and designing proteins (Notin et al., 2022; Strokach & Kim, 2022; Ferruz& & Höcker, 2022; Meier al., 2021). By distilling evolutionary information from billions of pro- tein sequences, these models encode rich biological knowl- edge about protein structure and function. Coupled with additional predictors, PLMs achieve oftte-of-the-ar perfor-
'Tencent Al Lab. Correspondence to: <hlongkai @ gmw.l.com>>...... Jianhua Yao <>.
Long- Kai Huang
Proceedings of the ngnd International Conference on Machine I Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we explore the potential of activation steer- ing in PLMs, aiming guide protein generation toward

---

Steering Protein Language Models
sequences with specifie properties. We begin by confirming that PLMs indeed encode knowledge about these proper- ties, as detailed in Section 3.1. Subsequently, we adapt the Activation Addition technique (Turne et al., 2023), orig- inally developed for LLMs, to steer the outputs of both auto-regressive and auto-encoding PLMs. Specifically, we compute a steering vector as the mean difference in inter- nal representations between proteins with and without the target property. During inference, add this vector to the models activations, biasing generation toward proteins with the desired characteristics.
eration. These models excel in diverse tasks: zero-shot mutation effect prediction (Meier et al., 2021), evolutionary trajectory modeling (Hie et al., 2022), and structure-awar design (Zheng et al., 2023). Their latent spaces encode bio- physical properties, supporting state-of-the-art performance in fitness prediction (Hie et al., 2024) and atomic-level struc- ture inference (Lin et al., 2023).
Recent advances extends PLMs to multitask settings (Pei et al., 2024; Lv et al., 2024) and controllable protein gen- eration (Lv et al., 2024; Madani et al., 2020; Ferruz & Höcker, 2022). However, steering PLM outputs toward user-specified functional traits, such as increased stability or solubility, remains challenging. Unlike natural language, protein generation requires preserving structural viability and evolutionary plausibility, which limits the effective- ness of standard LLM control methods like prompting or fine-tuning- While PLMM have revolutionized protein engi- neering, their latent spaces remain underutilized for targeted activation-based interventions, highlighting the potential for adapting inference- time steering techniques from NLP to protein design.
We further extend the proposed method to protein opti- mization tasks using auto-encoding PLMs, which predict beneficial mutations protein sequences toward the target property. Without steering, standard models, guided by co- evolutionary patterns learned from natural proteins, tend to predict sequences similar to those found in nature. To direct these models towards generating novel proteins with desired properties, we first identify mutation sites that are crucial for achieving these properties and then apply activation steering in the prediction. To implement this, we propose a novel al- gorithm that selects mutation sites based on the dissimilarity between the token representations and the steering vector. By integrating this algorithm with activation steering, our method effectively guides protein sequence optimization toward target properties.
Activation Steering modifies models behavior infer- ence time by perturbing its internal activations, without training or changing model weights. In LLMs, methods like activation addition (ActAdd) (Turner et al., 2023) compute steering vectors as contrasts between activations of opposing prompts (c.g., truthful vs. deceptive) and inject these into hidden states to influence outputs. Recent works improve this approach: contrastive activation addition (CAA) (Pan- ickssery al., 2023) aggregates steering vectors from hun- dreds of contrast pairs (e.g., truthful vs. hallucinated re- sponses) to reduce noise and improve robustness across diverse prompts. Other methods refine vector extraction through dataset-driven preferences (Li et al., 2024), optimal transport (Singh et al., 2024), conditional interventions (Qiu et al., 2024), or bi-level optimization (Cao et al., 2024), en- abling more precise control. These techniques have been used to steer attributes such as style, safety, and truthful- ness (7ou et al., 2023; Liu et al., 2023) with applications ranging from bias mitigation (Adila et al., 2024) to adver- sarial robustness (Wang& & Shu, 2023).
Our work makes three key contributions:
1) We present the first application of activation steering to PLMs, enabling property-specific protein generation without retraining:
2) We propose a novel protein optimization framework that integrates activation steering with mutation site identification in auto-encoding PLMS;
3) We provide comprehensive empirical validation on pro- tein generation and optimization across diverse PLM architectures (Pr, ESM2, ESM3) and biologi- cal properties, including thermostability, solubility, and green fluorescent protein (GFP) brightness.
2. Related Works
While activation steering is well-studied LLMs, its ap- plication to protein language models (PLMs) remains unex- plored. Unlike LLMs, PLMs generate outputs that reflect biophysical functionalities rather than linguistic semantics, and their latent spaces are shaped by evolutionary and struc- tural constraints. Existing LLM methods focus on abstract linguistic properties, while steering PLM generations re- quires grounding activation vectors in sequence- functiion relationships (e.g., stability, thermostability, or solubility). To bridge this gap, in this work, we adapt activation addi- tion to PLMs by constructing steering vectors from protein
Protein Language Models (PLMs) leverage transformer architectures to learn functional and structural patterns from evolutionary protein sequences. PLMs can be broadly cate- gorized into auto-encoding (AE) and autoregressive (AR) architectures. AE-PLMs like ESM2 (Lin et al., 2022) and ESM3 (Hayes et al., 2024) use masked language model- ing to capture bidirectional dependencies, which are essen- tial for understanding protein sequences. AR-PLMs like ProGen (Madani et al., 2020; Nijkamp et al., 2023) adopt causal language modeling, enabling de novo protein gen-
2

---

Steering Protein Language Models
sequences with contrasting functional traits, and demon- strate that PLM generations can be reliably steered toward user-specified properties.
ESM2
ESM3
ProLLaMA
Thermostabillty
Protein Optimization aims to design sequences with im- proved functional properties while preserving structural vi- ability. Traditional approaches, such as directed evolution (DE), rely on iterative mutation and screening (Romero & Arnold, 2009). Machine aearning-assisted DE (MLDE) ac- celerates this process by predicting fitness from sequence data (Wu et al., 2019; Wittmann et al., 2021), but its de- pendence on experimental labels limits scalability (Yang et al., 2025). Zero-shot predictors, such as PLM likelihoods, partially mitigate this by estimating fitness without labeled data (Meier et al., 2021; Notin et al., 2024), but they struggle to explicitly guide generation toward desired properties.
Lager
Uyerr
Salubility
Lage
layer
Loweed
Figure f. t-SNE visualization of PLMM activations from ESM2, ESM3, and ProLLaMA for proteins with (red) and without (green) the target properties: high thermostability (top row) or high sol- ubility (bottom row). Partial separation of clusters suggests that property-relatee information encoded in activation space.
Recent advances leverage generative PLMs for de novo protein design (Madani et al., 2023; Nijkamp et al., 2023) and latent space optimization (Stanton et al.. 2022; Kirjner et al., 2023). For example, (Kirjner et al., 2023) proposed to smooth noisy fitness landscapes using energy-based mod- els to guide Gibbs-with Sitbb--wit-GGrdientts sampling, while (Stan- ton et al., 2022) proposed optimize sequences in latent space via denoising autoencoders. However, these methods require training differentiable fitness proxies or imposing explicit structural constraints, limiting their flexibility. As for reinforcement learning (Lee et al., 2024b; Angermueller et al., 2019) and evolutionary algorithms (Ren et al., 2022) based methods, they both face trade-offs between explo- ration and computational cost.
reveals that proteins with and without target properties form partially distinct clusters in activation space. This pattern is consistent across different models. These observations con- firm that PLMs inherently encapsulate intrinsic knowledge about specific protein properties and motivate our approach to steering protein generation toward desired characteristics through activation space manipulation.
3.2. Activation Steering for PLMs
To guide protein generation toward desired properties, we employ activation steering, which modifies model activa- tions using steering vectors. Specifically, each layer 1, the activation is edited as:
In contrast, activation steering offers lightweight alter- native. By perturbing PLM activations at inference time, we bypass weight updates and expensive sampling and di- rectly inject functional preferences into generation. Unlike previous PLM-based optimization methods, which use like- lihoods as proxies or fine-tune models on labeled data, our method directly steers generation toward target properties without explicit fitness predictors.
h h + avi,
(1)
where h, and h are the activations in the I-th layer before and after steering, respectively, v: is the steering vector for the 1-th layer, and a is scalar hyper-parameter controlling the steering strength. After modification, the edited acti- vations h, is rescaled to have the same norm as hy before being passed to the next layer. The steering process is il- lustrated in Figure 2b. We apply activation steering to all layers except the input layer and across all tokens.
3. Activation Steering for PLMs
3.1. Premise Verification
Protein language models (PLMs) have been shown to pro- vide effective representations for downstream property pre- diction tasks (Notin et al., 2022), indicating that they capture relevant functional and structural information. This capa- bility forms the foundation for activation steering, where model activations are manipulated to influence the property of the generated proteins.
The steering vectors used in Equation (1) are computed as the mean difference in representations at the I-th layer be- tween sets of proteins with and without the desired property. These steering vectors point in the direction from undesired to desired properties. For AF-PLMMs, we use the average activations across all tokens; for AR-PLMs, we use the last token's activation. Formally, for AE-PLMs, the steering vectors are calculated as:
To support this premise, we analyze the internal representa- tions of several P..Ms As shown in Figure 1, t-SNE visual- izations of activations from ESM2, ESM3, and ProLLaMA
Vi
1 P
h
M
1
Σ h ang (Fn), E.EN
(2)
PP
3

---

Steering Protein Language Models
A
B
postive
amg (xp)
Vi
P
Transformer block
Input tokens
Transformer block
Transformer block I+1
Rr
negative set N
(xxx)
(8s)
input tokens
layer feature
D
input
prediction tokens x
hj
Relatedness
<MASK>
g
x
ki
Identified mutation sites Zzr
x²
Transformer black
Aclivation Steering
x-3
kF
22
<MASK>
g
Eq. (4)
ga
x
x
Figure Overview of Activation Steering for PLMs and Activation Steering based Protein Optimization (ASPO). (a) Computation stecring vectors: For each layer, the steering vector is computed the mean difference in activations between positive (desired property) and negative (undesired property) protein sets. (b) Activation steering during generation: At cach layer, model activations are modified by adding a scaled steering vector. (c) Identification of mutation sites: For given protein, token representations at selected layer projected onto the steering vector to compute relatedness scores. Tokens with the lowest scores (most negatively related to the target property) selected candidate mutation sites. (d) Mutation prediction: Identified mutation sites masked, and amino acids are predicted using activation steering.
and for AR-PLMs:
codes the direction of the desired property in the models representation space. For each token, we compute related- ness score as the cosine similarity of its representation h to V/:
1
Σ JpEP
h last (I,)
Σ hz)st(zn) J.EN
(In).
(3)
P
W
where P and N are the positive and negative sets of pro- teins regarding the desired property, respectively. They are the sets of proteins with and without the desired property. h h,"*((n (zp) and (l(" denote the average activation of all tokens and last token activation of the 1-th layer for quence input respectively. The computation of steering vectors is illustrated in Figure 2a.
s = sim(h vi)
(4)
| b
where s is the relatedness score of the k-th token in the I-th layer. The projection of a tokens representation onto vi quantifies relatedness to the target property. Tokens with large positive projection (large s²) indicate their correspond- ing AAs are strongly related to the property, while those with large negative projections (small s*) are less related opposed. After computing the relatedness scores for all tokens, rank these scores and select the T tokens with the lowest values (i.e., most negatively related to the target prop- erty) as mutation sites. The PLM then predicts new amino acids for these positions, guided by activation steering. This process illustrated in Figure 2c and Figure 2d.
3.3. Protein Optimization via Activation Steering
AB-PLMs predict the amino acid (AA) masked position in protein sequence using the contextual information of surrounding AAs. While this mechanism leverages coevolu- tionary patterns from natural proteins, does not directly optimize for specific target properties. To optimize a protein for desired properties, we propose to identify and mutate tokens that are negatively related the target property, and apply activation steering guide PLM's prediction at these positions toward desired properties.
To compute the relatedness score, we need identify the most informative layer. Specifically, split the positive (P) and negative (A) sets into training and validation subsets. For each layer, we train a linear classifier on the training subset to distinguish between representations from P and N,, and evaluate the validation accuracy on the validation subsets. We then compute the relatedness scores for the
key challenge is to systematically identify which AAs in sequence are most opposed to the target property. We address this by leveraging the steering vector vi, which en-

---

Steering Protein Language Models
Algorithm 1 Activation Steering based Protein Optimization (ASPO)
1: Input: protein sequence z, positive protein sequence set P. negative set NV, steering strength a, layer for relatedness
score computation, number of mutation sites per round T, and number of rounds R
2: Compute steering vectors {vi} for all layers 1 1,2, ..., I. using Equation (3)
3: for r to R do
4: Compute token representations h for all tokens k: = 1, 2, K at layer L.
5: Compute the relatedness scores s for all tokens using Equation (4).
6: Obtain the set of the token indices of the T lowest scores in {s{} } as Ir-
7: Mask tokens at positions in Zr.
8: Predict new amino acids at positions in Tr using activation steering (Equation (1)) with steering vectors {vi).
9:
end for
layer with the highest validation accuracy and use them to select the mutation sites.
for each protein sequence and report the average with stan- dard deviation. Detailed metric definitions are provided in Appendix A.1.
The mutation process is repeated for R rounds progres- sively steer the protein towards the desired properties. In each round, we compute token representations, calculate their relatedness scores, and select the mutation sites. We then mask the tokens in the selected positions and predict new amino acids using activation steering. We refer to this method as Activation Steering based Protein Optimization (ASPO), summarized in Algorithm 1.
To construct the positive and negative sets for steering vector extraction, we first predict thermostability or solubil- ity for all lysozyme-like proteins in the UniRef50 dataset using property-specifer predictors. For thermostabilliy, se- quences with predicted values above 70 form the high thermostability set, and those below 50CC form the low ther- mostability set. For solubility, we construct a high solubility set using the sequences with predicted soluble probability higher than 0.8 and a low solubility set using the sequences with predicted soluble probability lower than 0.15. We ran- domly sample sequences from each high and low set to form the positive (P) and negative (N) sets, respectively.
Data:
4. Experiments
In this section, we evaluate the effectiveness of our acti- vation steering method to control protein language models (PLMs) for property-drive -diven protein generation and optimiza- tion.
Hyper-parameter settings: We fix positive and negative set sizes for steering vector extraction 100 and set 1.0 by default. The sensitivity of these hyperparameters will be explored in Section 4.3.
4.1. Steering PLMs for Protein Generation
4.1.1. EXPERIMENTAL SETTINGS
Baselines:
We compare our method to two baselines: (1) PLMs fine-tuned on positive sets (Fine-tuning), and (2) the original, unmodified models (Original Model). For AE- PLMs, we fine-tune only the last layer. For AR-PLMs, we use LoRA (Hu et al., 2022) on all layers with rank 4 and alpha 16. To evaluate performance, we generate 1000 sequences for each method. For AE-PLMs, we randomly se- lect 1000 lysozyme-like proteins from the UniRef50 dataset as initial reference sequences to generate 1000 sequences.
Tasks: We focus on the generation of lysozyme-like pro- teins with enhanced thermostability or solubility by steering PLMs toward these properties.
Base Models:
We assess the effectiveness of our method across two types of PLMs: 1) auto-encoding PLMs (AE-PLMs), including ESM2 (650M) (Lin et al., 2022) and ESM3-open (1.4B) (Hayes et al., 2024); and 2) auto-regressive PLMs (AR-PLMs), including ProLLaMA (7B) (Lv et al., 2024). ProLLaMA enables controlled se- quence generation via superfamily descriptions, which we use to restrict outputs to the lysozyme-likk like family. For AE- PLMs, which do not generate sequences directly, we start from a reference sequence and, in each iteration, randomly select and mask 10% of the tokens without replacement, then regenerate them using the model.
4.1.2. RESULTS AND ANALYSIS
Table summarizes the performance of activation steering compared to fine-tuning and the original models across both auto-regressive (ProLLaMA) and auto-encoding (ESM2, ESM3) PLMs. For both thermostability and solubility tasks, activation steering consistently outperforms the baselines in terms of the target property. For example, on ESM2, activa- tion steering achieves a thermostability of 82.20 compared to 63.56 for fine-tuning and 56.48 for the original model. Similar improvements are observed for solubility, where ac-
To evaluate all methods, we assess their generated sequences for target property fitness, diver- sity, and novelty. For each metric, we calculate the value
Evaluation Metrics:
5

---

Steering Protein Language Models
Table Comparison of generating lysozyme-like protein with high thermostability or solubility. Results are reported as mean (std) for
each metric.
Thermostability Diversity
Solubility
Base Model
Method
Thermostability
Novelty t
Solubility 1
Diversity t
Novelty f
Original Model
56.18 (8.05)
0,931 (0.035)
0.767 (0.064)
0.230 (0.085)
0.931 (0.035)
0.767 (0.064)
ProLlama
Fine-tuning
57.24 (8.64)
0.958 (0.017)
0.798 (0.068)
0.241 (0.086)
0.958 (0.017)
0.838 (0.059)
Activation Steering
67.68 (12.86)
0.927 (0.027)
0.807 (0.063)
0.276 (0.110)
0.964 (0.016)
0.882 (0.056)
Original Model
56,48 (12.04)
0,954 (0.023)
0.591 (0.110)
0.289 (0.151)
0.963 (0.019)
0.596 (0.130)
FSM2
Fine-tuning
63.56 (14.87)
0.953 (0.023)
0.585 (0.099)
0.356 (0.213)
0.961 (0.020)
0.594 (0.132)
Activation Steering
82.20 (12.92)
0.971 (0.023)
0.739 (0.130)
0.494 (0.241)
0.998 (0.001)
0,880 (0.087)
Original Model
55,20 (11,14)
0,952 (0.021)
0.573 (0.100)
0.257 (0.177)
0.958 (0.017)
0.579 (0.123)
ESM3
Fine-tuning
62.82 (14.72)
0,949 (0.021)
0.568 (0.104)
0.318 (0.215)
0.955 (0.017)
0.570 (0.119)
Activation Steering
82.06 (12.06)
0.954 (0.019) 0.614 (0.115)
0.582 (0.264)
0.966 (0.019) 0.639 (0.123)
Table Comparison Lysozyme- like protein optimization toward high thermostabillty. Results are reported as mean (std).
Medium difficulty
Hard difficulty
Fitness f
Diversity
Dissimmr
Dissimagt
Fitness
Diversity
Dissim
Dissimaih
Before Optimization
59.78 (3.04)
0.879 (0.072)
0.601 (0.100)
46.38 (3.11)
0.923 (0.038)
0,708 (0.087)
AdaLcad
63.56 (11.94)
0,947 (0.036)
0.351 (0.166)
0.697 (0.096)
55.16 (9.29)
0.962 (0.018)
0.626 (0.185)
0.832 (0.069)
PEX
66.80 (10.95)
0,923 (0.053)
0.203 (0.087)
0.651 (0.086)
48.95 (5.75)
0.959 (0.022)
0.185 (0.094)
0.741 (0.073)
GWG
68.25 (9.35)
0.885 (0.068)
0.059 (0.024)
0.611 (0.096)
47.73 (3.90)
0.926 (0.036)
0.049 (0.010)
0.708 (0.081)
ESM2+ ASPO
84.34 (7.59)
0.840 (0.064)
0.290 (0.167)
0.661 (0.114)
74.69 (12.32)
0.828 (0.062)
0.291 (0.163)
0.734 (0.076)
ESM3 ASPO
88.42 (3.98)
0.803 (0.060)
0.110 (0.055)
0.603 (0.077)
86.43 (9.02)
0.865 (0.048)
0.161 (0.093)
0.714 (0.072)
Table 3. Comparison of lysozyme- like protein optimization toward high solubility. Results are reported as mean (std) for each metric.
Medium difficully
Hard difficulty
Fitness 1
Diversity
Dissim
Dis
Fitness
Diversity
Dissim
Dissimaig
Before Optimization
0.278 (0.012)
0,898 (0.054)
0
0.684 (0.108)
0.085 (0.011)
0.896 (0.056)
0
0.689 (0.097)
AdaLcad
0.617(0.247)
0,949 (0.021)
0.475 (0.162)
0.794 (0.079)
0.534 (0.283)
0.959 (0.017)
0.512 (0.177)
0.792 (0.080)
PEX
0.489 (0.246)
0,920 (0.044)
0.080 (0.032)
0.711 (0.101)
0.252 (0.240)
0.927 (0.041)
0.096 (0.043)
0.715 (0.089)
GWG
0,356 (0.1
0,912 (0.042)
0.060 (0.019)
0.694 (0.101)
0.165 (0.130)
0.919 (0.041)
0.071 (0.030)
0.707 (0.089)
ESM2+ ASPO
0,510 (0.282)
0,860 (0.061)
0.022 (0.011)
0.724 (0. 105)
0.349 (0.273)
0.838 (0.052)
0.018 (0.011)
0.710 (0.063)
ESMB ASPO
0.654 (0.273)
0.879 (0.054)
0.058 (0.039)
0,720 (0.109)
0.397 (0.247)
0.858 (0.055)
0.057 (0.038)
0.711 (0.060)
Table Comparison of GFP optimization toward high fluorescence brightness. Results are reported as mean (std) for each metric.
Medium difficulty
Hard difficulty
Fitness 1
Diversity
Dissimaa
Dissimaih
Fitness
Diversity
Dissimani
Dissimaah
Before Optimization
L.494 (0.340)
0.717 (0.002)
0
0.028 (0.005)
1.325 (0.279)
0.560 (0.002)
0.032 (0.005)
AdaLcad
1.179 (0.329)
0.737 (0.024)
0.060 (0.088)
0.085 (0.085)
1.255 (0.372)
0.596 (0.041)
0.071 (0.095)
0.099 (0.092)
PEX
1.426 (0.337)
0.719 (0.002)
0.004 (0.004)
0.032 (0.007)
1.320 (0.298)
0.563 (0.003)
0.004 (0.004)
0.036 (0.007)
GWG
1.683 (0.641)
0.721 (0.003)
0.021 (0.002)
0.039 (0.010)
1.510 (0.545)
0.568 (0.004)
0.021 (0.002)
0.043 (0.009)
ESM2+ ASPO
3.862 (0.329)
0.397 (0.005)
0.020 (0.007)
0.010 (0.007)
3.907 (0.247)
0.406 (0.006)
0,022 (0.009)
0.012 (0.009)
ESM3 ASPO
3.739 (0.357)
0.503 (0.004)
0.021 (0.007) 0.010 (0.007)
3.687 (0.321)
0.507 (0.005) 0.024 (0.009)
0.012 (0.008)
tivation steering reaches 0.494, substantially higher than the baselines. These results demonstrate that activation steer- ing is highly effective at guiding PLMs to generate protein sequences with desired properties.
fine-tuned models. This indicates that our method does not simply memorize or overfit to the positive set, but is capable of generating broad range of novel and diverse sequences. Overall, these results highlight the advantage of activation steering for controllable and diverse protein design.
In addition to property optimization, activation steering maintains or even improves sequence diversity and novelty. For instance, on ESM3, activation steering achieves the highest solubility (0.582) while also increasing diversity (0.966) and novelty (0.639) compared to the original and
Additional experimental results for our method using ESM- 3B and steering multiple target properties are provided in Appendix B.
6

---

Steering Protein Language Models
4.2. Steering AE-PLMs for Protein Optimization
ground-truth fitness during optimization and do not update the surrogate fitness predictor for these methods.
4.2.1. EXPERIMENTAL SETTINGS
Hyper-parameter settings: We use the same default set- tings as the experiments for protein generation to set the positive set and negative set sizes as 100 and steering strength a = 1.0. For protein optimization specific hyper- parameters, we set the number of optimization rounds R and the number of mutation sites per round T 4 for thermostabilityy experiments and set R = 4 and T = 2 for the solubility and GFP brightness experiments.
We evaluate our method on three protein optimiza- tion tasks: improving thermostability, solubility, and the fluorescence intensity of Green Fluorescent Protein (GFP). For thermostabilitty and solubility, we focus on lysozyme- like proteins. For GFP, we follow the established setup in (Ren et al., 2022; Kirjner et al., 2023).
Tasks:
Evaluation Metries: We assess all methods using four met- rics: target property fitness, diversity, dissimilarity to the initial set (Dissimant)) and dissimilarity to the high-fitness set (Dissimaigh). Detailed definitions of these metrics are provided in Appendix A.
4.2.2. RESULTS AND ANALYSIS
Thermostability Optimization. Table 2 shows the results for optimizing the thermostability of lysozyme-like proteins. Our ASPO methods, ESM2+ASPO and ESM3+ASPO, achieve the highest fitness scores across medium and hard difficulty settings. Specifically, ESM3+ASPO attains fit- ness of 88.42 (medium) and 86.43 (hard), significantly out- performing all baselines. While AdaLead and PEX improve fitness over the initial set, their gains are notably smaller. In terms of diversity and dissimilarity metrics, ASPO methods yield slightly lower diversity compared to baselines, but achieve low dissimilarity to the initial set and maintain com- petitive dissimilarity the high-fitness sets. This suggests that ASPO effectively steers sequences toward high-fitness regions without excessive exploration, focusing optimiza- tion on relevant sequence space.
As noted by (Kirjner et al., 2023), higher diversity and dissimilarity to the initial set do not necessarily equate to superior performance in protein optimization. Similarly, while high Dissimaigh suggests optimization toward the high- fitness reference set, it does not guarantee discovery of all possible high-fitness proteins. Therefore, it is possible for a method that generates high-fitness proteins but achieves just a fair value of Dissimaigh-
Data: For thermostability and solubility, we use the same positive and negative sets as in our protein generation exper- iments. For GFP brightness, we adopt the same data split as (Kirjner et al., 2023) and randomly select 100 sequences from easy difficulty as the positive set and 100 sequences from hard difficulty as the negative set. We follow (Kirjner et al., 2023) to construct medium difficulty task and a hard difficulty task for each target property. Each task has an initial set for optimization. For thermostability optimization, the hard difficulty task uses sequences from the low ther- mostability set as the initial set, while the medium difficulty task uses sequences with predicted thermostability between 50CC and 70ºC. For solubility optimization, the hard diffi- culty task uses sequences from the low solubility set, and the medium difficulty task uses sequences with predicted solu- ble probability between 0.3 and 0.6. To estimate Dissimmight we use the high thermostability set and high solubility set as the reference high-fitness fitness set. For GFP brightness, both initial and high-fitt fitne-iiessss sets follow (Kirjner et al., 2023).
Solubility Optimization. Table presents results for sol- ubility optimization. ASPO (especially ESM3+ASPO) achieves the highest or competitive fitness in both medium and hard solubility tasks, with fitness values of 0.654 and 0.397, respectively. Adal.ead also performs well in terms of fitness, but at the cost of dissimilarity to the initial set, indicating a broader but less targeted search. In contrast, ASPO methods maintain lower dissimilarity to the initial set, reflecting a more focused optimization process. These results demonstrate that ASPO can efficiently improve sol- ubility while generating sequences that are not excessively divergent from the starting set.
GFP Brightness Optimization. Table 4 presents results for GFP fluorescence brightness optimization. Here, both ESM2+ASPO and ESM3+ASPO achieve a significant in- crease in fitness, reaching 3.862 and 3.739 for the medium difficulty task, and 3.907 and 3.687 for the hard difficulty task, respectively. These values are more than double those of the best-performing baselines. Although the diversity of ASPO-genen generated sequences is lower, the marked improve- ment in the target property demonstrates the methods strong optimization capability. Dissimilarity metrics remain low, suggesting that ASPO finds high-fitness, variants close to the initial set, which may be advantageous for practical protein engineering where minimal sequence changes are preferred.
Baselines: We compare our Activation Steering-base based Pro- tein Optimization (ASPO) method, implemented on ESM2 and ESM3, against AdaLead (Sinai et al., 2020), PEX (Ren et al., 2022), and GGS (Kirjner et al., 2023). These baseline methods all require a surrogate fitness predictor. Therefore, we train the surrogate fitness predictor using the positive and negative sets for steering vector extraction. Note that the original AdaLead and PEX both update the surrogate fitness predictor using the generated sequences with ground- truth fitness obtained from wet-lab experiments in each round. To ensure a fair comparison, we assume no access to
7

---

Steering Protein Language Models
E542-45
ESM2 AACOO
12
22
26
303
350
203
250
0.00
25
360
156
200
256
0:28
25
26
120
126
200
(a) Generation
(b) Optimization Meedium
(c) Optimization-Ha
6.6
6.25
3.32
2.45
0:00
635
11
300
200
50
350
350
Novdar Sampis Carret Staering Wetto's
Swnder Sandles
Number Samaaaa Socaet Scaering Mastors
(d) Generation
(e) Optimization-Medlae Medium
(I) Optimization-Hart
Figure 3. Sensitivity to the number of samples used for steering vector extraction. Columns show results for protein generation, medium- difficulty protein optimization, and hard-difficulty protein optimization. The first row is for thermostability, and the second row for solubility.
Summary. Across all tasks and difficulty levels, ASPO consistently achieves the highest nigh-highest fitness val- ues. These results collectively demonstrate that integrating activation steering with mutation site identification enables more precise and reliable protein optimization compared to existing search-bane based methods.
near peak values the number of samples continues grow, except for the ESM2+ASPO in the hard difficulty task for thermostabilitty optimization. In contrast, ProLLaMA's peak performance is achieved with just 10 samples, after which there is a gradual decline. This pattern suggests that the bidirectional attention mechanisms of AE-PLMs benefit from larger, more diverse example sets to establish robust steering directions, whereas the average feature extracted from the last token in AR-PLMs may become less effective with larger sets, potentially due to increased divergence in the examples.
4.3. Sensitivity to Hyperparameters
This section analyzes the sensitivity of our activation steer- ing method to two key hyperparameters: 1) the size of the positive and negative sets used for steering vector extrac- tion, and 2) the hyperparameter used for controlling the stecring strength in activation addition. We evaluate their impact on both protein generation and optimization tasks.
These observations offer practical guidelines for setting the size of the positive and negative sets. Using 100 sam- ples generally provides reliable performance across various tasks and architectures, with diminishing returns observed beyond this point. Specifically, for activation steering ProILaMA in protein generation tasks, this configuration preserves over 90% of the maximum fitness performance. For ESM2 and ESM3, it captures more than 95% of the maximum potential gains. Thus, we recommend a default setting of 100 samples for these experiments.
Sensitivity to the positive and negative sets size. We in- vestigate how the size of the positive and negative sets, ranging from 10 to 250 samples, affects the effectiveness of the proposed activation steering across different model architectures and tasks.
The findings, illustrated in Figure 3, reveal varying trends and optimal set sizes depending on the architecture and task. For ESM2 and ESM3, performance generally improves as the set size increases from 10 to 100 samples, stabilizing
Sensitivity steering strength (a). We investigate the impact of the steering strength a, varying it from 0.05 to 20, across different tasks and models.
8

---

Steering Protein Language Models
35
64-
-ACEO
0.55
025
65
26
3.55
0.25
0.5
(a) Generation
(b) Optimization OptimizationMMedimm
(c) Optimization Hard
0.6
6.55
044
6.54
044
0:00
635
02
0.23
0.25
0.6
+2gg21
x78g21
(d) Generation
(e) Optimization Medium
Oppimization-Medium
(f) Optimization-Haann
Figure 4. Sensitivity of steering strength Columns show results for protein generation, medium-difficull protein optimization, and hard-difficulty protein optimization. The first row for thermostability, and the second row is for solubility..
As shown in Figure 4, the steering strength a exhibits task- dependent landscapes. For protein generation for enhanced thermostability (Fig. 4.2.2), both ESM2 and ESM3 achieves peak performance
generating and optimizing proteins with desired properties. By deriving steering vectors from contrasting protein sets and applying them to perturb PLM activations during infer- ence, our method enables precise, training- control over sequence generation. Our Activation Steering-basee Protein Optimization (ASPO) framework further enhances protein engineering by integrating activation editing with mutation site identification. Because our approach does not require model retraining or explicit fitness predictors, it offers a scalable and efficient alternative traditional methods such as directed evolution or reinforcement learning based pro- tein optimization. Ultimately, activation steering provides promising direction for programmable protein design using foundation models.
0.5 and remain stable for Z 0.5. This suggests that auto-encoding models benefit from moderate steering strength.
In the case of protein generation for enhanced solubility (Fig. 4.2.2), however, performance drops sharply for large a values. For both ESM2 and ESM3, we observe that their performance collapses completely
20, indicating in solubility task, the performance is more sensitive over- steering than thermostability task. Generally over-steering (a > 5) catastrophically degrades solubility performance while only mildly impacting thermostability.
Given these observations, we recommend a default a = 1.0 for most applications, achieving 90-98% of maximum per-
Acknowledgements
formance across tasks while avoiding performance cliffs. Practitioners may consider lowering solubility-focused applications raising to 2.0 for thermostability optimization.
LK Huang would like to thank Kangfei Zhao for the support during the writing of this paper.
0.5 for
Impact Statement
5. Conclusion
This paper presents work whose goal is to advance the in- terdisciplinary field of machine learning and protein design. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
In this paper, we demonstrate the viability of activation steering as a powerful paradigm for guiding PLM toward

---

Steering Protein Language Models
References
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:// openrevien net htt///ormmppp/ wetammiiinnneeeeeeeee...
Adila, D.. Zhang, S., Han, B., and Wang, Y. Discovering bias in latent space: An unsupervised debiasing approach. arXiv preprint arXiv:2406.03631, 2024.
Angermueller, C., Dohan, D., Belanger, D., Deshpande, R., Murphy, K., and Colwell, L. Model-based reinforcement learning for biological sequence design. In International conference on learning representations, 2019.
Jarzab, A., Kurzawa, N., Hopf, T., Moerch, M., Zecha, J., Leijten, N., Bian, Y., Musiol, E.,, Maschberger, M., Stoehr, G., et al. Meltome atlasthermal proteome stability across the tree of life. Nature methods, 17(5):495 503,, 2020.
Cao, Y., Zhang, T., Cao, B., Yin, Z., Lin, L., Ma, F., and Chen, J. Personalized steering of large language mod- cls: Versatile steering vectors through bi-directional pref- erence optimization. arXiv preprint arXiv:2406.0006 00045, 2024.
Khurana, S., Rawi, R., Kunji, K., Chuang, G.-Y,, Bensmail, H., and Mall, R. Deepsol: deep learning framework for sequence- protein solubility prediction. Bioinfor- matics, 34(15):2655-2613, 2018.
Kirjner, A., Yim, J., Samusevich, R., Bracha, S., Jaakkola, T. S., Barzilay, R., and Fiete, I. R. Improving protein optimization with smoothed fitness landscapes. In The Twelfth International Conference on Learning Represen- tations, 2023.
Chang, C. C. Song, J., Tey, B. T., and Ramanan, R. N. Bioinformatics approaches for improved recombinant protein production in escherichia coli: protein solubility prediction. Briefings in bioinformatics, 15(6):533-622 2014.
Lee, B. W., Padhi, I., Ramamurthy, K. N., Miehling, E., Dognin, P., Nagireddy, M., and Dhurandhar, A. Program- ming refusal with conditional activation steering, arXiv preprint arXiv:2409.05907 2024a.
Chu, S. K., Narang, K., and Siegel, J. B. Protein stability prediction by fine-tuning protein language model on mega-scale dataset. PLOS Computational Biology, 20(7): e1012248, 2024.
Lee, M., Vecchietti, L. F., Jung, H., Ro, H. J., Cha, M., and Kim, H. M. Robust optimization in protein fitness landscapes using reinforcement learning in latent space. arXiv preprint arXiv:2405..8990 18986, 2024b.
Ferruz, N. and Höcker, B. Controllable protein design with language models. Nature Machine Intelligence, 4(6): 521-532, 2022.
Li, K., Patel, O., Viégas, F., Pfister, H., and Wattenberg, M. Inference-time time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024.
Gordon, C., Lu, A. X., and Abbeel, P. Protein language model fitness is matter of preference. bioRxiv, pp. 2024 10, 2024.
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wig- gert, M., Badkundri, R., Shafkat, I., Gong, J., Derry, A., Molina, R. S., Thomas, N., Khan, Y. A., Mishra, C., Kim, C., Bartie, I.. J., Nemeth, M., Hsu, P. D., Sercu, T., Candido, S., and Rives, A. Simulating 500 mil- lion years of evolution with language model. bioRxiv, 2024. doi: 10. //doi. org / 1101 11w.12101022221
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., dos Santos Costa, A., Fazel-Zaraadi, M., Sercu, T., Candido, S.. al. Language models of pro- tein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. Evolutionary
01
URL https: 01. 07..0..053.......
prediction of atomic-level protein structure with a language model. Science, 379(6637): 1123 1130, 2023.
Hie, B. L., Yang, K. K., and Kim, S. Evolutionary velocity with protein language models predicts evolutionary dy- namics of diverse proteins. Cell Systems, 13(4):274-285, 2022.
Liu, S., Ye, H., Xing, L.,, and Zou, J. In-context vec- tors: Making in context learning more effective and con- trollable through latent space steering. arXiv preprint arXiv:231 1.f::23,,006668, 2023.
Hie, B. L.,, Shanker, V. R., Xu, D., Bruun, T. U., Weiden- bacher, P. A., Tang, S., Wu, W., Pak, J. E., and Kim, P. S. Efficient evolution of human antibodies from gen- eral protein language models. Nature Biotechnology, 42 (2):255-288) 2024.
Lv, L., Lin, Z., Li, H., Liu, Y., Cui, J., Chen, C. Y--.,, Yuan, L.,, and Tian, Y. Prollama: A protein large language model for multi-task protein language processing. arXiv preprint arXiv:2402. 16445, 2024.
10

---

Steering Protein Language Models
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P.-S., and Socher, R. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03497, 2020.
Romero, P. A. and Arnold, F. H. Exploring protein fitness landscapes by directed evolution. Nature reviews Molec- ular cell biology, 10(12):866877, 2009.
Sinai, S., Wang, R., Whatley, A., Slocum, S., Locane, E., and Kelsic, E. D. Adalead: A simple and robust adap- tive greedy search algorithm for sequence design. arXiv preprint arXiv:2010.0214: 2020.
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, M., Olmos, J. I.., Xiong. C., Sun, Z. Z.,, Socher, R., et al. Large language models gener- ate functional protein sequences across diverse families. Nature Biotechnology, 41(8):199-11166 2023.
Singh, S., Ravfogel, S., Herzig. J., Aharoni, R., Cotterell, R., and Kumaraguru, P. Mimic: Minimally modified counterfactuals in the representation space. arXiv preprint arXiv:2402.0963/ 2024.
Mardikoraem, M. and Woldring, D. Protein fitness pre- diction is impacted by the interplay of language models, ensemble learning, and sampling methods. Pharmaceu- tics, 1533: 2023.
Stanton, S., Maddox, W., Gruver, N., Maffettone, P., De- laney, E., Greenside, P., and Wilson, A. G. Accelerating bayesian optimization for biological sequence design with denoising autoencoders. In International Conference on Machine Learning, pp. 20459-200000 20459- PMLR, 2022.
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., and Rives, A. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in neural information processing systems, 34:29287-29303 2021.
Steinegger, M. and Söding, J. Mmseqs2 enables sensi- tive protein sequence searching for the analysis of mas- sive data sets. Nature biotechmology, 35111)10026--028, 2017.
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., and Madani, A. Progen2: exploring the boundaries of protein language models. Cell systems, 14(11):968-99) 2023.
Strokach, A. and Kim, P. M. Deep generative modeling for protein design. Current opinion in structural biology, 72: 226-236, 2022.
Notin, P. Dias, M., Frazer, J., Marchena-Hurtado. J., Gomez, A. N., Marks, D., and Gal, Y. Tranception: pro- tein fitness prediction with autoregressive transformers and inference-time retrieval. In International Conference on Machine Learning, pp. 16990-17017. PMLR, 2022.
Subramani, N., Suresh, N., and Peters, M. E. Extracting latent steering vectors from pretrained language models. arXiv preprint arXiv:2205.00 00.0005555/22 124, 2022.
Notin, P., Kollasch, A., Ritter, D., Van Nickerk, L.. Paul, S., Spinner, H., Rollins, N., Shaw, A., Orenbuch, R., Weitzman, R., et al. Proteingym: Large-scale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing Systems, 36, 2024.
Suzek, B. E., Wang. Y., Huang, H., McGarvey, P. B., Wu, C. H., and Consortium, U. Uniref clusters: comprehen- sive and scalable alternative for improving sequence sim- ilarity searches. Bioinformatiis, 31(6):926-933 2015.
Panickssery, N., Gabrieli, N., Schulz, Tong, M., Hubinger, E., and Turner, A. M. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023.
Turner, A. M., Thiergart, I.., Leech, G., Udell, D., Vazquez, J. J., Mini, U., and MacDiarmid, M. Activation addition: Steering language models without optimization. arXiv e-prints, pp. arXiv-2308, 2023.
Pei. Q.. Wu, L., Gao, K., Liang, X., Fang, Y., Zhu, J., Xie, S., Qin, T., and Yan, R. Biot5+: Towards generalized biological understanding with iupac integration and multi- task tuning. arXiv preprint arXiv:2402. 17810, 2024.
Wang. H. and Shu, K. Backdoor activation attack: Attack large language models using activation steering for safety- alignment. arXiv preprint arXiv:2311.00 1.09433, 2023.
Wittmann, B. J., Yue, Y., and Amold, F. H. Informed train- ing set design enables efficient machine learning- directed protein evolution. Cell systems, 12(11):1026- 1045, 2021.
Qiu, Y., Zhao, Z., Ziser, Y., Korhonen, A., Ponti, E. M., and Cohen, S. B. Spectral editing of activations for large language model alignment. arXiv preprint 240.022405.09719 2024.
Wu, Z., Kan, S. J., Lewis, R. D., Wittmann, B. J., and Amoll, F. H. Machine learning-assisted directed protein evolution with combinatorial libraries. Proceedings of the National Academy of Sciences, 116(18):8852-888) 8858, 2019.
Ren, Z., Li, J., Ding, F., Zhou, Y., Ma, J., and Peng, J. Proximal exploration for model-guided protein sequence design. In International Conference on Machine Learn- ing, pp. 18520-18536 PMLR, 2022.

---

Steering Protein Language Models
Yang, J., Lal, R. G., Bowden, C., Astudillo, R., Hameedi, M. A., Kaur, S., Hill, M., Yue, Y., and Arnold, F. H. Active baamiing-assisted directed evolution. Nature Com- munications, 16(1):714, 2025.
Zhang, Z., Notin, P., Huang, Y., Lozano, A., Chenthamarak- shan, V., Marks, D., Das, P., and Tang, J. Multi-scale rep- resentation learning for protein fitness prediction. arXiv preprint arXiv:2412.01 108, 2024.
Zheng, Z., Deng, Y., Xue, D., Zhou, Y., Ye, F., and Gu, Q- Structure-informed language models are protein design- ers. In International conference machine learning, pp. 42317-42338. PMLR, 2023.
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.K., al. Representation engineering: A top-down approach to ai transparency. arXiv preprint 20..:riiivi:.011140 2023.
12

---

Steering Protein Language Models
A. Supplementary Experimental Details
A.1. Definition of Metries
Fitness quantifies how well a protein exhibits the desired properties. We estimate fitness using predictors described in Appendix A.2 for thermostability and solubility. For GFP, we use the predictor from (Kirjner al., 2023) estimate the log fluorescence intensity.
Dissimilarity Score measures how different two sequences are. In our experiment, we estimate the dissimilarity in set-wise manner using MMseqs2 (Steinegger & Söding, 2017). Specifically, given query set and target set, we use MMseqs2 to align sequences from the query set to the target set, with a maximum E-value of 10.0 (-e 10.0) and sensitivity of 15.0 (-8 15.0). For each matched pair, we define dissimilarity as 1.0 - percent identity, where percent identity is reported by MMseqs2. For pairs with no match (not reported by MMseqs2), we assign a dissimilarity of 1.0.
Diversity assesses how distinct the generated sequences are from each other. We run MMseqs2 with the generated set as both query and target, compute dissimilarity for all sequence pairs (excluding self-pairs), and define the diversity of each sequence the minimum dissimilarity to any other sequence in the set. We use the same approach to evaluate diversity in protein optimization outputs.
Novelty measures how different the generated sequences are from a reference set. In our protein generation experiments, the reference set all lysozyme-like proteins in the UniRef50 (Suzek et al., 2015) dataset. We use MMseqs2 to align generated sequences (query) to the reference set (target), and compute novelty as the average dissimilarity for each generated sequence against all reference sequences.
Dissimilarity to Initial Set (Dissimmat) quantifies how different the optimized sequences from the initial set. We compute the average dissimilarity of each optimized sequence to all sequences in the initial set, following the same procedure as for novelty.
Dissimilarity to High-Fitness Set (Dissimnigh) is defined similarly to Dissimait- but uses the high- fitness set as the reference set.
A.2. Fitness Predictor
Thermostability Predictor. We construct a thermostabilitty predictor using ESM2-650M as the feature extractor. The predictor adopts the same architecture as the "Im_head" in ESM2-650M and is trained with mean squared error loss. For training, we use data from the Meltome Atlas (Jarzab et al., 2020), which provides melting temperatures for 48,000 proteins across 13 species (archaea to humans), with values ranging from 3 90%C. To focus on sequence-dependen effects and minimize species-specift variation, use the median melting temperature across all species for each protein its final label.
The dataset is split into 90% for training and 10% for testing. To reduce redundancy, we ensure a maximum sequence identity of 90% within the training set. Furthermore, any training sequence with 230% identity to a test sequence is removed, preventing information leakage and ensuring fair evaluation. The final dataset contains 24,817 proteins for training and 3.134 for testing.
On the test set, the predictor achieves a Spearman rank correlation of 0.76.
Solubility Predictor. The solubility predictor is binary classifier with the same architecture and training procedure as the thermostability predictor. We use the preprocessed dataset in khurana201 8deepsol, containing 28,972 soluble and 40,448 insoluble proteins. The data is split 90%/10% for training and validation. For benchmarking, we use an independent test set in (Chang et al., 2014), which includes 1,000 soluble and 1,001 insoluble proteins.
On this test set, our predictor achieves accuracy of 0.708, precision of 0.758, recall of 0.612, and Fl score of 0.677. demonstrating its effectiveness for sequence-based -based solubility prediction.
13

---

Steering Protein Language Models
Table Comparison of generating lysozyme-like protein with both high thermestability and solubility. Results are reported as mean (std).
Thermostability
Solubility
Diversity
Novelty
Original Model
56.45 (11.07)
0.328 (0.151)
0.967 (0.019)
0.596 (0.136)
ESM2
Fine-tuning Activation Steering
59.69 (13.22)
0.406 (0.
0.966 (0.020)
0.596 13.138)
68.02 (14.44)
0.483 (0.244)
0.992 (0.005)
0.950 (0.070)
Original Model
54,46 ( (10.48)
0.314 (0.1
0.962 (0.016)
0.572 (0.122)
ESM3
Fine-tuning
60.22 (13,90)
0.425 (0.206)
0.960 (0.016)
0.568 (0. (2.)
Activation Steering
66.75 (9.61)
0.451 (0.253)
0.980 (0.009) 0.925 (0.108)
Table 6. Comparison of generating lysozyme-likk protein with high thermostability or solubility. Results are reported mean (std).
Thermostability
Solubility Diversity 1
Base Model
Method
Thermostability 1
Diversity f
Novelty f
Solubility
Novelty 1
Original Model
56,48 (12.04)
0.954 (0.023)
0.591 (0.110)
0.289 (0.151)
0.963 (0.019)
0.596 (0.130)
ESM2-650M
Fine-tuning
63.56 (14.87)
0.953 (0.023)
0.585 (0.099)
0.356 (0.213)
0.961 (0.020)
0.594 10.)
Activation Steering
82.20 (12.92)
0.971 (0.023)
0.739 (0.130)
0.494 (0.241)
0.998 (0.001)
0.880 (0.087)
Original Model
56.05 (11.24)
0.968 (0.020)
0.632 (0.143)
0.298 (0.174)
0.971 (0.021)
0.622 (0.162)
FSM2 3B
Fine-tuning
64.22 (14.49)
0.965 (0.022)
0.629 (0.143)
0.385 (0 (0.236)
0.966 (0.022)
0.610 16.)
Activation Steering
83.33 (9.47)
0.990 (0.006)
0.915 (0.105)
0.631 (0.228)
0.996 (0.003)
0.951 (0.077)
B. Additional Experiment Results
B.1. Activation Steering for Multiple Desired Properties
Previous experiments demonstrated the effectiveness of Activation Steering for guiding protein generation toward a single desired property. In this part, we extend our evaluation to the simultaneous optimization of multiple properties. Specifically, we aim to lysozyme-like proteins with both high thermostability and solubility.
For our Activation Steering, we compute the steering vectors for thermostability and solubility as of offhrm and 1970, respectively. We then obtain the steering vectors for performing activation addition in Equation (1) as iq 0.5ufferm 1861. For the fine-tuning baseline, we fine-tune the model using the union of positive data sets for these two properties.
Table 5 shows that Activation Steering consistently outperforms both the original model and fine-tuning across all metrics and backbones (ESM2 and ESM3). For instance, on ESM2, Activation Steering improves thermostability from 56.45 to 68.02, and solubility from 0.328 (original) and 0.406 (fine-tuned) to 0.483. Similar trends hold for ESM3. Although gains are smaller when optimizing multiple properties at once, Activation Steering remains more effective than fine-tuning or the original model for jointly optimizing multiple protein properties, without reducing diversity novelty.
B.2. Activation Steering on Larger PLM
To assess the scalability of our activation steering method, we evaluate its performance on the larger ESM2-3B model, using ESM2-650M reference. The task settings and baselines are the same as the protein generation experiments described in Section 4.1.
Table 6 summarizes the results. Compared to the smaller ESM2-650M, ESM2-3B consistently achieves higher scores across all metrics, demonstrating the benefits of scaling up the model size for protein generation tasks.
On ESM2-3B, Activation Steering achieves the best performance for both thermostability and solubility, with mean values of 83.33 and 0,631, respectively. These improvements are accompanied by substantial gains in diversity and novelty. The performance gap between Activation Steering and the baselines is even more pronounced on ESM2-3B than on ESM2-650M, indicating that the effectiveness of Activation Steering is amplified the model size increases. This suggests that large protein language models are better able to leverage activation- steering for generating diverse and novel sequences with improved target properties.
In summary, these results demonstrate that Activation Steering remains effective as model size increases, and that scaling up the model further enhances its ability to generate proteins with desirable characteristics.
14


Reviews:
[{"review_id": "2REE19Kfbs", "reviewer": "Reviewer_ApT8", "summary": "This paper introduces activation steering, which is a technique adapted from large language models, to control protein language models for generating and optimizing protein sequences with targeted properties (e.g., thermostability, solubility, fluorescence). \nThe method modifies internal model activations using steering vectors derived from contrastive representations of proteins with desired and undesired properties. For optimization tasks, the authors propose ASPO (activation steering-based protein optimization) and identify critical mutation sites via projection onto steering vectors. Experiments across autoencoder (i.e., ESM2, ESM3) and autoregressive (i.e., ProLLaMA) protein LMs demonstrate significant improvements in target properties without model retraining, outperforming fine-tuning and traditional optimization baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point. \n\n**W1: the ESM3 model ... generate a full sequence from all-mask states**\n\nWe conduct experiments on full sequence generation using ESM3, maintaining the same settings as described in Section 4.1. The results are presented in the table below, demonstrating that Activation Steering significantly outperforms the baseline methods, thereby confirming its effectiveness.\n\n||Thermostability|Solubility|\n|-|-|-|\n|Original Model|52.8|0.376|\n|Fine-tuning|65.5|0.412|\n|Activation Steering|79.6|0.466|\n\n**W2: more details about experiments**\n\n- The number of iterations T and the value of K\n    - Protein optimization: thermostability: T=8, K=4; solubility: T=4, K=2; GFP: T=4, K=2\n- Length distribution of generated sequences\n    - Protein generation\n        - thermostability: 60~256, mean: 194.1\n        - solubility: 47~256, mean: 168.5\n    - Protein optimization\n        - thermostability\n            - medium difficulty: 60~256, mean: 183.9\n            - hard: 102-256, mean: 208.2\n        - solubility\n            - medium: 71~256, mean: 180.4\n            - hard: 47~253, mean: 158.9\n        - GFP: length of all sequences is 237\n\n**W3: further explain the lines 189-194**\n\nOur method first involves computing a relatedness score for each token to identify which should be mutated, as discussed prior to the mentioned lines. In lines 189-194, we propose to determine the layer for computing the relatedness score as the one with the highest validation accuracy. This ensures that the most informative layer are utilized for token selection.\n\nAfter determining the tokens for mutation, we replace these tokens with a mask token. Importantly, activation steering is applied during inference across all layers, except the input layer, to steer the model's prediction on the masked tokens towards the desired property. \n\nWe will revise lines 189-194 to enhance clarity on these points.\n\n**W4: Missing discussion of PPLM**\n\nThank you for pointing out the omission of PPLM. PPLM indeed pioneers the concept of steering by modifying key-value pairs in the model's attention mechanism, guided by gradients from an attribute model. In contrast, our method employs a simpler activation steering approach that directly manipulates activations and does not require training an additional attribute model or updating steering vectors. \n\nWe will include discussion of PPLM in the related work to highlight these distinctions.\n\n**Q1: Could the performance of steering be further boosted by enlarging the model scale**\n\n- We conducted experiments using ESM2-3B for protein sequence generation, maintaining the same settings as in Sec 4.1. The results are summarized in the table below.\n    - Compared to ESM2-650M, the proposed method Activation Steering shows similar performance in thermostability and significantly improves solubility. \n    - In contrast, fine-tuning performance decreases with larger ESM2, likely due to the need for more data to achieve optimal results.\n\n||Thermostability|Solubility|\n|-|-|-|\n|Original Model|56.1|0.298|\n|FT|64.2|0.385|\n|AS|80.5|0.631|\n\n**Q2: the performance when the number of samples is further reduced**\n\n- We appreciate the reviewer's interest in the performance of AR-PLM with fewer samples. To address this, we conducted additional experiments using ProLLaMA with sample sizes ranging from 1 to 10. Each configuration was tested 10 times to mitigate randomness. The results, summarized in the table below, indicate that the optimal number of samples varies depending on the property being optimized. For thermostability, the best performance occurs with 8 samples, while for solubility, it peaks at 3 samples. Notably, the performance does not consistently improve with fewer samples; the lowest number of samples (1 sample) did not yield the best results.\n- This suggests that while reducing the number of samples can sometimes enhance performance, likely by focusing the generation on a narrower subcluster of proteins with desired properties, there is a trade-off in terms of robustness. Performance becomes less predictable and can vary significantly depending on the specific samples used to compute steering vectors. \n- In conclusion, while fewer samples can sometimes be beneficial, the optimal number of samples depends on the specific application and desired property, balancing performance with robustness.\n\n||1|2|3|5|8|10|\n|-|-|-|-|-|-|-|\n|Thermostability|64.3|61.8|57.4|71.8|74.5|73.5|\n|Solubility|0.344|0.491|0.507|0.492|0.446|0.302|\n\n**Q3: compatible with other protein steering or optimization methods?**\n\nIn this paper, we focus on studying steering for PLM and we apply PLM steering exclusively within the proposed ASPO method for protein optimization, positioning ASPO as a competitor rather than complementary to existing methods. Integration with other protein optimization strategies remains an interesting direction for future research.", "claims_and_evidence": "Mostly supported.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Not applicable since there's no theoretical contributions.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "[PPLM] Plug and Play Language Models: A Simple Approach to Controlled Text Generation. In ICLR 2020.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The proposed activation steering is a training-free powerful method to generate protein sequences with target properties.\n2. The authors also propose a novel Activation Steering-based Protein Optimization (ASPO) framework to improve the protein optimization performance and identify the mutation site.\n3. Experimental results demonstrate that the proposed approach is able to significantly improve the steering generation performance on target property across various protein language models (ESM2, ESM3, ProLLaMA).\n\n**Weaknesses**\n\n1. In section 4.1, the ESM3 model is able to generate a full sequence from all-mask states, while ESM2 can not. Therefore, for the ESM3 model, authors should also provide the results of directly generating full sequences in addition to revising based on a reference sequence.\n2. Authors should provide more details about experiments, such as the number of iterations T, length distribution of generated sequences.\n3. Clarity issue. Could you please further explain the lines 189-194: \"For practical implementation, [...], A linear classifier to distinguish the representations from the desired and undersired sets.\"? Does this mean that you only steer the activation of the layer with the highest validation accuracy, instead of all layers, during inference?\n4. Missing discussion of PPLM, an important related work on steering pre-trained LMs for conditional generation.", "other_comments_or_suggestions": "**Minor:**\n\nTypo: line 361, \"Sensitivityive\".", "questions_for_authors": "1. ESM2/3 models both demonstrate the scalability that a larger scale model is able to obtain better performance in various tasks in their paper. Therefore, could the performance of steering be further boosted by enlarging the model scale (such as, using ESM2 3B/15B or ESM3 7B/98B)?\n2. In Figure 3, for AR-PLM, as the number of samples for extracting steering vectors increases, the relevant property values consistently decline. The line graph in the figure starts at 10 samples, and I am curious about the performance when the number of samples is further reduced (e.g., 1 sample). If the final trend shows that performance is always decreasing as the number of samples increases, does this mean that for AR-PLMs, users should use as few samples as possible for steering to obtain the best result?\n3. Considering that the proposed method is training-free, will the proposed steering approach be compatible with other protein steering or optimization methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZJTgCGEo79", "reviewer": "Reviewer_fV4m", "summary": "This paper adapts activation steering techniques from LLMs to Protein Language Models (PLMs) to guide protein generation toward desired properties without retraining. It introduces an Activation Steering-based Protein Optimization (ASPO) framework that outperforms existing methods on thermostability, solubility, and GFP brightness tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate very much your constructive comments on our paper. We have provided a detailed explanation for your questions as follows. Please feel free to let us know if you have any additional concerns or questions.\n\n----\n\n**Q1: \"How do you deal with multi-property optimization?\"**\n\nThank you for your insightful question regarding our method for multi-property optimization.. \n\n- We propose a simple solution to perform steering on multiple properties by computing a composite steering vector that incorporates the steering vectors of individual properties. Specifically, if $v_\\ell^{therm}$  represents the steering vector for thermostability \nand  $v^{sol}_\\ell$ for solubility, the combined steering vector used for multi-property optimization is given by \n\n$v_\\ell = v_\\ell^{therm} + v^{sol}_{\\ell}$.\n\n- The remaining settings are the same as single property optimization.\n- We empirically test this method in scenarios involving both protein generation and protein optimization. The results, as detailed in the tables below, demonstrate that our method effectively improves multiple properties, albeit with a slight trade-off compared to optimizing a single property.\n- We plan to further refine our method to better manage these trade-offs and explore more sophisticated methods for steering vector combination and weighting for multi-property optimization in future work.\n\n**Table 1**: Multi-Property Steering for Protein Generation\n\n| Based Model|Method|Thermostability|Solubility |\n| --|--|--|-- |\n| ESM2|Original Model|56.77|0.327 |\n| ESM2|Activation Steering|75.29|0.409 |\n| ESM3|Original Model|60.70|0.312 |\n| ESM3|Activation Steering|67.82|0.449 |\n\n\n**Table 2**: Multi-Property Steering for Protein Optimization on Thermostability Medium Difficulty Task\n\n| Method|Thermostability|Solubility |\n| --|--|-- |\n| Before Optimization |59.78|0.299 |\n| ESM2 + ASPO|76.19|0.401 |\n| ESM3 + ASPO|76.64|0.332 |\n\n**Table 3**: Multi-Property Steering for Protein Optimization on Thermostability Hard Difficulty Task\n| Method|Thermostability|Solubility |\n| --|--|-- |\n| Before Optimization |46.38|0.237 |\n| ESM2 + ASPO|78.11|0.278 |\n| ESM3 + ASPO|74.99|0.311 |\n\n----\n\n**Q2: \"In the ablation studies, it's better to show the trend of both properties and basic protein generation qualities with respect to the ablated variables.\"**\n\nThank you for your valuable suggestion. We include trends for diversity, distance to the initial set, and distance tothe  high fitness set in the following tables.  and will update them in the corresponding figures. We will also reflect these trends in the revised figures to provide a clearer visualization of how the ablated variables impact both properties and basic protein generation qualities.\n\n**Table 4**: Sensitivity to number of samples for steering vectors extraction in protein thermostability optimization (Fig. 4(a))\n\n| number of samples|10|25| 50|100|250 |\n| --|--|--|--|-- |-- |\n| ESM2-Medium-Diverisity|6.21|7.69|7.86|7.71 |7.71 |\n| ESM2-Medium-Dist_init |6.15|6.75|7.49|7.709|8.15 |\n| ESM2-Medium-Dist_high |10.87|10.77|10.54|10.63|10.51 |\n| ESM2-Hard-Diverisity|4.82|6.04|5.97|6.09|6.13 |\n| ESM2-Hard-Dist_init |6.66|7.21|7.02|7.29|7.30 |\n| ESM2-Hard-Dist_high |10.21|10.44|9.91|8.825 |9.10 |\n| ESM3-Medium-Diverisity|6.96|6.96|6.92|6.94|6.94 |\n| ESM3-Medium-Dist_init |6.08|6.09|6.09|6.00|6.00 |\n| ESM3-Medium-Dist_high |10.53|10.31|10.08|9.71 |10.12 |\n| ESM3-Hard-Diverisity|6.93|6.95|6.91|6.92 |6.932 |\n| ESM3-Hard-Dist_init |7.71|7.60|7.60|7.57|7.68 |\n| ESM3-Hard-Dist_high |9.94|9.74|9.46|9.25|9.27 |\n\n\n**Table 5**: Sensitivity of $\\alpha$ in protein thermostability optimization (Fig. 6(a))\n\n| $\\alpha$|0.05|0.25|0.5|1|2|5|20 |\n| --|--|--|--|--|--|--|--| \n| ESM2-Medium-Diverisity|7.01|7.68 |7.70|7.71|7.29|7.79|7.74 |\n| ESM2-Medium-Dist_init |7.67|7.68|7.71|7.71|8.00|7.88|7.89 |\n| ESM2-Medium-Dist_high |10.43|10.45|10.52|10.63|11.42|11.46|11.39 |\n| ESM2-Hard-Diverisity|6.83|6.05|6.07|6.09|6.348|6.370|6.348 |\n| ESM2-Hard-Dist_init |6.43|7.68|7.71|7.29|7.37|7.36|7.36 |\n| ESM2-Hard-Dist_high |8.88|8.88|8.86|8.83|9.19|9.15|9.19 |\n| ESM3-Medium-Diverisity|7.01|7.25|7.14|6.94 |6.91|6.91 |6.91 |\n| ESM3-Medium-Dist_init |6.00|6.00|6.00|6.00|6.01|6.01|6.01 |\n| ESM3-Medium-Dist_high |9.66|9.63|9.66|9.71|10.06|10.06|10.06 |\n| ESM3-Hard-Diverisity|7.15|7.14|7.02|6.92 |6.83|6.83|6.83 |\n| ESM3-Hard-Dist_init |7.59|7.58|7.58|7.57|7.57|7.57|7.57 |\n| ESM3-Hard-Dist_high |9.38|9.32|9.28|9.25|9.84|9.84|9.84 |\n\n----\n\n**Q3: \"the worse fine-tuning results might be due to insufficient data and large numbers of learnable parameters. Do you use parameter-efficient fine-tuning, like LoRA?\"**\n\n- We appreciate your comment regarding parameter efficiency in fine-tuning. Indeed, we employ LoRA, specifically with a rank of 8, which we found optimal in our experiments after evaluating various ranks [2, 4, 8, 12, 16]. The rank of 8 outperformed others, with rank 4 being the next best. Our choice of hyperparameter alpha is 16.", "claims_and_evidence": "The key claims are well-supported by experimental evidence across multiple PLM architectures. Results show significant improvements in target properties while maintaining diversity and novelty compared to fine-tuning and other baselines.\n\nHowever, the worse fine-tuning results might be due to insufficient data and large numbers of learnable parameters. Do you use parameter-efficient fine-tuning, like LoRA?", "methods_and_evaluation_criteria": "The methodology is sound, with appropriate evaluation metrics (fitness, diversity, novelty, distance measures) and comprehensive comparison against established baselines on relevant protein engineering tasks.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The number of use cases is small though.\n\nIn the ablation studies, it's better to show the trend of both properties and basic protein generation qualities with respect to the ablated variables.", "relation_to_broader_scientific_literature": "It should be easy for the people in science community to adapt this method with a pre-trained large models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "The single objective design is quite limited. How do you deal with multi-property optimization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S66MEjhCHC", "reviewer": "Reviewer_ADRt", "summary": "This paper introduces a method for steering Protein Language Models (PLMs) towards outputs with desirable properties. It is based on a technique called 'Activation Steering' used for LLMs, where internal activation vectors of LLMs are modified to shift them towards desired behaviour. \n\nThe authors show how this method can be applied to both Autoencoder PLMs  like ESM2 and ESM3 as well as Autoregressive PLMs like ProLLAMA to produce proteins with higher ‘fitness’, i.e. higher values of desired properties like thermostability and solubility. In line with previous work in this field, oracle models are created by training CNN models based on relatively large datasets.\n\nThe proposed method shows performance improvements over existing methods, while maintaining diversity and novelty of generated proteins.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and positive feedback. We have provided a detailed explanation for your concerns as follows. Please feel free to let us know if you have any additional concerns or questions.\n\n**W2: Assumptions .. continuity, linearity...Theoretical grounding**\n\n- Our method is based on two hypotheses: the **linear representation hypothesis** and the **superposition hypothesis** (T. Adly 2024). The linear representation hypothesis suggests that neural networks encode meaningful concepts as directions in their activation spaces, while the superposition hypothesis extends this by suggesting that networks utilize almost-orthogonal directions in high-dimensional spaces to represent features, embodying properties of additivity and homogeneity. These hypotheses underpin the design of many existing algorithms.  \n>Ref: T. Adly. Scaling monosemanticity: Extracting interpretable features from Claude 3 sonnet. Anthropic, 2024.\n\n- However, the theoretical validation of these hypotheses is ongoing. As such, the theoretical grounding of our method remains a subject for future research.\n\n**M1: the assumption .. only true if dataset has large numbers of samples**\n\n- Our method assumes that the pretrained PLM has already encapsulated comprehensive knowledge of protein properties, given its pretraining on a massive number of both naturally occurring and synthetically designed protein sequences. This extensive pretraining dataset supports our assumption that the PLM possesses a universal understanding of various protein properties, making it suitable for steering the PLM to explore specific properties.\n\n- However, we acknowledge that if the PLM lacks prior knowledge of a target property, our method may not be effective, which is a limitation. \n\n- Regarding the reviewer's point on fine-tuning, it is important to clarify that FT also relies on the pretrained model having some foundational knowledge of the desired properties. Without this, FT on a small dataset risks overfitting and poor generalization, similar to the limitations faced by our proposed method.\n\n**M2: unsure whether just adding a vector**\n\n- We appreciate the reviewer's concern regarding the potential risks of modifying latent representations. Indeed, adding arbitrary vectors can disrupt model outputs, akin to adversarial attacks. However, our method carefully defines the steering vector to ensure meaningful modifications aligned with desired properties.\n\n- To illustrate, consider a simple case with a positive sample ($z_p$) and a negative sample ($z_n$). To shift $z_n$ towards $z_p$, the intuitive direction for modification is $v=z_p-z_n$. Extending this, if we have sets of positive and negative samples, the steering vector can be defined as the mean difference between these sets, aligning changes with the observed data distribution.\n\n- Our method assumes that the latent space adheres to the linear representation hypothesis and superposition hypothesis, allowing these mean-based modifications to navigate towards desired properties effectively. Empirical results confirm that this method in steering the latent representation leads to the desired enhancements.\n\n**M3: not strictly true that adding the steering vector .. move it towards the desired property**\n\n- The effectiveness of the proposed method relies on the linear representation hypothesis and superposition hypothesis. The thought experiment designed by the reviewer, where the latent space exhibits non-linear characteristics, poses a challenge to our assumptions. In this case, the proposed method does not work.\n\n**M4&W3: Concerns about the magnitude .. reflecting relatedness to the property**\n\n- Normalization techniques like LayerNorm and RMSNorm in transformers constrain vector magnitudes, ensuring they don’t scale infinitely. This addresses the counterexample.\n\n- Additionally, the magnitude is indicative of the importance of the token. Our relatedness score takes the magnitude of the token representations into consideration, following the attention mechanisms in transformers, which compute attention scores via dot products between queries and tokens.\n \n**M5&W4: maintain unrelated properties**  \n\n- We evaluated both thermostability and solubility to ensure our method maintains unrelated properties during optimization. Due to length limitations, we only present solubility experiments. As shown in the following tables, steering for solubility has minimal impact on the unrelated property thermostability.\n\nTab: Protein generation\n|Base Model |Method|sol(target)|therm(unrelated)|\n|-|-|-|-|\n| ProLLaMA | Original|.23|56|\n|ProLLaMA|FT|.24|57|\n|ProLLaMA|AS|.28|56|\n|ESM2|Original|.33|57|\n|ESM2|FT|.41|56|\n|ESM2|AS|.44|57|\n|ESM3|Original|.32|54|\n|ESM3|FT|.39|55|\n|ESM3|AS|.49|54|\n\nTab: Protein optimization\n||Medium|Medium|Hard|Hard|\n|-|-|-|-|-|\n||**sol**(target)|**therm**(unrelated)|**sol**|**therm**|\n| Before Opt|.28|54|.09|54|\n| AdaLead|.62|50|.53|51|\n| ESM2+ASPO|.51|53|.35|54|\n| ESM3+ASPO|.65|53|.4|53|", "claims_and_evidence": "1.\tThe paper claims to enable property-specific protein generation without the requirement for retraining models, thus providing a scalable alternative to other resource intensive techniques.\n2.\tThe paper introduces a new iterative optimization method which involves choosing which tokens to mutate based on an estimate of their individual fitness score.\n3.\tAn empirical evaluation is performed on models with 3 different architectures to show the universal effectiveness of the method and its performance in comparison to some existing methods in the field.\n\nClaim (1) is supported by empirical evidence, at least for the models included in the paper. Activation steering is indeed less resource intensive than directed evolution/reinforcement learning based methods as well as fine-tuning, though it is unclear if it so in comparison with other benchmarks used in the paper. \n\nSome of the technical claims made in the paper are unconvincing to me, such as “magnitude of a representation’s projection onto [the direction of the steering vector] can indicate it’s relatedness to the property”, which I have detailed in the methods/questions section. The method itself also makes certain assumptions which I don’t think are true. Again, I have detailed these in later sections.\n\nWith respect to claim (2) – the paper does introduce a new optimization procedure to choose the tokens most weakly associated with the property of interest and update the internal representations of those tokens. It is unclear to me if the method of determining if a token contributes positively or negatively to the fitness function is completely correct, which follows from the point made above about projection onto the steering vector. Moreover, it is not clear to me from a theoretical standpoint how simply adding a common steering vector to the layer representation of any token, no matter where it is in relation to the regions of positive and negative fitness values, would improve overall fitness of the protein. Again, I have elaborated on this in the question section.\nFor claim (3), empirical evidence is shown on 3 different architectures. With the lack of any theoretical discussion or explanation I’m unconvinced on why this method universally applicable to all PLMs. Is there a fundamental property of models on which this method is based? Latent spaces of different models behave differently – some may be amenable to post-training modifications while some may not. \n\nOverall, I think this paper may still be valuable since it details a method which empirically shows better results than existing methods for a real-world task with meaningful utility. However, there are multiple methodological leaps and assumptions which mean that it doesn’t convince me in terms of being an actual novel technical contribution.", "methods_and_evaluation_criteria": "Methods:\n\nI’m unsure about certain parts of the method, which I have detailed below:\n\n1.\tFirstly, the assumption that “PLMs inherently encapsulate intrinsic knowledge about specific protein properties” is only true if the training dataset has sufficiently large numbers of samples exhibiting the negative to positive range of values you want to generate at inference time. Without a meaningfully high number of samples, the model would not gain a semantic understanding of the required property and would not encode this in the structure of its latent space. This is not true for fine-tuning based approaches, where you can fine-tune on an additional dataset with relatively few samples and get desired behaviour. Thus, this method is an alternative to fine-tuning based methods only when the property of interest is already in the dataset the model is trained on.\n2.\tSecondly, I’m unsure whether just adding a vector to the latent representation of a deep learning model will always lead to meaningful results. For example, adding a random vector to the latent representation of the model is likely to lead to gibberish outputs, as demonstrated in many papers on adversarial attacks. What if adding the steering vector to the latent representation takes the model to an unseen region, i.e. a region which doesn’t exist in the training set? \n3.\tEven if you assume that the model is well behaved under the steering transformation, it is not strictly true that adding the steering vector to the latent representation of a particular sample will move it towards the desired property. Concepts like functional continuity, linearity, and other properties necessary for this to be true are not necessarily naturally developed by the model. As a simple thought experiment, assume that there is a spherical region in the middle of the latent space that corresponds to positive fitness. Then, in order to transform samples outside the positive space to lie within the positive space, I need to add vectors to all samples pointing inward towards the positive region, whereas the steering vector would always point in the same direction. You are assuming that a translational transform in the latent space of each layer can shift any token towards the desired property, which is not true.\n4.\t“the magnitude of a representation’s projection onto [the direction of the steering vector] can indicate its relatedness to the property”. This is not strictly true. For example, you can infinitely scale a vector and its projection onto the steering vector will keep increasing. That doesn’t necessarily mean it’s relatedness to the property will keep increasing.\n5.\tOne question I have with this method is how you maintain existing properties of proteins not necessarily related to the fitness property, especially for protein optimization. This is especially possible with the iterative optimization where you repeatedly replace the tokens least associated with the property to optimize for. For example, say I have a protein with all the required properties but solubility. In optimizing for solubility using this method, do I lose other properties? This is important for protein optimization because I would generally start with a protein with certain properties and try to optimize for others. If the optimized protein is unrecognizable from the protein I start from, then this is just a conditional protein generation method, not an optimization method.\n\n[Discuss all method drawbacks here]\n\nEvaluation:\n1.\tIn line with previous works, this paper uses a trained model as an oracle. This model is trained on thousands (~40000) labeled examples and shows medium to high accuracy/correlation on a test set. While this is not optimal, it is in line with previous work in the area published at similar venues. In the absence of actual ground-truth values for the propertied of interest, this evaluation criteria, though not perfect, is the best that is possible. \n2. Apart from fitness, evaluation criteria, such as diversity and novelty are well-chosen to assess the effectiveness of the generated sequences. One thing that is missing is, as mentioned, maintenance of existing properties apart from the property to optimize, as mentioned previously.\n2.\tI’m unclear on the Dist_init and Dist_high metrics – are large or small values of these metrics favourable? I’m not sure how these metrics contribute to telling us about the quality of the output. There’s also no clear pattern in Tables 2, 3, 4 that tells me which models are better.", "theoretical_claims": "No major theoretical claims are made, the paper is mostly based on empirical results. It would be good to see some theory on major claims, such as whether adding steering vectors to intermediate layers always moves the output closer to the high fitness region, and whether this happens in a linear or smooth manner from the initial point to the final point as you perturb the activations more and more.", "experimental_designs_or_analyses": "Experiments are well designed to support the claims of the paper. Results are shown on major PLM structures (though only 3) and for three different properties.", "relation_to_broader_scientific_literature": "The paper is in line with previous work on protein fitness optimization.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses a topic of importance in protein engineering. \n2. The method is conceptually simple and seemingly shows good results despite its simplicity.\n3. The method is demonstrated on both AE-PLMs and AR-PLMs, implying some sort of generalizability. However, this is not surprising given that the method is based on a fundamental concept of semantic organization in neural network latent spaces, which could be counted as a strength of the method.\n\nWeaknesses:\n\nI have outlines multiple weaknesses related to the method in the 'Methods and Evaluation' section. I will briefly summarize them here:\n\n1. Optimizing for properties which are not reflected in the original training set. This is only a criticism because the method is compared to fine-tuning, which can be used to generate proteins with properties not in the original set using a small, new dataset.\n2. Assumptions about the steering transformation being a coherent transformation path in the feature space of the model, in terms of semantic continuity of the space, linearity, etc. Theoretical grounding would be useful here.\n3. Concerns about the magnitude of the projection of a vector reflecting it's relatedness to the property. Counterexample given in the 'Methods and Evaluation' section (point 4).\n4. Questions about maintaining existing properties unrelated to the property to optimize for.\n5. Model evaluation is based on oracle classification models without actual ground truth. However, as I mentioned, there's no real way to get around this short of actual experimental validation.", "other_comments_or_suggestions": "I will rate this paper as a 'weak accept' but I think certain issues need to be addressed, mainly point 2, 3 listed in the 'Methods and Evaluation' section above.", "questions_for_authors": "Most of my questions are listed in the sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gijakwnkrF", "reviewer": "Reviewer_Vbjw", "summary": "This paper presents a method to control the output of protein language models, inspired by the Activation Steering approach in the LLM domain, allowing the generated sequences to exhibit a given property. This method does not require retraining the protein language model and can be directly applied during the inference stage. The paper validates the effectiveness of this approach by manipulating properties such as thermostability and solubility.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point. Please kindly let us know whether you have any further concerns.\n\n----\n\n**Q1**: \"the evaluation may exhibit a certain level of **circular dependency**. ... This could potentially be considered an attack on ESM-2. In this part of the experiment (Table 1), we also observe that ESM-2 shows the best performance. \" \n\nWe thank the reviewer for raising the important issue of potential circular dependency in our evaluation. Using ESM-2 both as the base generative model and as part of the downstream predictor could potentially introduce biases. However, we emphasize the following points to address this concern:\n\n- While using ESM-2 as a base model achieves superior performance in Table 1 to other base models, by comparing the generation performance with Action Steering and without Activation Steering (Original Model), we can we can observe a significant performance gain. This validates that improvements are attributable to the steering mechanism rather than inherent biases in ESM-2.\n\n- Besides, our proposed Activation Steering method demonstrates consistent and significant improvements over both Fine-tuning and the Original Model across all three base models (ESM-2, ESM-3, and ProLLaMA). This cross-model consistency suggests the improvements stem from the method itself rather than inherent biases in ESM-2.\n\n-----\n\n**Q2**: \" It is necessary for the paper to include metrics that assess the authenticity of the generated sequences, such as pLDDT and sequence likelihood.\"\n\nTo address the concern about assessing the authenticity of generated sequences, we incorporate the pLDDT metric, evaluated using ESMFold, to measure the structural integrity of both initial and optimized protein sequences. The results, presented in the tables below, demonstrate that the pLDDTs remain consistent before and after applying activation steering. This consistency confirms that our method effectively guides protein generation towards desired functionalities while maintaining the authenticity of the sequences\n\n**Table 1**: pLDDT for protein generation.\n\n| Base Model | Method | Thermostability | Solubility |\n| -- | -- | -- | -- |\n| ProLLaMA | Original Model | 40.02 | 40.02 |\n| ProLLaMA | Fine-tuning | 37.79 | 37.68 |\n| ProLLaMA | Activation Steering | 39.51 | 39.52|\n| ESM2 | Original Model | 74.09 | 69.17 |\n| ESM2 | Fine-tuning | 72.62 | 68.16 |\n| ESM2 | Activation Steering | 73.02 | 68.36 |\n| ESM3 | Original Model | 76.14 | 73.54 |\n| ESM3 | Fine-tuning | 75.50 | 72.71 |\n| ESM3 | Activation Steering | 76.18 | 72.61 |\n\n**Table 2** pLDDT for protein optimization regarding thermostability.\n\n| | Medium difficulty | Hard difficulty |\n| -- | -- | -- |\n| Before Optimization | 79.75 | 79.99 |\n| AdaLead | 51.06 | 31.15 |\n| ESM2 + ASPO | 76.75 | 79.59 |\n| ESM3 + ASPO | 81.00 | 78.91 |\n\n**Table 3** pLDDT for protein optimization regarding solubility.\n\n| | Medium difficulty | Hard difficulty |\n| -- | -- | -- |\n| Before Optimization | 77.92 | 75.94 |\n| AdaLead | 36.87 | 35.89 |\n| ESM2 + ASPO | 78.38 | 77.13 |\n| ESM3 + ASPO | 78.15 | 77.98 |\n\n----\n\n**Q3:** \"I am curious whether the paper's method performs well for modeling more **niche properties**... as it fundamentally relies on the internal knowledge of PLM.\"\n\n- We appreciate the reviewer's interest in the applicability of our method to more niche properties. We would first note that the challenge with niche properties is the scarcity of labeled data, which can hinder the training of effective predictors to estimate the performance. \n\n- We conducted an additional experiment focusing on the niche property of hydrolysis activity at 30°C for polyethylene terephthalate (PET). We utilized the dataset from (Seo et al. 2025), comprising 184 annotated samples. \n    - To estimate the performance, we train a predictor on this dataset, achieving a Pearson correlation coefficient (R) of 0.57 in 5-fold cross-validation, indicating a reasonable prediction capability under data constraints.\n    - We design optimization tasks of varying difficulty:\n        - Medium Difficulty Task: Optimizing 69 samples with an initial average predicted activity of 1.61.\n        - Hard Difficulty Task: Optimizing 67 samples with an initial average predicted activity of 175.62.\n- The results post-optimization using our ASPO method combined with ESM3 are as follows:\n\n|Task Difficulty|Initial Average Activity|Post-Optimization Average Activity|\n|-|-|-|\n| Medium | 1.61 | 5.00 |\n| Hard | 175.62 | 236.91 |\n\n- These results demonstrate that our ASPO method can effectively optimize even niche protein properties, such as hydrolysis activity in PET, underlining the versatility and potential of our method in broader protein design applications.\n\n> Reference: Seo, Hogyun, et al. Landscape profiling of PET depolymerases using a natural sequence cluster framework. Science 2025.", "claims_and_evidence": "The paper experimentally demonstrates that the proposed method can generate sequences with improved performance on a given property, supporting its claim.", "methods_and_evaluation_criteria": "The method proposed in this paper is reasonable.\n\nHowever, I believe the evaluation criteria used in this paper have potential issues. During the generation process of ESM-2, the model's behavior (such as activation steering or certain activation parameters) directly depends on the structure and inference process of ESM-2, and the generated protein is then evaluated for properties such as thermostability using ESM-2. If the activation of the model is guided or modified during the generation process, then the evaluation may exhibit a certain level of circular dependency with the generation process, potentially causing the generated protein to perform exceptionally well in the ESM-2 evaluation, because the model has already been adjusted during the generation. This could potentially be considered an attack on ESM-2. In this part of the experiment (Table 1), we also observe that ESM-2 shows the best performance. Therefore, the paper needs to clarify whether this good performance is due to such an attack.", "theoretical_claims": "This paper is more \"application-oriented,\" and therefore does not include many theoretical claims.", "experimental_designs_or_analyses": "I have reviewed the experimental design section of the paper, and I believe more testing metrics need to be included. The paper primarily tests how well the generated sequences from the protein language model align with the target properties and measures diversity and novelty, but I believe an important metric for evaluating PLM-generated results is testing the authenticity of these sequences, which the authors have overlooked. It is necessary for the paper to include metrics that assess the authenticity of the generated sequences, such as pLDDT and sequence likelihood.", "relation_to_broader_scientific_literature": "The paper proposes a more novel method for controlling PLM output, which is insightful for protein design.", "essential_references_not_discussed": "I did not find any important missing references.", "other_strengths_and_weaknesses": "I am curious whether the paper's method performs well for modeling more niche properties. The model's performance was primarily validated in terms of thermostability, solubility, and fluorescence brightness, but these are already well-studied properties, and many protein sequences with these properties are included in the PLM training data. However, for protein design tasks involving more niche properties, I am uncertain whether this method would still perform well, as it fundamentally relies on the internal knowledge of PLM.", "other_comments_or_suggestions": "I don't have other comments.", "questions_for_authors": "I don't have other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper adapts activation steering from LLMs to PLMs for protein sequence generation and optimization, showcasing a training-free method and the ASPO framework with experimental improvements over baselines. Reviewers initially had concerns like lack of certain experiment details, unclear method descriptions, theoretical assumptions, and issues with evaluation criteria. The authors responded with additional experiments (e.g., full sequence generation for ESM3, niche property experiments), clarifications (such as method steps and related work like PPLM), and evidence to address these concerns, like using LoRA for fine-tuning, showing multi-property optimization solutions, and mitigating risks of latent representation modifications. Reviewers reassessed positively, with some raising their scores. Considering the research's significance, novelty of the approach, and the authors' effective responses to feedback, the paper is recommended for acceptance, provided the authors incorporate all relevant discussions and clarifications in the final version.
Decision: Accept (poster)