From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models
Authors: Etowah Adams
Liam Bai
Minji Lee
Yiyang Yu
Mohammed AlQuraishi
Venue: ICML 2025 spotlightposter (2025)
Keywords: protein language model
sparse autoencoder
mechanistic interpretability
scientific discovery
Primary area: applications->health_medicine
TL;DR: We train and evaluate SAEs to identify interpretable features in pLMs and show their potential for scientific discovery.

Abstract: Protein language models (pLMs) are powerful predictors of protein structure and function, learning through unsupervised training on millions of protein sequences. pLMs are thought to capture common motifs in protein sequences, but the specifics of pLM features are not well understood. Identifying these features would not only shed light on how pLMs work, but potentially uncover novel protein biology––studying the model to study the biology. Motivated by this, we train sparse autoencoders (SAEs) on the residual stream of a pLM, ESM-2. By characterizing SAE features, we determine that pLMs use a combination of generic features and family-specific features to represent a protein. In addition, we demonstrate how known sequence determinants of properties such as thermostability and subcellular localization can be identified by linear probing of SAE features. For predictive features without known functional associations, we hypothesize their role in unknown mechanisms and provide visualization tools to aid their interpretation. Our study gives a better understanding of the limitations of pLMs, and demonstrates how SAE features can be used to help generate hypotheses for biological mechanisms. We release our code, model weights, and feature visualizer.

From Mechanistic Interpretabilityy to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models
Etowah Adams *1
Liam Bai
Minji Lee
Yiyang Yu?
Mohammed AlQuraishi
Abstract
teins useful for downstream tasks including protein structure (Lin et al., 2023) and function (Rao et al., 2019; Notin et al., 2022) prediction. For this reason, pLMs have become a growing component of the protein biologist's toolkit.
Protein language models (pLMs) are powerful pre- dictors of protein structure and function, learning through unsupervised training millions of pro- tein sequences. pL.Ms are thought capture com- mon motif's in protein sequences, but the specifics of pLM features are not well understood. Identi- fying these features would not only shed light on how ML.Ms work, but potentially uncover novel protein biology-studying the model to study the biology. Motivated by this, train sparse au- toencoders (SARs) on the residual stream of a PLM, ESM-2. By characterizing SAE features, we determine that PLMs use a combination of generic features and family-specific features to represent protein. addition, we demonstrate how known sequence determinants of properties such as thermostability and subcellular localiza- tion can be identified by linear probing of SAE features. For predictive features without known functional associations, we hypothesize their role in unknown mechanisms and provide visualiza- tion tools to aid their interpretation. Our study gives a better understanding of the limitations of Ms,,, and demonstrates how SAE features be used to help generate hypotheses for biological mechanisms. We release our code, model weights and feature visualizer.
As these models are widely adopted, a number of studies have investigated their inner workings to understand their limitations. Previous work suggests pLMss do not learn the biophysics of proteins, but rather store common sequence motifs and contacts (Zhang et al., 2024). For many down- stream tasks, pI.M performance is driven by features learned in early layers and does not scale well with pre-training (Li et al., 2024). Yet exactly what pLMs have learned about proteins remains unknown.
Recent work in mechanistic interpretability points to path forward. Sparse autoencoders (SAEs) have successfully extracted interpretable features from large language models (LLMs) like GPT-4 and Claude (Gao et al., 2024; Temple- ton et al., 2024). SAEs decompose model activations into sparse, high-dimensional representation where individ- ual latent dimensions often have interpretable activation patterns. These features provide window into the inner workings of these complex models. Training SAFs on pl ML can serve an additional purpose. Unlike natural languages, which we intuitively understand, the language of proteins is more cryptic. SAE features from pLMs could potentially unveil patterns protein sequenees that we have not yet discovered.
In this work, we train SAEs the residual stream of ESM- 2, widely used pLM (Lin al., 2022) (Figure la). To interpret the SAE latents, we develop InterProt, a tool that visualizes latent activations on protein sequences and struc- tures. We find that some features are highly interpretable, corresponding to a range of concepts from secondary struc- ture elements to entire domains, as reported contempora- neously by Simon & Zou (2024). We conduct human evaluation study and find that SAE latents are consistently rated as interpretable, in contrast to the ESM baseline. Fur- thermore, we find that many features activate highly only on specific protein families, suggesting that pL.s rely on internal representations of protein families to perform their pre- pre-training task. We introduce a method to categorize SAE latents based on their family specificity and activation pat-
1. Introduction
Protein language models (pLMs) are language models trained on large datasets of protein sequences. In the pro- cess of minimizing loss on their pre-training mask-usually masked token prediction-they learn representations of pro-
Equal contribution 'Department of Systems Biology, Columbia University Ginkko Bioworks, Boston. Correspondence to: Etowah Adams <ca2901 @ @cllumbia.du>>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada, PMLR 267, 2025. Copyright 2025 by the author(s).

---

Interpreting Features in Protcin Language Models using Sparse Autoencoders
Training SAES
Interpreting SAE latents
Interpretable probing
Acticctes on mitochanriiil transit peptides
MSNGEE....
Analyze sequenoes structures using
Activattds hel
Läyer
Layer 2
MSKGEE...
" fils)) Prediction: Mitschendrion
Layer
Activetes on lylnee
EDELSTA nuees Beeese/eeeeeeeeeeeeeeeee souteeg
Purtate:
Faln
tolven-kesel representation
Sparse representation
Sparse representation for taken
Catagoriz by activation pattern and famiy specifiity
Pooled SAE activatione
Linear probe coefficients
ESM2
Figure 1. Overview of our paper. We train SAEs on the output of ESM intermediate layers. b. We interpret SAE latents using our latent visualizer InterProt and categorize features based their activation pattern and family specificity. c. By interpreting the weights of linear models trained on SAE latents, we show how features which correspond to known sequence determinants can be identified.
tern (Figure lb).
pLMs seem to use notion of sequence homology and recurring protein motifs. Using influence functions, Gordon et al. (2024) finds that pLM-derived sequence likelihoods are driven by homologous protein training data. Zhang et al. (2024) propose that pLMs store common motif's in proteins and the pairwise contacts between them. They arrive at this proposition by analyzing what residue segments must be unmasked recover a sequence contact (via the "categorical Jacobian").
Having developed framework for understanding features, we explore the impact of SAE hyperparameters and ESM layers. We find that increasing the sparsity imposed on SAE latents increases the number of features that are family- specific as does increasing the expansion factor. The number of family-specific features also varies substantially by ESM layer, with middle layers containing the most family-specific features.
Many studies have explored how MLMs work mechanisti- cally as well. Analyses of attention matrices in pLMs have been shown to resemble pairwise contact maps (Vig et al., 2021; Rao et al., 2021; Lin et al., 2023), and contain infor- mation such as binding sites (Vig et al., 2021) and allosteric sites (Kannan et al., 2024; Dong et al., 2024). They have been comprehensively mapped to Gene Ontology terms (Chen al., 2025).
With the goal of mapping SAE features to important protein properties, we train linear probes on SAE features and an- alyze the most predictive features. Our method identifies features corresponding to known sequence determinants for properties such as thermostability and subcellular localiza- tion. These results demonstrate that SAEs can help not only understand pL.ss, but also the data which they are trained. By making pL. representations interpretable, we envision that SAE features may help generate hypotheses for unknown biological mechanisms (Figure Ic).
Concurrent to our work, Simon & Zou (2024) trained SAEs on ESM-2 and presented methods to analyze their latents: visualizing them, evaluating them against Swiss-Prot anno- tations, and interpreting them with language model. Our work builds on this foundation with a different SAE archi- tecture and evaluation methods that focus coevolution, human preference, and downstream tasks (subsection A.3).
2. Related Works
2.1. Interpreting protein language models
Previous evaluations of pLM representations downstream tasks provide evidence of features that pLMs may be learn- ing (Rao et al., 2019; Dallago et al., 2021; Detlefsen et al., 2022; Li et al., 2024). In particular, Li et al. 2024 performs a comprehensive analysis of ESM performance on a range of downstream tasks across different model sizes and layers. Of the tasks they tested, they find that only structure- related prediction tasks scales with model size, and that most tasks "low-level features" learned early in pre-training. A key limitation of using downstream tasks to understand the fea- tures learned by pLMs is that features cannot be discovered without supervision. In contrast, SAEs can uncover features in more unsupervised manner.
2.2. Dictionary learning for feature extraction from models trained on biological data
Following the success of extracting interpretable features from LLMs, a number of works have applied sparse dic- tionary learning techniques to models trained on scientific data. Donhauser et al. (2024) evaluates the possibility of extracting features from models trained on microscopy data. Interpretable features have also been uncovered from sin- gle cell foundation models (Schuster, 2024; Adam Green, 2024). Associating the extracted features with functional labels via probing, however, has not yet been explored in
2

---

Interpreting Features in Protcin Language Models using Sparse Autoencoders
Festures related to secondary structure
Features specific to conserved motils and domains
Features relsted biochemical patterns in specific structural contexts
Inward facing residues in porins
Signal peptides
Every seventh residue in alpha helices
Cysteines disulfide bridges in antibody constant regions
Disordered regions
Halicase domains
Figure 2. Examples of SAE features. We find features related to secondary structure (a), conserved motif's and domains (b), and biochemical patterns in specific structural contexts (c). structures are colored according to activation (red: activation, blue: no activation).
this context.
them from the underlying conceptual features the model. This is in contrast to previous works which the word feature to refer to both (Bricken et al., 2024).
3. Background and Methods
3.1. Sparse autoencoders (SAEs)
3.2. Downstream tasks
An SAE is an autoencoder designed to learn efficient and meaningful representations by enforcing sparsity constraints on encoder output. We use SAEs with TopK activation to en- (2024). This activa-
Secondary structure. The secondary structure dataset from TAPE (Rao et al., 2019) contains residue-level labels in 3 classes: alpha helix, beta strand, other. To prevent data leakage, sequences are divided into training and test sets based on a threshold of 25% sequence identity.
force sparsity, as introduced in Gao tion function only allows the & largest latents to non-zero We choose TopK SAEs due to their improved reconstruction at given level of sparsity compared to other techniques, and because it allows us to directly set the LO-norm of the en- coding using k. We note that improved reconstruction may come at the cost of increased feature absorption (Karvonen al., 2024).
Subcellular localization. Subcellular localization a protein- label for where protein is localized within cell. The dataset from Almagro Armenteros et al. (2017) consists of UniProt-sourced eukaryotic proteins, with strin- gent homology-basee based train-test splits.
Thermostahility. The thermostability dataset from the Melteome Atlas (Jarzab et al., 2020) measures the melt- ing temperature of 48,000 proteins across 13 species. We use the train-test split from FLIP (Dallago al., 2021) con- taining all sequences clustered at 25% sequence identity.
The encoder and decoder are defined as: TopK (Wene (x - bpre))
= + bpre
where Wone projects the residual stream into the SAE latent space and Waee the reverse. TopK zeros all latents that are not in the top kt The loss is simply the reconstruction mean squared error L = ı .11 We train our TopK SAEs on million random sequences under 1022 residues from UniRet50 (Suzek et al., 2007).
Mammalian cell expression. The dataset curated by Mas- son et al. (2022) contains expression data of 2,165 proteins from the human secretome Chinese hamster ovary (CHO) cells, commonly used mammalian hosts for therapeutic pro- tein production. We use this dataset as a binary classification task determine whether protein can be successfully ex- pressed in CHO cells. We split proteins into training and
Following Lieberum et al. (2024), we refer to the hidden dimensions of the SAE "latents" clearly distinguish
3

---

Interpreting Features in Protcin Language Models using Sparse Autoencoders
C
Farrly specilic fealure: beter laclemeee
Non-family specilic festure: disordered
4000
dead latem:
Number of sequencces
10"
Not bele-lactamese family
In beta nata-macttmase family
Number of sequences
3000
2500
other
point
not enough data
2000
pertodi:
short moth (1-20)
1500
med motit (70-50)
long mont (SC-300)
10
whole
500
b
47
Layer
0.0
0.4
0.6
1.0
0.6
1253
Normalized activation
Normalizee
752
hamby speciiic
Contral halbx other enzymes
Central halix in beto- factamase
Low disurder
High disorder
0-
Layst
Figure 3. a. Example of a family specific feature, the central helix of a beta lactamase. The histogram shows the distribution of maximum activations the sequences which activate the latent. b. Example of non-family specific feature, corresponding disordered sequences. Latents classified by activation pattern across layers. d. Number of family-specific latents across layers. The SAEs used c and d have hidden dimension of 4096 and k=64.
test sets based on a threshold of 40% sequence identity.
To interpret a given SAE latent dimension (i.e., a latent), we examine the top activating protein sequences, i.e., those that produce the largest activation for that dimension. We devel- oped latent visualizer, InterProt, to streamline the process of identifying features. InterProt allows users to align the most-acttvating sequences and highlight their corresponding protein structure by activation strength. It is open source and can be accessed at ht https: /iinteerrrtrr.t// com.
3.3. Linear probes
For each task and for varying ESM layers, we train linear probes on both ESM and SAE embeddings. We mean-pool embeddings over the length the sequence for tasks with protein-level labels: subcellular localization, thermostabil- ity, and mammalian cell expression. For all linear regression and logistic regression tasks, we perform grid search over range of regularization strengths and pick the best model using validation set. See Table I for summary of the probe implementation for each task.
Through these visualizations, we gain insight into how ESM internally represents protein sequences (Figure 2). Many discovered features correspond recognizable biological concepts, such secondary structure elements (e.g., alpha helices and beta strands), short conserved motifs, or entire functional domains. Some features capture biochemical concepts like cysteine-cysteine bonds or the orientation of residue side chains. In many ways, latents capture charac- teristies of a protein that biologist might notice.
3.4. Manual feature interpretability study
Following Rajamanoharan et al. (2024), we conducted blinded study in which seven participants (graduate students and undergraduates familiar with protein biology) rated the interpretability of SAE and ESM latents from layer 24. Embeddings were computed on sequences from Swiss-Prot clustered 30% sequence identity (see subsection A.6). Using our InterProt visualizer, each participant viewed 100 randomly selected features (drawn uniformly from both models) and rated their interpretability as yes, no, maybe.
Notably, while many latents intuitive (e.g., focused conserved motifs), others are unexpectedly specific. For instance, rather than a single, generic "alpha helix" latent, we observe multiple helix-related latents that activate under more context-dependen conditions (e.g., at the beginning of a helix, only when the helix is buried, or when the helix meets a certain length criterion). Other examples include highly context specific amino acids, such asparagine residues in disordered regions.
4. Feature Analysis
4.1. SAES uncover interpretable features
To systematically evaluate interpretability. we conducted a blinded human study to rate SAE and ESM latents (subsec- tion 3.4). SAE latents were rated as interpretable far more
We train TopK SAEs on the residual stream of different lay- of ESM-2 (650M) to explore its learned representations.

---

Interpreting Features in Protcin Language Models using Sparse Autoencoders
133
SAE
ESM
Activation Pattern. Latents exhibit distinct activation pat- terns across their top sequences (Figure 1b). To capture these differences, we use a rule-based scheme that classifies features into categories such as: point (activates on single residues at a time), periodic (activates a regular inter- val, repeating every n residues), motif (activates in short, medium, or long contiguous intervals), and domain (acti- vates over nearly the entire sequence). Refer Table 2 for precise definitions.
Feequanry
0/
MO
MAYBE
YES
Flumnni essessmenn of whether feeture is interpretable
Family Specificity. To quantify family specificity, we use each latent for binary classification: given a protein, can this latent's activations be used to predict membership in a particular family? We label a latent "family-specific" if it achieves an Fl score >> 0.7 a certain activation threshold. We use all protein sequences from Swiss-Prot (Boeckmann et al., 2003) (clustered as 30% sequence identity) and cate- gorized into InterPro protein families (Paysan-Lafosse et al., 2023) for our evaluation. Refer to subsection A.6 for details.
Figure 4. Human rater scores of SAE ESM latents from layer 24. The error bars indicate the 95% confidence interval, assuming binomial 1-vs-all distribution.
often than those from ESM (Figure 4). Approximately 80% of SAE features received "yes" rating for interpretability, broadly consistent with previous findings on SAEs trained on language model activations (Rajamanoharan et al., 2024).
These classifications allow us to systematically group SAE features and draw broader conclusions about their roles in representing protein structure and function.
4.2. SAE latents reveal a learned notion of protein families
Intriguingly, large subset of latents appear to be protein samily-sp These latents activate weakly on most se- quences but strongly on proteins belonging to a particular family. For example, the latent shown in Figure 3a ap- pears highly activates only for the central helix in beta- lactamases, but only exhibits marginal activations when similar structural motif appears in proteins outside the beta-lactamase family.
4.4. Influence of SAE training choices on latent classification
Prior literature on L.LMs suggests that SAE-learned features depend on multiple factors, including model hyperparame- ters (e.g., the sparsity level and latent dimension size) and training data (e.g., which layer's activations are used). We therefore examine how these factors impact the types of features that emerge in PLMs.
By contrast, a non-family-speciff latent such as the "dis- ordered" latent in Figure 3b, responds broadly to intrin- sically disordered regions in proteins across many fami- lies. These observations collectively suggest that pLMs learn internal representations that, in part, mirror known eqquence-homologe based groupings of proteins, akin model-internal notion of protein families.
Effect of k. In TopK SAEs, the l hyperparameter controls how many latents can be active for a given input, effectively setting the sparsity level. We find that, for fixed hidden dimension, lower & (i.e., higher sparsity) yields more family- specific features (Figure 8a). One plausible explanation is that family-specific features constitute the most salient signals used in reconstruction.
To better understand the relationship between a family- specific latent and its associated protein family, we investi- gate what happens when we "steer" ESM using the latent. Specifically, we set the activation of family-speeific latent multiple of its maximum activation, and continue the model's is forward pass using the reconstructed activation. We find that, compared to steering random latents which acti- vate on sequence, steering family specific latents causes fewer residues to change (Figure 7). Investigating how to steer the model in useful ways remains an open direction for future work.
Effect of expansion factor. We also experiment with in- creasing the latent dimension, called the expansion factor. While initially hypothesized that larger latent dimen- sion might shift the distribution of feature activation patterns, we observed that activation pattern classifications (point, mo- tif, domain, etc.) remain largely consistent, while increasing the number of family specific features (Figure 8b).
Comparison across layers. Finally, we investigate layer- wise differences in the learned features. Previous studies have shown that downstream performance can vary substan- tially depending on which layer representations are used, suggesting that different layers capture different degrees of abstraction. In our experiments, family-specific features tend to peak in certain early-to sarly-to-mid layers of ESM, then
4.3. Classifying latents
Motivated by these observations, we propose two classifica- tion schemes for the SAE latents.
5

---

Interpreting Features in Protcin Language Models using Sparse Autoencoders
secondary structure, subcellular localization, thermostabil-
Seccedary Soumiee
Suocchd kcol cation
ity, as well as novel task, mammalian cell line expression prediction.
1.50
597-
6.75
We find that linear probes on SAEs achieve performance similar to their ESM baselines across all layers (Figure 5). For secondary structure prediction, the SAE probe consis- tently outperforms the ESM probe, a result we discuss in the next section and attribute to the abundance of SAE latents that correspond to secondary structure.
6.50
0.10
1.9
6.30
132
3.5
1.0
5.2. SAE probes uncover interpretable latents corresponding to known mechanisms
1.50
2.70
1.50
To assess the interpretability of linear probes trained on SAE latents, we analyze the distribution of their coefficients (Figure 10) and manually inspect the latents with the largest coefficients in the InterProt visualizer I We find that those latents often correspond to biological concepts with known relevance to each task (Figure 6). 2
12
35
20
2.
20
20
25
32
1,8090
1,800
Figure 5. Linear probe performance on ESM vs. SAE embeddings across ESM layers. In all 4 downstream tasks, SAE performs competitively with ESM For secondary structure prediction, SAE consistently outperforms ESM in most layers. All classifications tasks are measured using accuracy; thermostability is measured using Spearman's 's rank correlation.
5.2.1. SECONDARY STRUCTURE
Secondary structure prediction is task that is known to utilize the features learned during pLM pre-training (Li et al., 2024). Our analysis of the highest-weighted SAE latents in our secondary structure probe confirms this and provides an explanation for why early layers perform worse.
decline in later layers (Figure 3d). Similarly, features which activate contiguously over long sections (motif/domain) are more common in earlier layers, whereas later layers exhibit features with shorter contiguous activations (Figure 3c). Given that later layers are thought to specialize for final logit computation (Gao et al., 2024), one possibility is that shorter, more specific activations better serve this goal.
Our SAE probes reveal that most layers contain features corresponding to secondary structure elements. The most positive coefficients for the alpha helix class are typically latents that activate exclusively on helices (Figure 6a). They range from specific helix residues, to entire helices, to helix- helix interactions. We observe similar results for the two other classes: beta strand and other, where other tends to correlate with features that activate on disordered regions.
5. Interpretable Probing
Representations learned by PLMs can used in down- stream tasks. Often, this as simple as training a linear model on the pLM embeddings. Although this leads to predictive model, it does so at the expense of interpretability, as the coefficients of the linear model do not correspond to interpretable features. In this section, we explore the ability of SAE representations to overcome this limitation. Since SAE latents are often interpretable, those that contribute significantly to prediction could reveal biological reasons that explain the prediction.
Why is classification performance worse in early layers? We find that early layers do not contain generic features for secondary structure. For example, the top latent for helix classification in layer 4 (L4/2624) detects proline (P) residues, which often appear at helix boundaries due to their disruptive effect on hydrogen bonding. This reliance on a weak amino acid-level correlation highlights the absence of more generalized helix feature. Similarly, the top latent for the disordered classifier activates on N-terminal residues, reflecting the spurious correlation that many proteins have disordered regions at their N-terminus, rather than capturing
5.1. SAE probes perform competitively with ESM probes
In the context of LLMs, linear probes on SAE embeddings can achieve performance similar to, or in some cases better than, linear probes on their base model embeddings (Bricken et al., 2024; Kantamneni et al., 2024). To explore this dy- namic in pLMs, we train linear probes on ESM and SAE em- beddings across different layers. Following Li et al. (2024), we evaluate the probes on benchmarked downstream tasks:
'Linear probe coefficients are not comparable across tasks because different tasks may apply different regularization strengths, optimized via grid search.
"In this section and in Figure 6, we discuss latents from range of layers. The choice of layers is quriiativve-rimmailll based on latents we find particularly interpretable or notable- and does not imply the lack of interesting latents in undiscussed layers.
6

---

Interpreting Features in Protcin Language Models using Sparse Autoencoders
Top alpha
layer
L16/2611: 1: helix surface resiclus
localization sig signals (NLS)
L16/1895: 3- 3-halix matif
L28/2267: key reeidues in bipartite NLS
L16/1699: interactions between perallal helices
L28/3396: KR rich motifs in
disordered regions
L16/991: buried middle resiclus in some helicee
L16/2875: adjacent G, helix resicuees
L28/383: key bipartite NLS
33
SERKSHOCISOL BETLLNS
b
Most sigrificant thermostabillly latents in leyer 6
Top extracellular latents layer 28
L0/2064: hydrophobic core residuee
Signol pose
L28/1541: signal peplide cleavage site
LB/3632: specitic helx residue (usually
L20/1470: signal peptide
L28/158B: signal poprida
amino
bela strands
Hembrane
L28/2111: extracelluarr residucenext membranc
L0/3262: unidentified
LB/2553: acids
Cycoplasmic rogion
proteins
Figure 6. Linear probes identify predictive SAE latents for downstream tasks. a. The top alpha helix latents from the sccondary structure prediction task. These latents activate mostly exclusively on alpha helices (SAE trained on ESM layer 16). b. The latents corresponding to most positive latent activates on residues
the most positive 3 most negative coefficients from linear probe on thermostability. in the hydrophohic core, which is structurally important for thermostability: the most negative latent activates glutamine (Q) amino acids, whose absence is correlated with increased thermostabilitty (SAE trained ESM layer 8). e-d. Top nucleus and extracellular latents from the subcellular localization prediction task. The top nucleus latents activate on bipartite nuclear localization signals (NLS), which contain a variable-lenge linker and generally follow the pattern R/K(X)) 10-10 KRXK. The top extracellular latent activates on signal peptide cleavage sites, while another top latent activates on the extracellular regions adjacent to the membrane (SAE trained on ESM layer 28).
fundamental structural properties.
variants still remain unknown, our SAE-based approach has the potential to become novel method to aid their discovery and characterization.
5.2.2. SUBCELLULAR LOCALIZATION
Transport mechanisms involving sequence motifs such as transit peptides and localization signals determine the loca- tion of proteins within the cell. We find that linear probes trained on SAE latents can uncover some of these known mechanisms.
Extracellular localization. The most positive coefficients for the extracellular class relate signal peptides (Fig- ure 6d). Signal peptides short sequences the N- terminus that trigger the secretory pathway transport proteins outside the cell. L28/1541 activates the signal peptide cleavage site while L28/1470 and L28/1555 activate across the peptide. The top latents also include a latent that activates on the extracellular portion of membrane-bound proteins (L28/2111) 1), demonstrating precise recognition of transmembrane regions, as well as latent (L28/2472) that only activates cuticle proteins, extracellular proteins that support the exoskeleton of insects.
Nuclear localization. The most highly weighted latents for nuclear localization often correspond to nuclear local- ization signals (NLS). For example, the top latents in the probe for layer 28 probe activates on K/R rich sequence motifs (Figure 6с) and on bipartite nuclear localization signals (L288/375), which generally follow the pattern R R/K(X)nn-1) 12 KRXK (Lu et al., 2021). The latent activates strongly on the R/K motifs at each end and can flexibly recognize the lanitble-length linker sequence that connects them. These sequence motifs engage cellular mechanisms that enable the protein to enter the nucleus, As many NLS
For the other localization classes, we also find variety of interpretable latents, including mitochondrial transit pep- tides (L.88/277) and transmembrane regions (L288/2818, 1.28/1298) (Figure 9c). Overall, this task uncovered many

---

Interpreting Features in Protcin Language Models using Sparse Autoencoders
latents related signal peptides, consistent with the obser- vations from (Almagro Armenteros et al., 2017) that the attention maps of their transformer model trained on this task focus on sequence termini.
et al., 2022; Gordon et al., 2024), the use of motifs (Zhang et al., 2024), and the unexpected effectiveness of early layers some downstream tasks (Li et al., 2024). To facilitate further exploration, we introduce InterProt, an open-source visualizer for interpreting SAE latents.
5.2.3. THERMOSTABILITY
Mechanistic interpretability methods like SAEs hold promise not only for understanding how complex models function but also for advancing scientific discovery (Don- hauser et al., 2024). However, existing approaches often rely on matching features to concepts already known to the scientific community, potentially overlooking novel features. To address this gap, we developed a probing strategy that associates latents with candidate functional roles by training linear probes on SAE latents. Through this process, we uncover features that align with known sequence determi- nants, such as nuclear localization signals, which our probe identified as strongly predictive of nuclear localization. Had the role of these signals been unknown, our probing results would have provided a strong hypothesis for subsequent validation. More broadly, probing SAE latents offers a means to explain how PL.Ms achieve their performance on downstream tasks. Notably, we find that thermostability prediction relies heavily on latents which provide simple amino acid composition statistics.
Thermostability prediction is a task that does not scale with increased pre-training and larger models (Li et al., 2024). Our SAE probes reveal themtthrrosttabiiity prediction relies most on simple amino acid statistics (Figure 6b).
Unlike the other tasks, the middle layers perform worse than early or late layers. We find that in middle layers the most significant latents tend to recognize specific amino acids. For example, layer 24's most positive latents correspond to Arginine (R), Tyrosine (Y), and Leucine (L); its most negative latents recognize Glutamine (Q), Aspartic acid (D), and Threonine (T). The absence of Glutamine is known to correlate with increased thermostability (Farias & Bonato, 2003).
In contrast, earlier layers seem to contain more relevant features. For instance, the most positive coefficient in layer 8 (L8/2064) corresponds a latent that activates primarily on hydrophobic residues, which are important for stability (Figure 6b).
In our probing experiments, we observe that not all predic- tive latents are easily interpretable. For instance, a predictor for membranc localization, latent 1544 predominantly activates on poly- sequences, whose functional rele- vance remains unclear. Such hard-to-interpret but predictive features could result from biases in training data, limitations of our SAE, or ESM. However, another intriguing possibility is that these features correspond to biological mechanisms that have yet to be discovered.
5.2.4. MAMMALIAN CELL EXPRESSION
We were interested to see if our SAB probing method could be used to help interpret a novel task with relevance to drug development: binary human protein expression in CHO cells. The two latents most predictive of expression ac- tivate on terminus-specific long motifs (Figure 9b). The latent most predictive of failed expression recognizes ATP binding site, suggesting that its interactions with ATP may disrupt essential metabolic processes in the host cell.
Limitations. Our study has several limitations. First, we focus exclusively on the 650M -orr-m-mmmtter variant of ESM-2, and extending our analysis to other model sizes, checkpoints, and architectures could yield richer insights into how fea- tures vary with pre-training scale and model architecture. Second, the features uncovered by our SAES are sensitive to the training data distribution; although we use a diverse set of sequences from UniRef50, different datasets may pro- duce a distinct feature set. Third, our probing approach relies on mean-pooling across the sequence for protein-level tasks-a a strategy known to have shortcomings (NaderiAl- izadeh & Singh, 2024)-and exploring alternative pooling methods may yield stronger associations between latents and functional properties. Finally, our capacity to interpret latents is inherently constrained by the current state of bio- logical knowledge, leaving open the possibility that some of our interpretations, particularly for latents showing complex activation patterns, may be incomplete or misstated.
While we found many latents hard to interpret, this task demonstrates the potential of our method in practical set- ting. If SAEs can help discover biological mechanisms that affect expression in CHO cells, then can guide us in designing protein therapeutics with better developability profiles.
6. Discussion
In this work, we train sparse autoencoders (SAEs) to extract interpretable features from protein language model (pLM). We categorize SAE latents based on their activation patterns and specialization across protein families, systematically analyzing how SAE sparsity, width, and layer choice affect feature extraction. Our work offers a new perspective on previous efforts to understand MLMs, confirming known results such as the central role of sequence homology (Lin
Future directions. The success of our SAE linear probes
8

---

Interpreting Features in Protcin Language Models using Sparse Autoencoders
in uncovering known biological mechanisms suggests their potential for discovering new ones. A key next step is to formulate and experimentally test hypotheses around un- explained predictive features. Since SAE-derived features depend on the dataset (Kissane et al., 2024), investigat- ing how data diversity influences feature emergence could be valuable-training on task-relevant sequences may yield more function-speeiffe latents. Another promising direc- tion is model steering via SAEs: while Simon & Zou 2024 provides proof of concept, further research is needed to determine whether SAFs can be used as practical tool for protein engineering.
tein knowledgebase and its supplement trembl in 2003. Nucleic acids research, 31(1):365-37, 2003.
Bricken, T., Marcus, J., Mishra-Sharma, S., Tong, M.. Perez, E., Sharma, M., Rivoire, K., and Henighan, T. Using dictionary learning features as classifiers, 2024. classifiers/index.hine
Chen, H., Tian, B.,, Zhong, J.. Zhang, X., Chen, J., Guo, L., Xiong, X., Zhang, X., Liu, X., and Xiao, B. Automatically defining protein words for diverse functional predictions based on attention analysis of a protein language model. bioRxiv, pp. 2025-01, 2025.
Impact Statement
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhat- tacharya, N.. Goldman, S., Madani, A., and Yang, K. Flip: Benchmark tasks in fitness landscape inference for proteins. bioRxiv, pp. 2021-11,202
This paper presents work whose goal is to better understand protein language models. Our findings may aid in biological discovery, particularly in understanding fonntioe-function relationships. While our work has potential applications in protein engineering and drug discovery, we do not fore- sec any immediate ethical concerns beyond those generally associated with machine learning research.
Detlefsen, N. S., Hauberg, S., and Boomsma, W. Learning meaningful representations of protein sequences. Nature communications, 13(1):1914, 2022.
Acknowledgments and Funding
Dong, T., Kan, C., Devkota, K., and Singh, R. Allo-allo: Data-efficient prediction of allosteric sites. bioRxiv, pp. 2024-09, 2024.
We thank Elana Simon, Neil Thomas, Andre Commman, Yunha Hwang. Garyk Brixi, Peter Koo, Samuel Maffa, Philipe Chlenski, and Neel Nanda for helpful discussion around understanding and analyzing SAE latents. We thank Diego del Alamo, Daniel Saltzberg, James Michael Krieger, and many others on Twitter/X for contributing to SAE la- tent interpretation. We thank Pranay Satya for discussion on training SAEs. We thank the Evolved 2024 hackathon for providing compute resources for an carlier version of this project. We thank Zeming Lin, Uri Laserson, Noah MacCallum, Tess van Stekelenburg, and Matthew Nemeth for their feedback and support during the hackathon. M.I.. and M.A. are supported by NIH grant R35GM150546.
Donhauser, K., Ulicna, K., Moran, G. E., Ravuri, A.. Kenyon- Dean, K., Eastwood, C., and Hartford, J. To- wards scientific discovery with dictionary learning: Ex- tracting biological concepts from microscopy foundation models, 2024. URL 2412.16200
Farias, S. T. and Bonato, M. Preferred amino acids and thermostability. Genet Mol Res, 2(4):383-393, 2003.
Gao, L., Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scal- ing and evaluating sparse autoencoders. arXiv preprint arXiv:2406. 04093,0000...00 2024.
References
Adam Green Through a glass darkly: Mechanistic interpretability the bridge to end-to-end biology, 2024. URL http::////wwmawk.bbbbb.. bio researseee boct--interr--aah--t---e---
Gordon, C., Lu, A. X., and Abbeel, P. Protein language model fitness is a matter of preference. bioRxiv, pp. 2024- 10, 2024.
Jarzab, A., Kurzawa, N., Hopf, T., Moerch, M., Zecha, J., Leijten, N., Bian, Y., Musiol, E., Maschberger, M., Stoehr, G., et al. Meltome atlas-thermal proteome stability across the tree of life. Nature methods, 17(5)4955-53, 2020.
Almagro Armenteros, J. J., Sønderby, K., Sønderby, K., Nielsen, H., and Winther, O. Deeploc: prediction of pro- tein subcellular localization using deep learning. Bioin- formatics, 33(21):3387-339) 2017.
Boeckmann, B., Bairoch, A., Apweiler, R., Blatter, M.-C., Estreicher, A., Gasteiger, E., Martin, M. J., Michoud, K., O'Donovan, C., Phan, I., et al. The swiss-prot pro-
Kannan, G. R., Hie, B., and Kim, P. S. Single-sequence, structure free allosteric residue prediction with protein language models. bioRxiv, pp. 2024-10. 2024.

---

Interpreting Features in Protcin Language Models using Sparse Autoencoders
Kantamneni, S., Engel. J., Rajamanoharan,
Notin, P., Dias, M., Frazer, J., Marchena-Hurtado,, J., Gomez, A. N., Marks, D., and Gal, Y. Tranception: pro- tein fitness prediction with autoregressive transformers and inference-time retrieval. In International Conference on Machine Learning, pp. 16990-17017. PMLR, 2022.
S., and Nanda, N. Sae probing: What is it good for? absolutely something, 2024.
Karvonen, A., Rager, C., Lin, J., Tigges, C.,, Bloom, J., Chanin, D., Lau, Y.-T., Farrell, E., Conmy, A., Mc- Dougall, C., Ayonrinde, K., Wearden, M., Marks, S., and Nanda, N. SAEBench: A comprehensive benchmark for sparse autoencoders, December 2024. URL ht htt:s: //www. www.oneddia....o.m.mmm ber.a//eeeaaaa.oeeeeeeeeeeccccc info.
Paysan-Lafosse, T., Blum, M., Chuguransky, S., Grego, T., Pinto, B. L.. Salazar, G. A., Bileschi, M. L.,, Bork, P., Bridge, A., Colwell, L.., et al. Interpro in 2022. Nucleic acids research, 51(D1) 2023.
Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. Jumping ahead: Improving reconstruction fidelity with jumpuelu sparse au- toencoders, 2024. URL https: wawpivxvvvv..//. org abs/ 14435.
Kissane, C., Krzyzanowski, R., Nanda, N., and Conmy, A. Saes are highly dataset dependent: A case study on the refusal direction. Alignment Forum, 2024. URL http:://ww..aliggeett.ooo hTtp/posts//rtppn1/223/pppppppp saes- www.-rrew-iill--tttttteeeeeeeeee
Rao, R.. Bhattacharya, N., Thomas, N., Duan, Y., Chen, P., C. P. P., and Song, Y. Evaluating protein transfer learning with tape. Advances in neural informa- tion processing systems, 32, 2019.
Li, F.-Z., Amini, A. P., Yue, Y., Yang, K. K., and Lu, A. X. Feature reuse and scaling: Understanding transfer learn- ing with protein language models. bioRxiv, pp. 2024-022400 02, 2024.
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., and Rives, A. Transformer protein language models unsuper- vised structure learners. In International Conference on Learning Representations, 2021. URL https: openreview.oo net/ forrmmid=ffylllleae..
Lieberum, T., Rajamanoharan, S., Conmy, A., Smith. L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., and Nanda, N. Gemma scope: Open sparse autoencoders everywhere all at once gemma 2, 2024. URL https: // //arxiv. org/ abs/ / 2408 05147.
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L.,, Ma, J., and Fer- gus, R. Biological structure and function emerge from scaling unsupervised learning to 250 million protein se- quences. Proceedings of the National Academy of Sci- ences, 118(15)ee01622311 2021. doi: 10. 10.1073/pnas. 2016239118. URL https: www. www.wppaas..mm wwwpppaa.sp..pppppppp.p.....ppp. wtgg///ll abs/ /1cs/1... 1073/ .wwponas...11.......mmmm...m...a 6239118.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., dos Santos Costa, A., Fazel-Zarandi, M., Sercu, T., Candido, S., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022:500902, 2022.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., al. Evolutionary scale prediction of atomic-level protein structure with a language model. Science, 379(6637): 1123-11 1130, 2023.
Schuster, V. Can sparse autoencoders make sense of latent representations?? 2024. URL htttttt/t///ii...../ 0ba//2410.1144
Sievers, F., Wilm, A., Dineen, D., Gibson, T. J., Karplus, K., Li, W., Lopez, R., McWilliam, H., Remmert, M., Söding, J., et al. Fast, scalable generation of high-quality protein multiple sequence alignments using clustal omega. Molecular systems biology, 7(1):539, 2011.
Lu, J., Wu, T., Zhang, B., Liu, S., Song, W., Qiao, J., and Ruan, Types of nuclear localization signals and mech- anisms of protein import into the nucleus. Cell communi- cation and signaling, 19(1):60, 2021.
Masson, H. O., Kuo, C.-.,, Malm, M., Lundqvist, M., Siev- ertsson, A., Berling, A., Tegel, H., Hober, S., Uhlén, M., Grassi, L., et al. Deciphering the determinants of recom- binant protein yield across the human secretome. bioRxiv, pp. 2022-0222222 12, 2022.
Simon, E. and Zou, J. Interplm: Discovering interpretable features in protein language models via sparse autoen- coders. bioRxiv, pp. 2024-11, 2024.
Steinegger, M. and Söding, J. Mmseqs2 enables sensi- tive protein sequence searching for the analysis of mas- sive data sets. Nature biotechmology, 35(11):1026-100 2017.
NaderiAlizadeh, N. and Singh, R. Aggregating residue- level protein language model embeddings with optimal transport. bioRxiv, pp. 2024-01 2024.
10

---

Interpreting Features in Protcin Language Models using Sparse Autoencoders
Suzek, B. E., Huang, H., McGarvey, P., Mazumder, R., and
Wu, C. H. Uniref: comprehensive and non-redundant uniprot reference clusters. Bioinformatics, 23(1))12282- 1288, 2007.
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C.. and Henighan, T. Scaling monosemanticity: Ex- tracting interpretable features from claude sonnet. Transformer Circuits Thread, 2024. URL https: //t.ra/nforrrrrr.crtttttt.../o// scaling scaling-monossnneneo monosemantiiiity//deee htmm.
Vig. J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., and Rajani, N. F. Bertology meets biology: Interpreting attention protein language models, 2021. URL : //arxiv..v.////////// org/ 2206 15222.
Ye. J., McGinnis, S., and Madden, T. L. Blast: improve- ments for better sequence analysis. Nucleic acids re- search, 34(suppl22:WW-WW9 2006.
Zhang, Z., Wayment-Steele, H. K., Brixi, G., Wang, H., Kern, D., and Ovchinnikov, Protein language models learn evolutionary statisties of interacting sequence mo- tifs. Proceedings of the National Academy of Sciences, 121(45)e2400285522, 2024.
Author contributions
E.A. and L.B trained the SAE models, built InterProt visualizer, and interpreted SAE latents. E.A. led the feature characterization experiments. L.B led the interpretable prob- ing experiments. M.L. wrote the code for SAE steering and performed the family-specific latent steering experiments. Y.Y. contributed to the training code. E.A., L.B., M.A., pro- vided feedback on all experiments. E.A. and L.B. wrote the manuscript with feedback from M.L. and M.A. All authors read and approved the manuscript.
Competing interests
M.A. is member of the scientific advisory boards of Cyrus Biotechnology, Deep Forest Sciences, Nabla Bio, and Ora- cle Therapeutics.

---

Interpreting Features in Protein Language Models using Sparse Autoencoders
A. Appendix
A.1. Code availability and model weights
Our SAE training and evaluation code is publicly available at The feature visualization tool, InterProt, can be accessed at https://ánterrpro.: com, and pre-trained SAE weights hosted https: : //wggiuiiii// ngface co hambss//ntterrrrtt-53222
A.2. Background on protein language models
Protein language models (pLMs) apply techniques from natural language processing model biological sequences, leveraging the ability of transformer architectures learn patterns from large-scale data. Just as language models capture structure in text, pL.s ESM family of pLMs are BERT-style transformers trained with masked language modeling objective to learn contextual embeddings of amino acids (Rives et al., 2021; Lin et al., 2023). In this work, we focus on the 650M 5:00--arrmeter variant of ESM-2 (Lin et al., 2023) which has 33 total layers.
learn meaningful representations of proteins from large protein sequence databases. The
A.3. Key differences from InterPLM
Concurrent to our work, Simon & Zou (2024) also trained SAEs on ESM-2 models. Some key differences from work are as follows:
SAE Architecture. Simon & Zou (2024) used ReLU-based SAEs, we employ TopK SAEs, which allow explicit control over L0 sparsity.
Evaluation. Simon & Zou (2024) proposed automated evaluation framework based Swiss-Prot annotations, establish- ing that more SAE latents correspond to annotations compared to ESM neurons. We support this finding with manual evaluation where participants blindly rated SAE latents and ESM neurons based on their interpretability in our visualization tool.
Analysis. Simon Zou (2024) used an LLM to generate SAE feature descriptions based on activation examples and annotations. They also explored the application of SAE features in identifying missing annotations. Our analyses focused on classifying SAE latents based on activation patterns and family-speciffcity and using linear probes to uncover features most predictive in downstream tasks.
Visualization. Both works provide interactive visualizer for SAE latents. interplm.ai from Simon & Zou (2024) display UMAP visualizations, results from Swiss-Prot concept mapping, and other statistics. Our visualizer, interprot.com, focus on the top activating sequences of each latent, displaying activation patterns overlaid on structure, sequence alignments, and information on shared protein family. We used this interface to conduct our human ratings experiment.
A.4. Probe details
Table summarizes the linear probes used in each downstream task.
Table J. Summary of downstream tasks for interpretable probing
DATASET
TASK TYPE
LABEL LEVEL
EVALUATION METRIC
PROBE IMPLEMENTATION
SECONDARY STRUCTURE
CLASSIFICATION CLASSIFICATION
RESIDUE
ACCURACY
PYTORCH LINEAR CLASSIFIER
SUBCELLULAR LOCATION
PROTEIN
ACCURACY
SKLEARN LOGISTIC REGRESSION
THERMOSTABILITY
REGRESSION
PROTEIN
SPEARMAN'S
SKLEARN RIDGE REGRESSION
CHO CELL EXPRESSION
CLASSIFICATION
PROTEIN
ACCURACY
SKLEARN LOGISTIC REGRESSION
12

---

Interpreting Features in Protcin Language Models using Sparse Autoencoders
A.5. Steering speclf- latents
Conserved (*)
Conserved substitution ()
Somewhat conserved ()
Not conserved
1.000
famiy randirm
0.975
Unchanged Retlo
0.925
0.900
0.875
0.850
0.800
1.2
15
2,0
2.
50
33
4.0
1.0
15
20
2.3 Strength
3.0
3.3
4.0
1.0
23
2.0
2.5
300
35
4.0
1.0
15
2.0
25
5.0
3.5
4.0
Strength
Strength
Strength
Figure? 7. The fraction of residues that remains unchanged after steering, averaged for 2089 family specific features revealed using SAE.
We steer each family-specific latent and analyze the resulting sequence changes compared to steering random latents or latents corresponding to amino acid features. Steering refers to clamping of the activation specific latent to a fixed level-either increasing it to amplify the feature or decreasing to suppress For each family-specifie latent corresponding to an InterPro family, we identify the top-activating sequences, conduct protein-prae protein BLAST (Ye et al., 2006), and align sequences within the same InterPro family to obtain a multiple sequence alignment using Clustal Omega (Sievers et al., 2011). Finally, determine the maximum activation of family- latent and clamp the activation of the activating residues to 1x, 2x, 3x, and 4x the maximum value. As a baseline, we steer the residues which activated the family-speciff latent with two random latent dimensions (1 and 1 11) and two amino acid specific latent dimensions (3267 for alanine and 3830 for glycine in the SAE with a hidden dimension 4096 from layer 24). Because we choose two latents as representative random and amino acid latents, we take the average of the unchanged ratio. The unchanged ratio the fraction of residues across the whole sequence that remains unchanged after steering.
Figure 7 compares the fraction of residues that remains unchanged after steering of family-specific (labeled "family"), random (labeled "random"), and amino acid (labeled "aa") latents, by the degree of conservation (decreasing order, left to right). For any degree of conservation, steering family-specific latent has the least effect in the decoded sequence for the same steering strength, suggesting that family-specific latents capture underlying evolutionary constraints that resist disruption. The difference unchanged ratio between family-specifh and random latents is larger in conserved residues, implying steering family-specific latents tends to keep the residues with high consensus. Still, it is interesting that absolute unchanged ratio is roughly in a similar range for any degree of conservation.
A.6. Classifying latents by family specificity
We evaluate the ability single SAE latent to classify protein as belonging to a specific InterPro-defined protein family Paasan-Lafosse al., 2023). To do this, we determine the optimal normalized activation threshold that maximizes the FI score for binary classification of protein families. Specifically, sweep threshold values from 0.1 0.9 in 0.1 increments and select the threshold that results in the highest F1 score. Our final evaluation is conducted held-out test set.
For our evaluation, we use protein sequences from Swiss-Prot (Boeckmann et al., 2003). To reduce sequence re- dundancy, we cluster all Swiss-Prot sequences with a length of 1022 residues or fewer at 30% sequence identity us- ing MMseqs2 (mmseqs easy-cluster swissprot. fasta session tmp --min-seq-id 0.3 3 -c 0. 9
-8. max-seqs 1000000 --cluster-mod 1) (Steinegger & Söding, 2017). Given the low sequence identity between clustered sequences, we perform random train-test split for classification evaluation.
To determine which protein family assess for each latent, identify the InterPro family of the sequence that exhibits the highest activation for that latent. The Fl score is then computed for this family using the thresholding approach described above.
13

---

Interpreting Features in Protein Language Models using Sparse Autoencoders
A.7. Classifying latents by activation pattern
Table 2. Activation pattern classification criteria
Category
Criteria
Dead Latent
If the latent is never activated by any test sequences, it is classified dead latent.
Not Enough Data
less than 5 sequences activate the latent then we say there is not enough data.
Periodic
Features that exhibit consistent activation patterns at regular intervals. These features must satisfy: (1) a high frequency of activation at specific positions (over 50% of distances between activations are the same two values), (2) large number of activation regions (there are more than 10 activations per sequence), and (3) relatively short contiguous activation spans (median length of the top activating contig is less than 10).
Point
Features that activate in highly localized manner, defined by single, prominent activation site (the median length of the highest activating region is 1).
Motif (Short: 1-20)
Features that activate in short contiguous regions (median length of the highest activating region > and < 20) and have an overall mean activation coverage of less than 80%.
Motif (Medium: 20-50)
Features that activate short contiguous regions (median length the highest activating region is 20 and 50) and have an overall mean activation coverage of less than 80%.
Motif (Long: 50-300)
Features that activate in short contiguous regions (median length of the highest activating region is 2 50 and < 300) and have overall mean activation coverage of less than 80%.
Whole
Features that are active across nearly the entire sequence (overall mean activation coverage of greater than 80%.).
Other
Features that do not meet any of the above criteria are classified as "other." "
14

---

Interpreting Features in Protein Language Models using Sparse Autoencoders
A.8. Testing SAE hyperparameters
8000
7000
Family specific feature
6000
5000
4000
3000
dead latent
enough
1000
other
128
256
16
32
64
128
256
perlodic
point
short matif (1-20)
med motif 50)
motil (50-300)
Proportion Features
whole
Family specific feature
3000
0.8
0.6
2000
1000
0.2
0.0
2048
4096
B192
16384
2048
4096
8192
16384
hidden size
hidden size
Figure Testing SAE hyperparameters. k parameter sweep (other hyperparameters held constant, layer=24, hidden size=8096) showing the number of family specific features (left) and the features categorized by activation pattern (right). b. SAE hidden size hyperparameter sweep (other hyperparameters held constant, layer=24, k=64) showing the number of family specific features (left) and the features categorized by activation pattern (right).
15

---

Interpreting Features in Protein Language Models using Sparse Autoencoders
A.9. Additional latent visualizations from linear probes
Top bota shoet intents loyer 16
Top mitochandrion latents layar 28
L16/103: alternating residues sheets
L28/3277: mitcchondrial transit paptide
L16/3788:
residue of some beta sheets
L28/1948 unidentified
L16/3667: unidentified
L28/1690 signal peptice
L18/3666: some beta sheets and helices
L28/2745: unidentified
L16/2770: unidentilidd
unidentiiied surface helx--ellted feature
weggh
Most signtficant CHO expression latents in layer
Top call membrane latents layer 28
domain C-terminus
L28/3815: C-terminus propeptide
L8/1220: unidentifed N-terminus
Cytoplosmic region
ragion following eignial peplide
L8/S167: undanttied
Extrace lulor
L28/2618 transmembrane region
LS/3429: beta barrel domein
L28 2000 transmembrane region
L8/2061: halix-mllated feahure residue bias
polyA regions some helices
L28/3154
ATP-binding
site
Figure 9. Linear probes identify predictive SAE latents for downstream tasks. The top 5 beta sheet latents from the secondary structure prediction task (SAE trained on ESM layer 16). b. The latents corresponding to the most positive and most negative coefficients from the linear probe CHOO cell line expression. The most positive latent activates activates on an ATP binding site (SAE trained on ESM layer 8). c-d. Top 5 mitochondrion and cell membrane latents from the subcellular localization prediction task. The top nucleus latents identify transit peptides. The top cell membrane latents correlate with signal peptides and transmembranc regions (SAE trained on ESM layer 28).
unidentified motif; the most negative latent
16

---

Interpreting Features in Protein Language Models using Sparse Autoencoders
A.10. Distribution of linear probe coefficients
Linear probe coefficient distribution (alpha helix class, layer 16)
ESM Weights Weights
20
Probability Density
15
10
-0.3
-0.2
-0.1
0.0
0.1
0.2
Coefficient value
Figure 10. Example distribution of ESM SAE linear probe coefficients. Shown here coefficients for the alpha helix class from the secondary structure prediction task, using layer 16. The SAE coefficients have higher variance (2.03e-3) than the ESM coefficients (3.24e-4) because more latents are individually predictive of the label. The SAE coefficients also have larger negative skew (-2.05e-1) 1) compared to ESM (-6.51e-22) 2). This can be explained by the SAE latents being more monosemantic: they sort more distinctly into helix vs. non-helix classes, and there are more non-helix examples than helix ones.
17


Reviews:
[{"review_id": "T6dNFxAWYc", "reviewer": "Reviewer_5Na5", "summary": "This paper studies sparse autoencoders trained on the protein language model ESM-2. They find that the SAEs contain a variety of generic and family-specific features, as well as features that can be used to identify sequence determinants of properties such as thermostability and subcellular localization. They also provide an interactive visualization tool to help in the labeling or interpretation of SAE features. Finally, they explore the impact of SAE hyperparameters and ESM layers on the features learned by their SAEs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for these thoughtful comments and suggestions.\n\nWe created an anonymous link for our InterProt visualizer at http://icml.interprot.com and hope that it can provide context on our manual interpretation process and showcase some interpretable features. To support our claims around interpretability, we agree with the need for more human label data and quantitative analysis. As done in the Gated SAE paper (Rajamanoharan et al.), we will conduct a blinded human rater experiment where 5 raters who are familiar with protein biology will assess the interpretability of an SAE latent and ESM baseline as being interpretable (yes/maybe/no). We will include the results in a revision. We hope that this can provide more robust evidence on the value added by SAE compared to directly analyzing the base model.\n\nWe agree with the lack of details on how our work differs from Simon et al. and plan to include more information in our revision. The key differences are as follows:\n\n- We trained TopK SAEs while Simon et al. used ReLU SAEs.\n- We used a larger, 650M param variant ESM-2, compared to the 8M variant used by Simon et al. The 650M param model is far more widely used.\n- We propose a framework for systematically categorizing latents by family-specificity and activation patterns; Simon et al. does not do latent categorization. We also evaluate the effect of different SAE hyperparameters on these feature categorizations. \nWe use linear probes on four downstream tasks to extract interpretable features with the goal of enabling scientific discovery. Simon et al. focuses on demonstrating the feasibility of SAEs on pLMs and performs quantitative interpretability comparisons between ESM and SAE. \n- We share an open-source visualizer InterProt with data visualizations aimed for manual feature interpretation. For example, each SAE latent is displayed with a collection activating sequences across different activating ranges, whether they cluster within specific protein families, providing options to align them. InterProt also enables searching a sequence across all latents, a feature that has enabled discovery of interesting, interpretable latents starting from a protein of interest. Simon et al. also provides a visualizer though it focuses more on displaying activating distributions of each latent and whether it has been linked to any Swiss-Prot concepts.\nSimon et al. proposed a method to automate the interpretation of SAE features using an LLM. We did not explore this approach.\n\nWe see models like ProtoPNet with built-in interpretability as an exciting and complementary direction to post-hoc interpretation (of usually larger and more performant pLMs) via SAEs. Concept bottleneck models have also been applied to proteins, and have shown competitive performance with pLMs [2]. While we agree that a comparison of SAEs to these methods is outside the scope of this work, we plan to add a discussion of these references to our Related Works section.\n\nFinally, we thank the reviewer for the reference on the previous work by Makelov et al. on the effects of training data distribution, a direction we plan to explore in follow-up work.\n\n[1] https://arxiv.org/abs/2404.16014\n[2] https://arxiv.org/abs/2411.06090", "claims_and_evidence": "The claims made in this paper are supported by clear evidence, with none being problematic. Please see more on the methods and evaluation criteria for parts that can be strengthened.", "methods_and_evaluation_criteria": "The proposed datasets are intuitive for the task, however, the evaluation criteria is not very clear. In particular, given that one of the main claims of this paper is that SAEs can help to interpret pLMs, more could have been said about how interpretation was performed and the agreement between different labelers of the SAE features. Ideally, the proposed visualization tool, InterProt, would have been anonymously provided with the submission. Alternatively, some examples of most-activating sequences for a few different latents could be provided.", "theoretical_claims": "This work is mainly empirical, and thus no proofs were provided.", "experimental_designs_or_analyses": "The soundness and validity of all experimental analyses were checked, in particular for Sections 4.3, 4.4, and 5.2. Note that much of the analysis was highly qualitative in nature, and thus could not be rigorously checked without access to the proposed visualizer.", "relation_to_broader_scientific_literature": "The key contributions of this paper are mainly related to literature in mechanistic interpretability that explore how sparse autoencoders can be trained on LLMs and VLMs to understand the various features encoded by models. Given the growing popularity of SAEs, recent works have proposed applying them to scientific models, including similar concurrent work by Simon and Zou (2024) that trains on pLMs, SAE-Rad that trains on a medical VLM (Abdulaal et al, 2024), and more. By attempting to explain models in an unsupervised manner, SAEs have the potential to discover novel concepts that humans may not have been able to explain before. This paper validates the utility of SAEs/dictionary learning for understanding what features pLMs may be leveraging to perform downstream tasks, with novel findings on the differences between layers of pLMs and the organization and prevalence of various features. Thus, this paper's main contribution with respect to prior literature is evaluating if SAEs are interpretable or useful for this previously unexplored modality.", "essential_references_not_discussed": "While this paper cites Simon & Zou (2024) as a concurrent work and notes the differences between the two works, it would be useful if more details were provided on the similarities and differences between the results of each paper, even if the models explored between the two are different. Did the authors of the other paper find similar breakdowns for each type of latent? Were there notable discrepancies between the conclusions drawn in the two papers? I believe further expanding on this discussion would strengthen this paper and highlight the potential generalizability and reproducibility of the claims made.", "other_strengths_and_weaknesses": "Other strengths:\n- The main strength of this paper is the ability to essentially create concept bottleneck models on top of pLMs by training probes on top of SAE features, thus being able to explain how pLMs are able to solve various downstream tasks and coming up with potentially hypotheses for the underlying mechanisms of those tasks.\n- The proposed visualization tool also seems like a significant contribution of this work, but is currently unverified as it was not submitted with the paper for review. \n\nOther weaknesses: \n- It is not clear exactly what the use of SAEs unlocked in this paper. I think the authors could include more discussion of what new scientific understanding of protein modeling / pLMs was gained specifically as a result of using SAEs that would not have been found through rigorous evaluation of the base model or by probing it more traditionally. \n- I wonder how SAEs compare against a prototype-based model, such as a ProtoPNet, that directly relies on canonical examples of features rather than learning relevant latent directions that must be labeled post-hoc to understand. While potentially outside the scope of this work, comparison against other methods would strengthen the work and provide stronger evidence for why SAEs are a reasonable and efficient method for understanding pLMs.", "other_comments_or_suggestions": "In the \"future directions\" section, the authors note that \"training on task-relevant sequences may yield more function-specific latents.\" I would suggest they look at [1], which provides evidence for this behavior in task-specific SAEs. \n\n[1] Makelov, Aleksandar, George Lange, and Neel Nanda. \"Towards principled evaluations of sparse autoencoders for interpretability and control.\" arXiv preprint arXiv:2405.08366 (2024).", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kwAHFsVaUt", "reviewer": "Reviewer_qqyw", "summary": "The paper investigates the interpretability of protein language models by training sparse autoencoders on pLM latents (in particular from ESM2). The goal is to extract and analyze features that pLMs use to represent protein sequences, with the broader aim of linking these features to biological properties. The authors develop a visualization tool, InterProt, to examine the learned features and categorize them based on activation patterns and specificity to protein families. Theyfind that a large subset of latents are family-specific, but some are generic and pick things like intrinsically disordered regions, motifs and known heuristics (e.g. glutamine count for thermostability), down to single residues (prolines at helix boundaries). The authors train linear probes on SAE embeddings and compare their predictive performance to standard ESM embeddings across several downstream tasks. They demonstrate that SAEs can uncover biologically relevant features such as nuclear localization signals and thermostability determinants. The study also explores how SAE hyperparameters influence feature extraction, showing that increased sparsity leads to more family-specific features. Overall, the paper provides a framework for using SAEs to interpret pLMs and proposes that such models can facilitate biological discovery by identifying novel functional patterns in protein sequences.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and many good suggestions.\n\nWe agree with the reviewer’s points around the limitations of SAEs. The presence of a large number of family-specific features suggests that SAEs do indeed learn, or memorize, MSAs. Furthermore, the activation pattern of family-specific features support the hypothesis proposed by Zhang et al. [1] that pLMs store evolutionary statistics via common motifs. Although one cannot conclude from the analysis of SAE features that pLMs have learned any biophysics (as Zhang et al. suggest, they likely do not), the presence of non-family-specific features do present evidence that ESM2 has learned some patterns beyond memorizing common motifs within protein families. For example, features corresponding to generic helixes (L16/2611), hydrophobic core residues (L8/2064), and nuclear localization signals (L28/2375) suggest that pLMs do contain generic notions not specific to a protein family. Our mental model is that pLMs not only store co-evolutionarily information from MSAs but also some useful statistical interpolations between them.\n\nAs is common in interpretable ML, understanding SAE latent representations still relies heavily on human intuition. In the case of pLM SAEs, we benefit from established biological knowledge—such as secondary structure and known motifs—which gives us useful reference points. It's encouraging that many SAE latents align with these expected features. However, our claims would be stronger with a quantitative comparison showing that SAE latents are more interpretable than an ESM baseline. To that end:\n- Though using a smaller model in the ESM family and with a different architecture, Simon et al. [2] showed that more SAE latents map to SwissProt annotations compared to ESM neurons. For a more comprehensive comparison, see our response to reviewer 5Na5. \n- In A.7., we show some analysis of linear probe weight distributions that weakly suggest SAE probe coefficients to be more interpretable.\n- As done in the Gated SAE paper, we will conduct a blinded human rater experiment where 5 raters who are familiar with protein biology will assess the interpretability of an SAE latent and ESM baseline as being interpretable (yes/maybe/no). We will include the results in a revision.\n\nRecent criticism of SAEspoint to the limitations of LLM-based auto-interpretation techniques [3], which were not used in our work. It has also been found that SAEs trained on the same data learn different features [15]. \n\nMany other mechanistic interpretability techniques  have been applied to pLMs. For example, attention matrices have been shown to resemble pairwise contact maps ([4], [5], [6]), and contain information about motifs such as signal peptides [7], which are also represented as SAE features. More recent methods such as model diffing [8], trans-coders/cross-coders ([9], [10]), and other SAE architectures ([11], [12]) are promising future directions for this work.\n\nWe agree that mean-pooling SAE embeddings when performing downstream probes has important limitations. For the presence of a short functional motif or residue-level signal, max-pooling will likely yield clearer results. We would also like to try more sophisticated pooling methods such as aggregating via optimal transport [14] in the future. \n\nFor the definition of family-specific features, we set the F1 threshold to 0.7 because we observed there to be a significant dropoff in F1 score below that point. We will update the supplement with a figure demonstrating this. \n\nFinally, we agree with the suggestion to include more quantitative evaluations of SAE features. Doing so across different hyperparameter schemes would additionally enable a more principled hyperparameters selection strategy. Our initial hope was to achieve this via our feature categorization analysis but only observed small effects across different hyperparameters (A.4). We will evaluate our probes across SAEs trained with different hyperparameters. Specifically, a range of SAE latent hidden dimensions and a range of k values. We hope this will help demonstrate the robustness of our results. \n\n[1] https://tinyurl.com/m8a5ratb \n[2] https://tinyurl.com/3xsfefaj  \n[3] https://arxiv.org/abs/2501.17727 \n[4] https://arxiv.org/abs/2404.16014\n[5] https://arxiv.org/abs/2006.15222 \n[6] https://tinyurl.com/yja6xyhf \n[7] https://tinyurl.com/mub4tuy5 \n[8] https://tinyurl.com/4hwm4tp8  \n[9] https://arxiv.org/abs/2406.11944 \n[10] https://tinyurl.com/4hwm4tp8 \n[11] https://arxiv.org/abs/2404.16014 \n[12] https://arxiv.org/abs/2407.14435 \n[13] https://tinyurl.com/yc2memtc \n[14] https://tinyurl.com/yc8j3fzt \n[15] https://arxiv.org/abs/2501.16615", "claims_and_evidence": "The main claim is that pLMs do not merely memorize protein sequences but instead encode meaningful biological patterns. The authors argue that training SAEs on pLM activations allows them to reveal such patterns in a structured way which is easier to interpret. The claim that follows from the above is that these SAE-derived features are useful for biological discovery, as they can highlight functional determinants of protein properties that might otherwise be hidden in black-box model representations. The claims are reasonably well supported by experimental results: the authors train autoencoders on different layers of ESM-2 and develop the InterProt tool to inspect the learned features. They show that some latent dimensions correspond to biological concepts e.g. secondary structure, conserved sequence motifs, and biochemical patterns, and that many of these features activate within specific protein families. Furthermore it is shown that adjusting SAE hyperparameters (sparsity, expansion factor) influences the number of family specific features extracted.\n\nNevertheless, there are some remarks wrt the claims made: \n1. As a result, the authors suggest pLMs encode biological knowledge beyond simple memorisation. In my opinion this is not entirely accurate. The interpretable latents are a compelling finding but e..g the presence of protein family specific latents could be interpreted as evidence of memorization (as opposed to true generalization) and so does not necessarily mean the model has learned fundamental biological principles rather than statistical correlations present in training data. Such limitation however, is alluded to by the authors in the discussion section. \n2. Whether SAEs truly offer a systematic path to understanding pLMs or if they simply provide another layer of abstraction that still requires human intuition to decode remains to be determined not just in the biology domain but the wider ML space. Furthermore, their susceptibility to different hyperparameter settings is known (and something the authors also look at in the paper). The recent scrutiny by several works in the literature also calls them into question. I would have liked to see this addressed in a bit more detail with some further thought on other techniques from the mech interp literature that could be used for pLMs.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are well matched to the problem. SAEs have been widely studied by the community in particular to interpret activations from LLMs, so their application to pLMs make sense in a similar fashion to how language modelling tasks were applied to protein sequences in the first place. Protein language models have shown strong predictive power for biological tasks but indeed their internal representations have not been as well studied, and this paper as well as a small number of recent works aim to fill this gap. Linear probing to evaluate feature importance is aligned to this goal from the perspective of identifying which features can be most predictive in several downstream tasks. One remark regarding mean pooling: in this case, mean pooling for protein level tasks can obscure important sequence specific details e.g. where functional determinants are localized to specific residues. While it simplifies model inputs, it may limit the ability of the probes to capture fine-grained sequence information. The authors do outline that there could be further work in this direction", "theoretical_claims": "not applicable", "experimental_designs_or_analyses": "Experimental design and analysis are generally well structured and the experiments the authors choose to conduct make sense in this context. The only remark I have here relates to specific choices of hyper parameters in each setting. In this paper we are dealing with SAEs, which as mentioned earlier in this review, and as the authors themselves acknowledge, are sensitive to different hyperparameter choices. The authors do explore this direction a bit, but in my opinion could have done so in a bit more detail. The authors do not provide enough detail on hyperparameter selection strategy. Due to the biases that could be introduced by this step, this hinders reproducibility. The use of logistic regression for classification tasks and ridge regression for thermostability prediction is a reasonable choice, as these methods provide interpretable coefficients that can be analyzed to identify important features. The datasets used for evaluation are well-chosen and appropriate for the study. Secondary structure prediction, subcellular localization, thermostability, and mammalian cell expression are all biologically meaningful tasks. The classification of SAEs into different categories based on activation patterns and family specificity is useful for interpretability, but it is based on heuristics. The threshold of F1 > 0.7 for defining family-specific features is somewhat arbitrary.", "relation_to_broader_scientific_literature": "This paper builds upon research in protein language models and mechanistic interpretability. The field of mechanistic interpretability has seen increasing interest due to the advent of LLMs and techniques like such as sparse coding and dictionary learning have been applied to them to identify human interpretable features (see e.g. [1]). The idea of applying techniques from the interpretability literature to interpret pLMs is not in itself new but it is timely and a promising area of research gathering increasing interest by the community (see e.g. [2, 3, 4]). \nPerhaps the most related work is InterPLM [3] which uses SAEs to identify human-interpretable features and correlating them with biological concepts such as binding sites, structural motifs and functional domains.  \n\n[1] https://arxiv.org/abs/2309.08600\n[2] https://arxiv.org/abs/2411.06090\n[3] https://arxiv.org/abs/2412.12101\n[4] https://arxiv.org/abs/2502.09135", "essential_references_not_discussed": "not applicable", "other_strengths_and_weaknesses": "One of the main strengths of this paper is its originality in applying SAEs to interpret pLMs. As mentioned earlier, this is a relatively new field and a great effort from the authors to test various properties of pLMs through this machinery. The introduction of InterProt as a tool for visualizing these learned features adds practical value. The paper is clearly written and well structured. \n\nThere is a concern with respect to novelty, as there have been other works from the literature doing several similar things, and by itself this work is not presenting a new method but applying known techniques to pretrained models. Furthermore, as mentioned earlier, the paper could benefit from more content and detail (in particular in the supplementary section) supporting their findings and the main thesis of the paper. There are some potential concerns with respect to statistical validation (e.g. F1 threshold) and the results would be stronger if there were quantitative comparisons between extracted features and known biological annotations. In the downstream tasks, evaluations demonstrate that SAE embedding are predictive but there is no ablation analysis to determine whether specific latents are directly responsible for performance improvements. Finally, there is some concern with respect to generalisability - the experiments focus on a single pLM (it is unclear what results other pLMs would yield) and dataset.", "other_comments_or_suggestions": "not applicable", "questions_for_authors": "not applicable", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oyMFM95Q0G", "reviewer": "Reviewer_ZoZr", "summary": "The authors train SAEs on ESM-2 (a large protein language model), characterize the discovered features, use these organized features to better understand how ESM-2 learns protein representation.\nThey also develop a visualization tool, and find SAE features that correspond to known properties such as thermostability and subcellular localization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed reading and thoughts. \n\nWe acknowledge that the lack of other pLMs in our analysis beyond ESM-2 means that we should not make general conclusions about all pLMs. We intend to update the text to reflect this, replacing pLM with ESM-2 to narrow the scope of the results. \n\nAs the reviewer points out, the performance of SAE probes on the mammalian expression task lags behind the ESM probes. In the text, we say that “linear probes on SAEs achieve performance similar to their ESM embedding baselines across all layers”. We intend to update the text to qualify this statement, that they achieve similar performance on most tasks. \n\n> In the related work, the authors argue that their work focuses on coevolution and scientific discovery. I don't see any experiments focusing on coevolution; am I misunderstanding?\n\nThis is a good point, we could have been more specific. By coevolution, we refer to our analysis of features which seem to correspond to protein family-specific motifs. \n\nWe would like to expand on why we view our results as significant from a biological perspective. Understanding biological sequences such as protein sequences remains a difficult problem. Traditionally, biologists have used tools such as multiple sequence alignments and statistical models to identify local patterns in protein sequences. Recently, protein language models trained on millions of sequences across evolution have been successful in providing useful protein representations. Embeddings from pLMs are often used as input to simple models (e.g. linear regression) on downstream tasks such as protein structure and variant effect prediction. Yet as these representations are uninterpretable, it is not easy to identify what features the pLM has learned which contribute to task performance. We advance on prior work by demonstrating SAEs can help reveal what ESM-2 learned features are important for downstream task performance. For example, by probing on SAE latents, we can identify that amino acid composition is an important predictor for thermostability. This may allow a biologist to gain insight into novel sequence determinants of a protein property which would have required manual feature crafting previously. As representations from ESM are widely used, we offer SAE representations as a drop-in replacement with (usually) similar performance as the ESM representations but with interpretable dimensions. We hope this clarifies the advantage of SAE-derived representations.", "claims_and_evidence": "Overall, the claims are well-supported by evidence.\nI would prefer to shrink the claims from \"pLMs\" in general to ESM-2 unless the authors analyze more than one pLM (which they do not, as far as I can tell).\nFor example, \"we determine that pLMs use a combination of generic features and family-specific features to represent a protein.\" (abstract, lines 024-026).\n\nWhile I agree that SAEs are likely to apply to multiple pLMs, I think all SAE+pLM work has used ESM-2 (the authors note concurrent work, Simon and Zou 2024, also analyzes ESM-2). This is noted in the Limitations section already.\n\nMinor: the authors claim that linear probes on SAE features are more reliably than linear probes on ESM features. This doesn't really appear true for Mammalian cell expression (Figure 4, lower right).", "methods_and_evaluation_criteria": "The methods and evaluation are appropriate.\nThe authors are not proposing any new algorithmic innovation; they apply TopK SAEs to a pLM (ESM-2).\nTheir evaluations are qualitative in nature because they are interested in qualitative understanding of how useful SAEs are in the context of pLMs.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experimental design is good.\nIn the related work, the authors argue that their work focuses on coevolution and scientific discovery.\nI don't see any experiments focusing on coevolution; am I misunderstanding?", "relation_to_broader_scientific_literature": "The authors fairly position their work in the landscape of interpreting pLMs, using SAEs on pLMs, and dictionary learning methods on biological models.\n\nHowever, it's unclear to me how SAEs improve on prior work.", "essential_references_not_discussed": "No essential references are missing.", "other_strengths_and_weaknesses": "This paper applies SAEs to pLMs.\nIt is one of the first works to do so.\nThis is undoubtedly a strength.\n\nUnfortunately, the positioning (or lack thereof) makes it challenging for me (a non-biologist) to understand why the results are impactful.\nI would improve my score if the authors can explain why the results are significant from a biology perspective.\nDo SAEs offer a path towards answering questions that were previously unanswerable?\nAre SAEs significantly cheaper, or more scalable, or easier to tune than other interpretability methods?\nIt's not clear to me why I should care about SAEs+pLMs---I'm sure there's a reason, but this current iteration does not convince me to care.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LOLwvni1ox", "reviewer": "Reviewer_fdKK", "summary": "This paper shows that SAEs can be used to better understand PLMs. They show that the features are interpretable via a series of case studies, including some clean histograms of activating examples. They use the SAEs to make high level observations about the PLMs, such as by categorizing the feature activation styles (point, short motif, etc), and studying the variation over the course of layers. Finally, they investigated probing for properties of proteins, using mean pooling, both on the residual stream and the feature activations. There was comparable performance suggesting that key information has not been lost by the SAE. Further, they find that the features used by a probe are often interpretable and find related concepts. Some of these relationships are not obvious but were discovered in prior research (?) suggesting SAEs have the potential to generate new hypotheses to validate.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their suggestions and enthusiasm for our work. \n\nWe agree that one of the limitations of the paper is a heavy reliance on qualitative measures of feature interpretability. Towards making feature analysis more quantitative, we introduced the family specificity and activation pattern categorizations. In our experience, family specific-features are interpretable (can be interpreted as a specific sequence motif in a specific family), so the number of seemingly interpretable features is lower bounded by the number of family-specific features. But these measures admittedly don’t truly measure interpretability, and we would love to include results for human raters, and compare it to the ESM baseline. As done in the Gated SAE paper, we will conduct a blinded human rater experiment where our 5 raters who are familiar with protein biology will assess the interpretability of an SAE latent and ESM baseline as being interpretable (yes/maybe/no). We will include the results in a revision.\n\nYour interpretation of our goal in the probing section is correct; we are aiming to show that SAE representations haven’t lost key information (as demonstrated by task performance) and that the highest weighted latents correspond to features which makes biological sense with respect to the task. You are right that we could identify relevant features using a simpler method. Our motivation for probing, besides simply identifying relevant features for a task, was to demonstrate how SAE embeddings could be used as a drop-in replacement for dense ESM embeddings, without a significant hit to performance. Since in the pLM literature, linear probing is typically used to measure downstream task performance [1], we opted to do the same to make a head-on comparison. \n\nRegarding steering family specific latents, you make an excellent point. We will add a control where we steer a different (not family specific) latent to better demonstrate that family specific features are more important for conserved regions than other features. We will include this in the camera ready version. This is our attempt at getting more “causal” evidence to interpret SAE latents. As touched upon in our future directions section, getting interpretable steering results using SAE latents beyond simple amino acid specific features has remained challenging, and is a direction we will continue to explore in the future.\n\nIn our calculation of the F1 score for family specificity, we also include sequences which do not activate the latent of interest. We believe this should help account for the false negative problem you raise. \n\nTo make our work more accessible to the broader mechanistic interpretability, we will add a primer to introduce protein language models and some domain specific terms we often use in the paper. There have been some good reviews recently published which we can link as well [2]. And we’ve happy you like the name of our project! \n\n\n[1] https://www.biorxiv.org/content/10.1101/2024.02.05.578959v2 \n[2] https://www.nature.com/articles/s41587-024-02123-4", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Covered well by the paper", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "See below", "other_comments_or_suggestions": "## Major Comments\n*The following are things that, if adequately addressed, would increase my score*\n\n1. This was a delightful paper! I'm now fairly convinced that PLMs learn interpretable concepts and that SAEs have the potential to be a valuable tool for scientific discovery by extracting them. Interpretability for scientific discovery has long been a dream of the field, and this seems like a step in the right direction! This is a clear accept and I am open to a strong accept if the concerns below are addressed\n    - Caveat: I work in interpretability, but not with PLMs, so my assessment of the novelty there is shaky. Ditto, I have not carefully checked the PLM related details of the work.\n2. The main problem with the paper, in my opinion, is that it rests too much on qualitative case studies. These are great and add a lot of flavour and detail to the work, but are vulnerable to potential cherry picking. Assessing what I see as your key contributions for this:\n3. Claim: SAE features are interpretable\n    - This is OK. I think figure 3a is great, and I love the InterProt visualisations, but these are equally consistent with 5% of features being interpretable and 95% being\n    - The main fix I would recommend is randomly choosing 100 features and for each one trying to interpret the top activating examples and just rating each for whether there's a consistent pattern. As done in eg the Gated Sparse Autoencoders paper\n4. Claim: Features in different layers have different types of activation patterns\n    - This seems well supported to me. Table 2 was very helpful and clear and seems reasonable. The area plot in 3c is great!\n5. Probing\n    - I struggled to follow the exact point you were trying to make here. The claims I took away from this were:\n        - Probing still works on SAE activations, therefore they haven't broken anything or lost key info (but it need not be interpretable)\n        - Given a labelled dataset for some concept, we can find related features by looking at large coefficients after training a probe\n    - I found it a bit odd that you were using dense probes on the SAE activations, rather than sparse probes (which is what I consider to be standard - see eg Finding Neurons In A Haystack for discussion of different methods). But scoped into the two claims above this is OK, as the probe is just a method to find key latents\n        - Note: I predict that the probing is unnecessarily complication and that you could find similarly good features by looking for those with the highest mean difference or difference in how often they activate between the positive and negative labels\n    - It would be even better if you took random concepts (from some external list of suitable datasets, filtering for those that a dense linear probe works well for) and looked at the associated SAE latents, and said how often there's an interpretable connection (and how often it's trivial vs non-trivial)\n6. I don't understand the \"steering with a family feature changes it least at evolutionarily conserved points for that family\". What exactly is the computation being done here? Shouldn't the pLM already be confident in the conserved points, so it's harder to shift? Have you compared to a control like steering with a feature of a different family?\n    - An alternative experiment would be to look at the log prob of the relevant token and do gradient based attribution from each SAE feature to that log prob - ie, take the gradient of that log prob with respect to each feature, elementwise multiply by the activations, and add it up across the sequence dimension (basically a saliency map over SAE features). Family features should have high attribution in their families and not otherwise, according to this hypothesis. I recommend excluding the first token when doing this, as ablating SAE features is not a valid operation there (it's constant) and will often mess with your results\n7. There may not be time in rebuttals, but I think your paper would be even stronger with causal evidence for the interpretable role of SAE features.\n    - For example, showing that when you ablate a feature, the times when the correct next log prob decrease the most are interpretable\n    - Or showing that you can steer the model in predictable ways by adding it in to unrelated contexts\n    - One useful technique for exploring this would be gradient based attribution (approximating the effect of an ablation) of each SAE feature at each token to the next log prob at some interesting token - you could visualise this and see which tokens seemed most relevant and where, and if this matched your hypotheses\n        - If you do this, be careful about hindsight bias! Eg do blinding, where you get shown 5 features, of which one had the highest attribution, and have to predict which is why\n\n## Minor Comments\n*The following are unlikely to change my score, but are comments and suggestions that I hope will improve the paper, and I leave it up to the authors whether to implement them. No need to reply to all of them in the rebuttal*\n\n1. When, eg in Figure 3a, you argue that a feature has some explanation, you need to also check for false negatives - things in the beta-lactamase family that it doesn't fire on. Otherwise it could be much more specific, eg activating on every other token in beta-lactamase\n    - This is a minor comment because basically every other SAE paper also fails on this point, and I expect it to in fact be more specific than that explanation\n2. I am very familiar with SAEs but not PLMs. If you want people like me to be able to engage well with your work, having a short appendix primer on PLMs (or linking externally to one) defining key things would help a lot, eg what ESM-2 was trained on, how things are tokenized, what protein jargon like residues & secondary structures mean, etc. \n3. If \"quality of names\" was a category, I'd give InterProt 5/5\n4. I wonder if the several alpha helix features thing is feature splitting (as discussed in Towards Monosemanticity). You could test this by training an SAE with smaller dictionary size and seeing if an alpha helix feature forms\n5. Mammalian cell expression seems very exciting - this doesn't change my assessment of the paper as it lacks any real exploration but I think it's reasonable to leave out of scope. I'd love to see the follow up paper though!", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper trains sparse autoencoders on a large protein language model (ESM-2), characterizes the discovered features, and uses these to better understand how ESM-2 learns protein representation. The authors also developed a visualization tool, and find features that correspond to known properties such as thermostability and subcellular localization. Overall, the claims are well-supported by evidence.
I agree with the reviewers that this is a solid work on a timely topic.
Decision: Accept (spotlight poster)