Controlling Repetition in Protein Language Models
Authors: Jiahao Zhang
ZEQING ZHANG
Di Wang
Lijie Hu
Venue: ICLR 2026 Poster (2026)
Keywords: Protein Language Models
Interpretability
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Protein language models (PLMs) have enabled advances in structure prediction and de novo protein design, yet they frequently collapse into pathological repetition during generation. Unlike in text, where repetition merely reduces readability, in proteins it undermines structural confidence and functional viability. To unify this problem, we present the first systematic study of repetition in PLMs. We first propose quantitative metrics to characterize motif-level and homopolymer repetition and then demonstrate their negative impact on folding reliability. To address this challenge, we propose UCCS (Utility-Controlled Contrastive Steering), which steers protein generation with a constrained dataset.
Instead of naively contrasting high- vs. low-repetition sequences, we construct contrastive sets that maximize differences in repetition while tightly controlling for structural utility. This disentanglement yields steering vectors that specifically target repetition without degrading foldability. Injected at inference, these vectors consistently reduce repetition without retraining or heuristic decoding. Experiments with ESM-3 and ProtGPT2 in CATH, UniRef50, and SCOP show that our method outperforms decoding penalties and other baselines, substantially lowering repetition while preserving AlphaFold confidence scores. Our results establish repetition control as a central challenge for PLMs and highlight dataset-guided steering as a principled approach for reliable protein generation.

Published as a conference paper at ICLR 2026
CONTROLLING REPETITION IN PROTEIN LANGUAGE MODELS
Jiahao Zhang' Zeqing Zhang ., Di Wang", and Lijie Hu 1-f
1 Moohaeed bin Zayed University of Artificial Intelligence (MBZUAD) 2 KKing Abdullah University of Science and Technology (KAUST)
ABSTRACT
Protein language models (PLMs) have enabled advances in structure prediction and de novo protein design, yet they frequently collapse into pathological repeti- tion during generation. Unlike in text, where repetition merely reduces readability, in proteins it undermines structural confidence and functional viability. To unify this problem, we present the first systematic study repetition in PLMs. We first propose quantitative metries characterize motif-level and homopolymer repeti- tion and then demonstrate their negative impact on folding reliability. To address this challenge, we propose UCCS UUillity-Controll Contrastive Steering), which steers protein generation with constrained dataset. Instead of naively contrast- ing high- vs. low-repetition sequences, construct contrastive sets that maxi- mize differences in repetition while tightly controlling for structural utility. This disentanglement yields steering vectors that specifically target repetition without degrading foldability. Injected at inference, these vectors consistently reduce rep- etition without retraining heuristic decoding. Experiments with ESM-3 and ProtGPT2 in CATH, UniRef50, and SCOP show that our method outperforms de- coding penalties and other baselines, substantially lowering repetition while pre- serving AlphaFold confidence scores. Our results establish repetition control as a central challenge for PLMs and highlight datasct-guided steering as a principled approach for reliable protein generation.
INTRODUCTION
Protein language models (PLMs) are trained on large corpora of protein sequences in a manner analogous to large language models (LLMs) in natural language processing (Consortium) 2024). They have enabled impressive advances in structure prediction and de novo protein design (Lin et al. 2022 Hayes et al. 20255 Ferruz et al. 2022] Madani et al., 2023 Li et al. 2024 Hesslow ict al. 2022). Despite these successes, PLMs may suffer characteristic generation failures. Among them, the most prominent is pathological repetition, where outputs collapse into redundant motifs or long homopolymers, severely limiting the biological utility generated sequences. To illustrate this phenomenon, we present representative
in Table []
Repetition is not unique to proteins: in natural language generation, LLMs frequently produce low- diversity continuations that harm readability (Holtzman et al., 2020 Welleck et al., 20200) Meister let al. 2023). However, in the protein domain, the consequences are substantially more severe. Repetitive amino acid sequences lack structural diversity, often resulting in unstable folds and non- functional proteins. Moreover, repetition in proteins is more complex than in text because it directly interacts with structural constraints: suppressing too aggressively may degrade folding reliability and reduce biological plausibility. Surprisingly, this issue has remained largely overlooked in the protein modeling community.
In natural language processing, two classes of approaches have been explored to mitigate repetition. The first are decoding- -based heuristics, including stochastic sampling strategies (Fan et al. 2018
"The first two authors contributed equally to this work. CCorrsponndecce Lijie Hu (1 jie, hu@mbzuai ac).

---

only tailored to the protein Published as a conference paper at ICLR 2026
Table 1: Representative cases of natural proteins vs, PLM generations arranged by sequence length (rows) and dataset/model (columns). Color indicates pLDDT confidence: blue high (> 90), cyan/green medium (70-90), orange/red low (< 70)
length (au)
CATH
SCOP
Uniref50
ESMB
ProtGPT2
128
256
512
Holtzman et al. 2020 n- n-gram blocking (Kulikov et al. 2019), and repetition penalties. More re- cently, interpretabilty from degenerate behaviors (Yao et al. 2025). In PLMs, decoding-time interventions such as repeti- tion penalties, temperature scaling, and sampling adjustments have been adopted, largely by directly transferring techniques from NLP (Ferruz al. 2022; Chen al., (024). Some of these heuristics may incidentally reduce repetition, but they are not specifically designed to address this problem in proteins. Moreover, they often introduce undesirable side effects, such as lowering structural confidence (e.g., reduced AlphaFold pLDDT (Jumper et al.) (00T)), thereby undermining biological viability. This trade-off highlights the need for approaches that domain but also more principled and interpretable.
methods leverage internal representations identify and steer away
Addressing this problem poses several challenges. First, repetition and structural utility are tightly entangled, such that naïve reductions in repetition often degrade folding reliability. Second, PLMs provide no explicit mechanism disentangle repetition from other generative factors. Third, con- ventional text-based repetition metrics capture only surface-level diversity and fail to reflect the bio- logical consequences of repetitive patterns. Since pathological repetition is newly recognized issue in protein generation, the first step is to formally define this failure mode and establish evaluation metries. this end, we introduce two complementary metrics. The first is unified repetition score R(x), integrating token-level entropy, motif-level r--gram diversity, and homopolymet penalties into a single measure of degeneracy. The second is a utility score U(x), (z)) derived from AlphaFold-bass based structural proxies (pLDDT, PTM)), which quantifies whether generated sequences remain plausibly foldable. These metrics allow us to formulate repetition control as a constrained optinization prob- lem: reduce degeneracy without compromising structural plausibility. This unified perspective not only provides principled evaluation, but also forms the basis of our method.
After we unify this problem, seek to overcome this trade-off between repetition and folding reli- ability. Specifically, we propose level intervention that disentangles repetition from structural plausibility. UCCS builds contrastive datasets matched in utility but separated in repetition, derives steering vectors from hidden acti- vations, and injects them during generation suppress degeneracy without damaging foldability. We extensively evaluated UCCS on two representative PLMs (ESM-3 (Hayes et al.) 2025) (MLM) and PROTGPTY (Ferruz al.) 2022) (AR) across three standard protein datasets (CATH (Knud-) 2024). SCOP(Murzin et al. 1995)) in two generation settings (unconditional and conditional). Results demonstrate that UCCS consistently achieves best trade-off:: it significantly improves the repetition score R(xr) while maintaining even enhanc- ing the utility score U (z). outperforming decoding heuristics (e.g., temperature, top-p sampling) and
Contrastive Steering (UCCS), representation-
sen & Wiui 2010).
'Columns 2-4 of Table 1 show natural proteins, while columns 5-6 display PLM "LM--gnerated sequences. Generated cases often exhibit repetition artifacts (c.g., long homopolymer stretches in orange/red, indicative of poor foldability), whereas natural proteins display diverse and structurally coherent patterns.
2

---

Published as a conference paper at ICLR 2026
mechanism baselines. To the best of our knowledge, this work presents the first systematic study that identifies, characterizes, and unifies pathological repetition as a critical failure mode in protein generation. We summarize our contributions as follows.
We present the first systematic study pathological repetition in PLMs, introducing principled metrics that capture both motif-level and homopolymer repetition, and validating their ability to reflect folding reliability.
We introduce Utilityy Contry-- Contrastive Steering (UCCS), a dataset-guides steering method that disentangles repetition control from biological utility.
We conduct a comprehensive evaluation across modeling paradigms (auto-encoding and au- toregressive PLMs), generation settings (unconditional and conditional), and multiple protein datasets (CATH, UniRef50, and SCOP). Results show that UCCS consistently reduces repeti- tion while preserving foldability, outperforming decoding heuristics and other baselines.
2 RELATED WORK
Generative PLMs for protein design. Recent advances in protein language models (PLMs) have enabled impressive progress
novo protein design. By training on large-scale sequence databases such as UniProt (Consortium, (024). models like ESM (Hayes et al.) 2025). Prot- GPT2 (Ferruz et al., 2022). and ProGen (Madani et al.] (023) learn rich statistical representations of amino acid usage and have been applied to generate novel, diverse, and partially functional proteins. These works demonstrate the potential of generative PLMs as powerful priors for protein engineer- ing, showing applications in structure prediction (Lin al. 2022). mutational effect estimation (Li) eet al. 2024), and controlled design with conditioning signals (Hesslow et al. (022). However, pre- vious studies overwhelmingly emphasize positive outcomes of PLM generations, while systematic failure modey in particular pathological repetition has received little attention. This motivates us to look at related phenomena in natural language generation, where repetition and degeneration have been extensively studied (see more related work in the Appendix A
Representation steering. Beyond decoding heuristics, recent work in NLP and vision has explored manipulating internal representations to steer model behavior (Chen et al. 2025 Li et al., 2025] Hu al. 2025 Yu et al. 2025] Hu al.) 2024). Activation steering methods identify directions in hidden states correlated with specific attributes and adjust them to amplify or suppress those at- tributes (Turner et al. 2024 Rimsky et al. 2024 Hong et al., 2024). Related approaches such as task arithmetic (Ilharco al.) (023) and representation engineering (Meng et al. 2022 Dai et al. 2022 Jiang et al., |025] et al. 2024) Yang et al. 2024) demonstrate that linear operations in embed- ding activation space achieve controlled changes without retraining. These findings suggest that neural representations encode disentangled factors of variation, but to our knowledge, no prior work has attempted isolate control repetition-speeifie directions in protein language models. Beyond these computational approaches, insights from biological systems further underscore the importance of repetition control, as discussed next.
Biological perspectives on repetition. While most previous discussions focus on computational as- pects. biological evidence also supports the importance of controlling repetition. Although natural proteins can contain ordered repeats (e.g., 3-propellers or coiled coils (Kajava 2012)), uncontrolled repetition is typically deleterious: pathological repeat expansions such as polyglutamine tracts in Huntington's s disease drive misfolding and aggregation (Gatchel & Zoghbi 2005). while dipeptide repeats from C9or772 expansions in ALS/FTT disrupt cellular homeostasis (Freibaum et al. (015).. Even modest low-complexity repeats are enriched in proteins prone aggregation or aberrant phase separation (Hughes et al. 2018). These biological findings reinforce our motivation: reducing patho- logical repetition in PLMs is not only modeling concern but also critical for maintaining foldability and realistic protein design. For a broader discussion covering interpretabillty -rrientt PLM analy- and parallels to repetition in natural language generation, please refer to Appendix A
UNIFYING REPETITION PLMS
3.1 PROTEIN LANGUAGE MODELS
Let A denote the amino acid alphabet, and x = (aI,..., ..Tr.. each St. A. A pretrained protein language model (PLM) M can be trained in either of two
.T)) a protein sequence of length T, where
3

---

Published as a conference paper at ICLR 2026
paradigms: (i) Masked language modeling (MLM): The model estimates conditional probabilities of the form (xx Iixe), where Pi, denotes the sequence with position masked out. (ii) Autore- gressive (AR): The model estimates conditional probabilities M(x, xice) of the next token given its prefix. In generation, PLMs can operate in the unconditional setting (starting from special beginning token) or in the conditional setting, where prefix p is provided.
3.2 PATHOLOGICAL REPETITION IN PL.MS
We focus characteristic failure mode of PLMs known pathological repetition. Generated sequences often collapse into degenerate, low-complexity patterns that reduce diversity and impair folding into stable 3D structures. We categorize these failures into two canonical forms.
Motif-level repetition: Short n-gram fragments recur repeatedly (e.g. AGAGAG). This corre- sponds to local looping behavior, where the model fixates on short subsequence and generates eyclically.
Homopolymer repetition: A single residue is extended into long runs (e.g.. AAAAAA). This represents a global collapse into low-complexity sequence dominated by one amino acid type.
We argue that these two categories suffice to characterize pathological repetition in PLMs. In con- trast natural language, where large vocabulary enables many types of semantic redundancy, protein sequences are restricted to an alphabet of only 20 amino acids. This limited vocabulary con- strains the space of degenerate behaviors, which therefore manifest relatively simple but severe collapse patterns. Our empirical analysis (Appendix F2 Fig. [ confirms that virtually all repet- itive failures produced by PLMs fall into either motif-level looping or homopolymer expansion, supporting the sufficiency of this taxonomy. The two categories are complementary: motif-level repetition corresponds to local cyclic degeneracy, while homopolymer repetition represents a global collapse into complexyy sequences. They capture the dominant failure modes that undermine the structural plausibility of PLM- PLM-generated proteins.
3.3 QUANTIFYING PATHOLLOICCAL REPETITION
To systematically study pathological repetition, we adopt and extend diversity metrics originally used NLP, while adapting them to the specific characteristics of protein sequences. Token-level entropy and distinct-n have proven be effective in capturing global imbalance and local motif currence in natural language, and provide a natural starting point for analyzing degeneracy in PLMs. However, directly relying these measures is insufficient for proteins: due the much smaller amino acid vocabulary, PLMs exhibit extreme collapse behaviors, most notably long homopolymer runs, which are rarely observed in text. To address this, we design three complementary metries: entropy, distinct-n, and novel homopolymer diversity score. Each is motivated by distinct failure mode. We will discuss them in details as following.
Token-level entropy (Hnorm). PLM generations often collapse into a small set of residues (e.g., dominated by alanine and glycine). To measure such global imbalance, we compute the normalized Shannon entropy of the empirical unigram distribution.
H Hnorm (I) E.EA p(a) loga p(a) Hnorm (x) E [0, 1].
log, [A]
where p(a) is the frequency of amino acids a in sequence v. High entropy indicates balanced amino acid usage, while low entropy reflects global collapse into a few residues. As further demonstrated in Appendix C.2 Fig 6. PLM-generated sequences indeed exhibit skewed amino acid distributions, with over-representation of few residues (c.g., alanine A in ESM3). This empirical evidence supports the use of Haorm as a biologically meaningful measure of global degeneracy(Fig
n-gram diversity (Distinct-n). To quantify local motif repetition, use the distinct-n metric Li ct al. 2016), defined as
Distinct-n(z) lnmgq ngrams,, (x)ams,(()
Ingramns, (=)
We focus 2, 3 for both methodological and biological reasons. In NLP, Distinct- 1/2 stan- dard indicators of short-range redundancy, whereas Distinct-1 is less informative in protein mod- eling due to the small amino acid vocabulary. Biologically, dipeptides and tripeptides constitute
4

---

Published as a conference paper at ICLR 2026
Repelilion metrics: KDE
JS divergence
10+
6.-
0.0
04
0.8
19pay
CATH
86
05
ES49
Uniaudo
Acre
BM4
CAte
Unviets
BM0
oate
usua
Procante
Lnpiudo
Percanz
BD
Dalaner
Datanel
Datanet 8
Datonel
Figure 1: Repetition metrics reveal distinet failure modes in PLMs. Top: Kernel density esti- mates (relative density, peak = 1) repetition metrics across natural (CATH, UniRef50, SCOP) and -L.eggeneraed (ESM3, ProtGPT2) datasets. Bottom:
divergence heatmaps comparing dataset pairs. Large JS divergence between PLMs and natural datasets confirms that our metrics cleanly separate repetitive, degenerate PLM outputs from biologically diverse natural pro- teins. Huorm separates both PLMs from natural proteins, confirming global repetition pathology. Distinct-2/3 highlight lotif-level repetition characteristic of autoregressive models like ProtGPT2, while Repoly captures long-run run homopolymer collapse unique to masked models like ESM3.
fundamental structural motifs (e.g., loops, Gly-X-Gly signatures). Empirically, PLMS often degenerate into repeated short patterns (e.g., AGAGAG), which sharply reduces Distinct-2/3 values. In contrast, natural proteins maintain consistently higher scores (see Appendix [F2 Figs. [ These observations support Distinct-2/3 as effective measures of motif-level diversity in protein sequences.
Homopolymer diversity score (Rhpoly). Unlike natural language, PLMs often degenerate into long homopolymer stretches (e.g., AAAAAAAA,, which are not captured by g-gram diversity: repeating single residue trivially yields high n-gram overlap. To address this, we introduce the homopolymer diversity score:
1 Rupoly (x) =1 1 Σ
l 1(4 2).
where é, is the length of the i-th homopolymer run, and is threshold (default = 4). The choice k 4 is biologically motivated: natural proteins occasionally contain short runs of 2-3 identical residues (e.g., AA or GGG), which are not pathological, but the lengths 2 4 almost always correspond low-complexity, unstable regions. Empirically, we find that such long runs are pervasive PLM generations but rare in natural proteins. As shown in Fig. [ Appendix F.2 Rhpoly sharply separates natural and synthetic datasets, confirming its effectiveness dedicated measure of homopolymer collapse.
Aggregation. Each metric captures a different granularity degeneracy: Hnorm reflects global diversity, Distinct-2/3 capture local motif recurrence, and Rapoly captures long-run collapse. Indi- vidually, none of them suffices: for instance, low entropy may arise from either motif repetition or homopolymer collapse, while Distinct-2/3 fail to penalize homopolymers. We therefore aggregate them into a unified repetition score R(xx) (see Appendix E.. for details), which provides a compre- hensive quantification of degeneracy. In Section we pair R(xx) with complementary utility score (xx) derived from structural proxies (e.g., AlphaFold PLDDT), ensuring that repetition control is evaluated jointly with foldability. Importantly, the validity of our repetition metrics is empirically supported: as shown in Appendix [22 they sharply separate natural proteins (POS) from PLM gen- erations (NEG), confirming their ability capture pathological repetition in practice.
5

---

Published as a conference paper at ICLR 2026
Unilit-Conttolled Contrustive Steering (UCCS); Mechanism Overview
Problem
Steering
Control Repetition
Campute mean difference vector
Steer during generstion
Protein Language Model
De protein design
Commmna Layp
App
D+
(x) (x)
LLAGAGAG... ENPNVPNPP...
Representation
ProANN.....
Lnrsttad alaering)
Motf.Levdl
Homogolymer
sweet spot
Contrustive
Pathodoginal Repetition
Reduce foldabilitt
Diverse
Repseition Srawn Pyy
Broeting)
Reperition Snow RiX)
entropy
Repetitive
gram diveralty
Homopplymer diveriily
Steering strength
Bistogioat Ubitty UP0
PTM
Biolagioal Utilit
Figure 2: Overview of Utility-Controlled Contrastive Steering (UCCS). The diagram illustrates (I) the identification of pathological repetition modes (motif-level and homopolymer), (2) the con- struction of contrastive pairs (D*, D-) and computation of the steering vector v", and (3) inference- time control of hidden representations via ov to mitigate repetition while preserving biological utility.
4 UTILITY-CONTROLLEY CONTRASTIVE STEERING
Decoding heuristics such as sampling adjustments (Fan et al. 2018 Holtzman et al. 2020) or n- gram penalties (Kulikov et al. 2019) can reduce repetition, but often at the lowering struc- tural utility. We instead intervene at the representation level by extracting latent directions that specifically encode repetition. Our method Cetility-Controley structs paired datasets to disentangle repetition from utility, computes mean difference vectors from hidden activations, and injects them into selected layers to control generation. This provides a sim- ple, model-agnostic mechanism for repetition control without retraining sampling tricks. An overview the proposed approach shown in Figure
Contrastive Steering (UCCS) con-
4.1 PROBLEM DEFINITION
We define repetition control in protein language models (PLMs) the task of reducing degenerate sequence redundancy while preserving structural plausibility. Given pretrained PLM M and an optional prefix p. the model generates a sequence
(I1,...,IT). We evaluate each sequence with two complementary scores: a repetition score R(x), capturing motif and homopolymer level collapse, and utility score (x), reflecting biological plausibility based on AlphaFold confidence.
The atomic metrics underlying R(xx) and (x) were introduced in Section 3 Their aggregation, weighting, and biological justification are provided in Appendix The objective to design modification to M such that
min R(f(Mp)) s.t. U(f(M,))) 2 U(,
where is a small tolerance for utility degradation. This formulation is model and method-agnostee i.., any intervention from decoding heuristics to latent-space steering can be assessed under the same constrained optimization framework.
4.2 DATASET CONSTRUCTION FOR CONTRASTIVE STEERING
Motivation. Activation steering requires contrastive sets that differ along the target attribute while remaining aligned with potential confounders. However, in proteins, repetition and biological plau- sibility are naturally entangled: PLM generations with extreme repetition almost always exhibit low

---

Published as a conference paper at ICLR 2026
Repettion Metrica
Bicdogged Mertie
0.8
Cabbel
Oisherd
(a) Repetition metrics.
(b) Utility metrics.
Figure 3: Effect of contrastive selection. Raw negatives: ESM3 (P n = 10k); raw positives:
UniRef50 (P+, = 10k). After selection (n 100 each): (a) repetition separation enlarges, (b)
utility remains aligned.
structural confidence. Therefore, a naive comparison of positives and negatives would conflate R(x) (repetition) with (xx) (utility), making it unclear whether steering improves diversity, plausibility, or both. Our goal is to decouple repetition from utility so that the steering captures a clean signal of pathological repetition.
Candidate pools. We construct raw candidate (Appendix F.1 from two complementary sources: (i) natural proteins sampled from CATH, SCOP, and UniRef50,, and (ii) synthetic proteins generated by ESM3 and PROTGPT2. These provide sufficient coverage of low and high repetition regions across masked and autoregressive PLMs.
Scoring and filtering. Each sequence annotated with (R(x), (ax) as defined Section 3.3 To
weaken the intrinsic correlation between the two dimensions, first filter out sequences with U (x:) outside a tolerance band around the reference mean Ü, defined as the average utility score across the candidate pool (computed separately within each length bin; Appendix F.3). This step discards trivial cases with extremely poor folding, ensuring that utility is not the dominant separating factor.
Contrastive selection. We then select D (low repetition) and D (high repetition) by maximizing their separation in R(xr), the repetition score, subject to a utility constraint. Formally, let
AR EEED [R(x)] - EEED [R(z)],
AU E
[U(z)]
EEED [U(x)]],
where Uxx) is the biological utility score and is tolerance threshold. We then solve:
(D+ D-) arg max AR s.t. AUSE Se. D D
In practice, we instantiate this procedure using either Pareto-based frontier or a composite score ranking, with random sampling as a baseline. Algorithm details, length/utility binning, and pseu- docode are provided in Appendix F
This pipeline yields contrastive datasets that sharply separated in repetition but comparable in utility (see Fig. 3 for dataset-level statistics). As a result, the steering vectors extracted in Sec- tion 4.3 capture latent directions of pathological repetition rather than foldability artifacts, providing a priccplldd foundation for our method.
4.3 CONTRASTIVE STEERING VECTOR
Preliminaries on representations. Let (x:( E Rd denote the hidden representation at layer L. for token z; in sequence æ of length We define a sequence-level summary representation (x) differently for masked language models (MLMs) and autoregressive language models (AR-LMs)):
+ + (x)), (MI.M: mean-pooled across tokens),
xx)x)=
h4(2),
(AR-LM: last-token embedding).
This choice aligns with the training objectives of the two paradigms, ensuring (x) is faithful to how each PLM encodes global context: pooling reflects the bidirectional nature of MLMs,

---

Published as a conference paper at ICLR 2026
while last-token embeddings in AR-LMs are known to concentrate predictive information of the whole prefix (Xing et al. 2025).
Mean difference vector. Given contrastive datasets D (low repetition) and D (high repetition)
constructed in Section 4.2 we compute the steering vector as the difference mean sequence-level
representations:
2 E (x) (o
By construction, o² captures the latent direction that best separates high-R((a) from low-R(æ: quences, while controlling for (x). We refer to v contrastive steering vector in the layer L.
Injection. To control repetition during generation, we modify hidden states by injecting v the chosen layer L. At each decoding step, hidden activations are updated as:
(x) (x) aa- +
where E R is a scalar coefficient that controls the strength and direction of steering. In all main experiments, we set = 1 for simplicity, while ablation studies (Section (.3) investigate the sensitivity of results to different values. This procedure requires retraining and can be applied blug-and-play, making it an efficient and interpretable mechanism for repetition control under the framework defined in Section 4.]
5 EXPERIMENTS
5.1 SETUP
Datasets. We evaluate our method on three widely used protein sequence databases: CATH Knud- sen & Wiuf, 2010), UniRef50 (Consortium) 2024), and the SCOP database (Murzin et 1995). From each dataset, we randomly sample approximately 10,000 protein sequences, retaining only those shorter than 1024 residues. Detailed preprocessing steps are provided Appendix F
Models. We experiment with two classes of protein language models (PLMs): masked language models (MLMs) and autoregressive language models (AR-LMs). Specifically, we adopt ESM-
3 (HAYES ET AL. 2025) for the MLM setting and PROTGPT2 FERRUZZ ET AL. 2022 for the
AR-LM setting, with default generation configurations listed in Appendix
Baselines. We compare against three categories of baselines: (1) The original models without steering, using standard decoding with no intervention. (2) Decoding-level heuristics, where for ESM-3 evaluate entropy-based unmasking (from the original release), top-p, and temperature sampling. and for PROTGPT2 we implement top-p, temperature, repetition penalty, and n n-gram penalty strategies (details in Appendix G.2). (3) Mechanism-level interpretability methods, includ- ing neuron-based 2021]. (Appendix G.3).
DnneiivaaiinRRadfooi et al. 2017) and probe-based steering vectors [Elazar et al.,
Tasks. We consider two sequence generation paradigms: (i) unconditional generation (uniform- 50-512), where models generate novel protein sequences with length uniformly sampled between 50 and 512 residues; and (ii) conditional generation (prefix-O),, where models generate sequences conditioned on prefix extracted from natural proteins. For each sequence, the first 10% of residues are used as the prefix, and the remaining residues are generated with the total target length uniformly sampled between 50 and 512 residues. In both the MLL and AR-LM settings, we evaluate on both tasks, which balance computational efficiency and biological plausibility.
We evaluate outputs with two composite scores: the repetition score R(z), aggregating entropy, motif-level n-gram diversity, and homopolymer penalties; and the utility score Uxr, the average PLDDT and p'TM from AlphaFold confidence. Details of metric construction are given in Appendix
Metrics.
E Results are averaged over 5 random seeds with standard deviations.Fo each method, hyperparameters are selected according to the harmonic mean R(x)) and (xxx) (Appendix G.4). ensuring balanced trade-off between repetition reduction and biological plausibility.
Implementation Details. All experiments are conducted on an AutoDL server with NVIDIA RTX 5090 GPUs. We use the open-source ESM3 (esm3-sm-onl wa-sssmmopeeeye open-v1) and PROTGPT2 (ProtGPT2) mod- els from HuggingFace. Default generation parameters are listed in Appendix G.I repeated with random seeds, and report the mean standard deviation.
Each experiment
8

---

Published as a conference paper at ICLR 2026
Table 2: Pro1GPT2 results for two tasks with utility constraint: (a) unconditional generation, (b)
conditional generation.
CATH
UniRef50
SCOP
Method
R1 t
U
R
U 7
R
Unconditional generation
Original Model
0.728 0.728 20,010
0.62 0.621
+0,02*
20,000
0.621 +0.022 0.621+0.002
0. 0.728 40.000
0.621
Temperature Sampling
0.750
0.612
0..002
0.61220.09
0. 70022002
0.612=0my
Tampp-Snmplin
Top-p Sampling
40.04
0.608 20000
40.03
0.608 =.00
0.714 #0,03
0.608
No Repeat N-gram
0.730
0.613
go.00
0.613gomx
0. +0.0t0
0.613_omm
Repetition Penalty
10,005
0.622 v
0.780
0.622
0. 7.780 20000
0.622.10.018
Neuron Deactivation
0.719 4..19 20,017
0.61020001
0.718 2.718 20,019
0.614_o022
0. 20009
0.615
Probe Steering
0.722 20.017
0.607
0.732
0.624
10.009
0. 20.002
0.619_00016
(CCS()
0.845 10,010
0.711
10.004
0.824 10.012
0.703 10,018
0.835 10,008
0.722 10,008
(b) Conditional generation
Original Model
0.836 10,011
0.70410.025
0.704 10.025
0.836 0.836_0.018
0.70410.025
0.704 10.025
0.83610.018
0.704 0.70410.003 10,025
Temperature Sampling
0,847 1.,47 10,015
0.700
0.847 10.013
0.700 20.221
0.847.0.003
0.700 2.002
Top-p Samplling
20.014
0.710
0.827 20.014
0.710 0.710
0.827
0.710 0.710
No Repeat N-gram
10.011
0.708 10.025
0.828 10.013
0. 1,708 10,025
0,828 10,013
0. 0.708 10,005
Repetition Ponalty
0.803+0.015
0.716
0.843
0.7 0.71040.001
0.8 0.843
0.7 0.716
Neuron Deactivation
0.836
0.709 +0.012
0.828 20.08
0. 0.706go0015
0. 0.827
0. 0.701 10001
Probe Steering
20.019
0. 70220.000
0, 0.829
0. 1.600
0. g..04
0. 0.686_0.06
UCCS(ours)
0.877 +00009
0.743 #0.013
0. 28000
0.725
0.800
0.737 20.014 #0,014
ESM3-UCCS (122)
ESM3-UCCS (253)
PRO1GPT2-UCCS (68) PRO1GPT2-UCCS (236)
Figure 4: Generation showcase of UCCS. Unconditional protein generations with UCCS All cases exhibit high structural confidence (dominantly blue regions, pL.DDT > 90) and diverse folds, highlighting reduced repetition and improved foldability compared raw generations in Table 1
5.2 MAIN RESULTS
Tables 10 10 and Figures report results across both backbones (ESM3, PROTGPT2), datasets (CATH, UniRef50, SCOP), and tasks (unconditional vs. conditional generation). We evaluate repe- tition R(x) and biological utility Uzx), marking U 2 Uong with a checkmark. For brevity, the main text shows scatter plots for ESM3 (Figure 8 while full PROTGPT2 results are in Appendix H.T
UCCS achieves the best trade-off. Our method consistently improves R(x)) while maintaining enhancing U(x) across all settings. As demonstrated in Table 10(a), ESM3, UCCS yields large gains in R(x) compared to the original model (e.g., +55% under unconditional generation on CATH), while preserving or even slightly improving U(x)((ee Fig 4 and Appendix K for visual- ization cases). On PROTGPT2(see Ta)le UCCS achieves the highest repetition control (up to +20% relative to temperature sampling) and the only method that consistently satisfies the utility constraint across datasets and tasks. These results confirm our central claim: dataset- -aiiseet-guiedd con- trastive steering offers a plug-and-play yet robust mechanism to reduce pathological repetition with- out compromising foldability. Compared to other mechanism-level approaches. UCCS explicitly decouples utility from repetition when constructing the mean- -efu-ndfffeeecce vector(section F ensuring that the injected direction corresponds to genuine repetition signal rather than entangled features. This alignment explains why UCCS provides stable gains in R(x) without sacrificing (xx).
2 Notes. R: repetition score; U: biological utility. We annotate cells with when U > Uoni (utility constraint satisfied). Decoding-level heuristies and original models do not require supervision; therefore their hehavior remains consistent across datasets. particular, the choice dataset does not affect unconditional generation.
"enngths shown in parentheses after each case IDD.

---

Published as a conference paper at ICLR 2026
uncondtoona: Rid vs. UNO
Xoo 000
centiiied
3.1-
0:66
All
6.75
600-
3.00
6.0
6.0
0.50
555
3.50
3.03
0.04
0.00
44 01
0.5
(a) Dataset sampling (ESM3)
(b) St>Seeering strength (ESM3)
(c) Layer ablation (ProtGPT2)
Figure 5: Representative ablations. (a) dataset sampling, (b) steering strength, (c) steering layer.
Full results (both models, unconditional/condtiinaa)) in Appendix H.2
Decoding heuristics reduce repetition but harm foldability. Sampling strategies such as increas- ing temperature or applying top-p thresholds can raise R(x) substantially (c.g., temperature sam- pling at T = 1.3, using entropy based sampling for ESM3, as shown in Figure g a()). but they consistently reduce (zx), indicating worse structural reliability. The reason is that such heuristics operate only at the output distribution: they encourage surface-level diversity by flattening token probabilities, yet remain agnostic to the internal relationships among residues. As a result, higher temperatures often produce locally diverse but globally incoherent motifs, disrupting the residue patterns that underlie stable protein folds. This suggests that token-level diversity does not automat- ically translate into structural plausibility, and explains why decoding heuristics, while effective in NLP, can be detrimental in protein generation. In contrast, UCCS directly disentangles repetition from biological utility, steering generation along directions that preserve foldability while reducing degeneracy.
Mechanism-level baselines are unstable. Neuron deactivation and probe steering sometimes im- prove R(z), but gains are inconsistent and often come at the cost of reduced Uxx). For instance, in the unconditional task on F.M3,, neuron deactivation shows some promise: across all three datasets it moderately increases both R and U (see Figure 8 (b),)), suggesting that suppressing small sub- set of high-activation neurons can reduce degeneracy in this setting. However, the same method performs poorly in the conditional task: on ESM3-SCOP, it lowers repetition and thus raises R, but simultaneously damages U, leading to worse overall trade-offs (Figure ())). Probe steering is even less reliable: while occasionally yielding minor improvements, its effects fluctuate strongly across datasets and settings, offering no consistent pattern of benefit. These observations reinforce our claim that mechanism-level baselines, which manipulate isolated features without explicitly de- coupling repetition from utility, lack robustness compared to dataset-guided contrastive steering.
5.3 ABLATION STUDY
To assess the contributions of different components, we conduct ablations along three axes: (i) con- trastive dataset construction, (ii) stecring strength a, and (iii) injection layer L. Across both ESM3 and ProtGPT2 backbones, three consistent trends emerge: (1) structured dataset construction strate- gies (Pareto and Composite) outperform simple random sampling (see Fig. Sal detailed in Fig. TO] and Tables 16 in Appendix H.2 ; (2) moderate steering strength (a z 2) yields the most balanced trade-off (see Fig. 5b) with full results in Figs. Ti 12 and Tables 17 18); and (3) effective steering typically arises in later layers, with the optimal range depending on model depth (see Fig, [c]] with per-layer curves and tables in Figs. 13 and Tables 20). In practice, Pareto/Composite sampling not only improves mean performance but also reduces variance across seeds, especially in the unconditional setting. For a. both backbones show a unimodal trend: performance improves up to a 2, after which repetition control saturates while utility declines, with conditional sam- pling remaining more robust. For layer ablations, ESM3 benefits most from steering at the final 2-3 layers, while ProtGPT2 shows a smoother optimum in the last third of its stack, consistent with their architectural differences. Comprehensive visualizations and results are provided in Appendix H.2
6 CONCLUSION
We conducted the first systematic study of pathological repetition in protein language models (PLMs) and introduced metrics framing it as constrained optimization problem. To tackle this, we proposed Utility-Controlled Contrastive Steering (UCCS), which disentangles repetition from structural utility via dataset-guided activation edits. Experiments across diverse PLMs and datasets show that UCCS mitigates degeneracy while preserving foldability, outperforming existing heuris- ties and interventions.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
Lijie Hu, Jiahao Zhang, and Zeqing Zhang are supported by the funding BF0100 from Mohamed bin Zayed University of Artificial Intelligence (MBZUAI). Di Wang is supported in part by the funding 111111669-11-01.GG//311111
FCC/1/5940-20-05, FCC/1/5940-06-02,, and King Abdullah University of Science and Technology (KAUST) Center of Excellence for Generative AI, under award number 5940 and a gift from Google.
ETHICS STATEMENT
This work does not involve human subjects, personally identifiable information, sensitive user data. All datasets used (CATH, SCOP, UniRef50) are publicly available protein sequence databases. Generated sequences are synthetic and intended solely for methodological evaluation; they are not directly deployable for therapeutic or industrial applications.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure reproducibility. All datasets used are public and detailed pre- processing instructions provided in Appendix Model configurations, hyperparameters, and evaluation metries are fully documented in Sopenniics/G.Ilaaa [n Pseudocode for dataset construc- tion and steering is included in Appendix F.4 All code, along with configuration files and scripts for reproducing figures and tables, will be released upon publication.
REFERENCES
Bo Chen, Xingyi Cheng. Pan Li, Yangli-ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, al. xtrimopglm: unified 100b-scale pre-trained transformer for deciphering the language of protein. arXiv preprint 240::06999 2024.
Xin Chen, Junchao Wu, Shu Yang, Runzhe Zhan, Zeyu Wu. Ziyang Luo, Di Wang. Min Yang, Lidia S Chao, and Derek Wong. Repreguard: Detecting Ilm-generated text by revealing hidden representation patterns. Transactions of the Association for Computational Linguisties, 13:1812- 2025.
The UniProt Consortium. Uniprot: the universal protein knowledgebase in 2025. Nucleic Acids Research, 53(D1))D609 D617. 11 2024. ISSN 1362-4962. doi: 10. 1093////ar/g 1010. URL
https://dot www.ooo.................o....... 1093 /ar////aarrra nar gkae 010
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 8493-8502 2022.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals, 2021. URL 00995
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, 2018.
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is a deep unsupervised language model for protein design. Nature Communications, 13(1), July 2022. ISSN 2041-1723. doi: 10.1038/ s41467-022 3200777.. URLbttp:://axx http://xxxxii...ooo///o www.:ooo......oo..ooooooo 10 101.1100184446700000000 0200767--222220---7-77- 32007-7
Brian D. Freibaum, Yubing Lu, Rodrigo LonezeGonzalez, Nam Chul Kim, Sandra Almeida, Kyung- Ha Lee, Nisha Badders, Mare Valentine, Bruce L. Miller, Philip C. Wong, Leonard Petrucelli, Hong Joo Kim, Fen-Biao Gao, and J. Paul Taylor. Ggggee repeat expansion in c9orf72 compro- mises nucleocytoplasmic transport. Nature, 525(7557):299911115555 133, A August 2015. ISSN 1476-4687. doi: 10. nnntturee 14974. http:://xx..o...oooo h|taa//eooo/////////////o/o///o0

---

Published as a conference paper at ICLR 2026
Jennifer R. Gatchel and Huda Y. Zoghbi. Diseases of unstable repeat expansion: Mechanisms and common principles. Nature Reviews Genetics, 6(10):743-775 October 2005, ISSN 1471-0064.
doi: 10.1 URL http: doi. wttp:o..00/..../o...... nrql6991n/
Alex Graves. Generating sequences with recurrent neural networks, 2014.
URL https:/ //
arxiv. wtx.v.org/abs// 308 0850
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with language model. Science, 387(6736):850 8588858 2025.
Daniel Hesslow, Niccoló Zanichelli, Pascal Notin, lacopo Poli, and Debora Marks. Rita: a study on scaling up generative protein sequence models. arXiv preprint arXiv:2205.05778) 2022.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https //openzeview net http:///ooooomomdddooooo-ee-re
Yihuai Hong, Yuelin Zou, Lijie Hu, Ziqian Zeng, Di Wang, and Haiqin Yang. Dissecting fine-tuning unlearning in large language models. arXiv preprint 2xXiv:2400006666, 2024.
Jingyu Hu, Shu Yang. Xilin Gong, Hongming Wang, Weiru Liu, and Di Wang. Monica: Real- time monitoring and calibration of chain-of-thought sycophancy in large reasoning models, arXiv preprint arXiv:2511.064/1 2025.
Lijie Hu, Liang Liu, Shu Yang, Xin Chen, Hongru Xiao, Mengdi Li, Pan Zhou, Muhammad Asif Ali, and Di Wang. A hopfieldian view-based interpretation for chain-of-thought reasoning. arXiv preprint arXiv:2406. 12255, 2024.
Long- Kai Huang, Rongyi Zhu, Bing He, and Jianhua Yao. Steering protein language models, 2025. URL htttps://axxi..ooott org/ absa /2099. 07983
Michael P. Hughes, Michael R. Sawaya, David R. Boyer, Lukasz Goldschmidt, Jose A. Rodriguez, Duilio Cascio, Lisa Chong, Tamir Gonen, and David S. Eisenberg. Atomic structures of low- complexity protein segments reveal kinked sheets that assemble networks. Science, 35966376): doi: 10.1 http:////ccincceaa6699999 URL https: www.wciiec...rrg..oooooooooom abs/10..126/
698-701, 2018
aan6398
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https:: htt.:tss//peeeeeeew..etttt net/ htt///orrmmiim
6tCKff8 jri
Aya Abdelsalam Ismail, Tuomas Oikarinen, Amy Wang, Julius Adebayo, Samuel Don Stanton, Hec- tor Corrada Bravo, Kyunghyun Cho, and Nathan C. Frey. Concept bottleneck language models for protein design. In The Thirteenth International Conference on Learning Representations, 2025.
URL https:://// http::ttp/////ee/// http://ppenrreeee www.eeeeiwwww... net forum? iidiid--tt 9CFhhOF
Xinyan Jiang, Lin Zhang, Jiayi Zhang, Qingsong Yang, Guimin Hu, Di Wang, and Lijie Hu. Msrs: Adaptive multi multi-subssaae representation steering for attribute alignment in large language models. arXiv preprint arXiv:2508. 10599, 2025.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-558))
2021.
Andrey V. Kajava. Tandem repeats in proteins: From sequence to structure. Journal of Structural Biology, 179(3):279 288, September 2012. ISSN 1047-8477. doi: 10.1016//.shh.001 1.08.009. URL http::/ob www.ooorr/.o.................... 00. 101 6/3.6.15..01. jso. 201 08.08..00 0009
Michael Knudsen and Carsten Wiuf. The cath database. Human Genomics, 4(3):207, 2010 ISSN 1479-7364. doi: 10.1186/1479- 737-9-73334-----11 4-7-207.
URL http:://dd......ooo/110.......
www.
1479-7364-4-3-20
12

---

Published as a conference paper at ICLR 2026
Ilia Kulikov, Alexander Miller, Kyunghyun Cho, and Jason Weston. Importance of search and evalu- ation strategies in neural dialogue modeling. In Proceedings of the 12th International Conference on Natural Language Generation, pp. 76 87, 2019.
Hongji Li, Manjiang Yu, Priyanka Singh, Xue Li, Di Wang, Lijie Hu, et al. Towards reasoning- preserving unlearning in multimodal large language models. arXiv preprint arXiv:2512.17911 2025.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity objec- tive function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110-119, San Diego, Califor- nia, June 2016. Association for Computational Linguistics. doi: 10.18653/w1/N16. 1014. URL
Mingchen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Huiqun Yu, Ziyi Zhou, Wanli Ouyang, Bingxin Zhou, Pan Tan, and Liang Hong. ProSST: Protein language modeling with quantized structure and disentangled attention. In The Thiitt--ightth Annual Conference on Neural Information Pro- cessing Systems, 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos San- tos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Ali Madani, Ben Krause, Eric R. Greene, Subu Subramanian, Benjamin P. Mohr, James M. Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z. Sun, Richard Socher, James S. Fraser, and Nikhil Naik. Large language models generate functional protein sequences across diverse fam- ilies. Nature Biotechnology, 4188):1099 January 2023. ISSN 1546-1696. doi: 10.1038/
941587-022-01618-2
Valerio Mariani, Marco Biasini, Alessandro Barbato, and Torsten Schwede. Iddt: local superposition score for comparing protein structures and models using distance differ-
ence tests. Bioinformatics, 29(21):2722-2728 August 2013. ISSN 1367-4811. doi: 10.
1093/hoinformaticc/ht477.... URL /ttt:://t// dww.k.oo.ioooooooo wtw.o.oorg/..... 10t3///11...0/////////// format css/ bt.
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Locally typical sampling. Transac- fions of the Association for Computational Linguisties, 11:102-121, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems. 35:17359-1732 2022.
Alexey G. Murzin, Steven E. Brenner, Tim Hubbard, and Cyrus Chothia Scop: structural classifi-
cation of proteins database for the investigation of sequences and structures. Journal of Molecular Biology, 247(4):536 April 1995. ISSN 0022-2836 doi: 10. 1011/00002---210 101006//0002-28660000111444 2415144 URL http: http//xx.ooo.... org. 10. 101110011000000 2836 (05) 80134-2
Divya Nori, Shivali Singireddy, and Marina Ten Have. Identification of knowledge neurons in protein language models, 2023. URL
10770
Nithin Parsan, David Yang, and John Yang. Towards interpretable protein structure prediction with sparse autoencoders, 2025. URL
Alec Radford, Rafal Jozefowicz, and llya Sutskever. Learning generate reviews and discovering sentiment, 2017.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steer- ing llama via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15504-15522, 2024.
Elana Simon and James Zou. Interplm: Discovering interpretable features in protein language mod- els via sparse autoencoders, 2024. U.l.htt..e///aarrrrrrr////aa.2.. 12101
13

---

Published as a conference paper at ICLR 2026
Yawen Sun, Rui Wang. Zeyu Luo, Lejia Tan, Junhao Liu, Ruimeng Li, Dongqing Wei, and Yu-Juan Zhang. Esm2_amp: an interpretable framework for protein protein-protein interactions prediction and biological mechanism discovery. Briefings in Bioinformatics. 26(4). July 2025. ISSN 1477-4054.
doi: 10. 1090//bb/bbbbff544. htth:ttp:// /xtp:://xxx.o doi .rg. 1093 ht199////bbb//
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024. URL
https://axx: /ttp:://axxvvva owg/abs.//aaooooooooo.oooo...ras 2308. 2ttp://33...222.3...
Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners. arXiv preprint arXiv:2402. 18567, 2024.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference Learning Rep- resentations, 2020. URL http:ttt//pprrreeeeeese http://eewww..ttteeerrrroo/e//ee
of Jinming Xing, Dongwen Luo, Chang Xue, and Ruilin Xing. Comparative pooling mech- anisms in llms: A sentiment analysis perspective, 2025. URL http:ss///rxiiv.rr//////
241 14654
Shu Yang, Shenzhe Zhu, Liang Liu, Lijie Hu, Mengdi Li, and Di Wang. Exploring the personality traits of i1ms through latent features steering, arXiv preprint 1rXi::2410010863, 2024.
Junchi Yao, Shu Yang, Jianhua Xu, Lijie Hu, Mengdi Li, and Di Wang. Understanding the repeat curse in large language models from feature perspective. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 7787-7815, Vienna, Austria, July 2025, Association doi: 10.18653/v1/2025 fin.:///...nnnns-cl..... fin.iiiinniisssii 406/
for Computational Linguistics. ISBN 979-8-89176-256-3 406.
Manjiang Yu, Hongji Li, Priyanka Singh, Xue L.,, Di Wang, and Lijie Hu. Pixel: Adaptive steering via position wise injection with exact estimated levels under subspace calibration. arXiv preprint ar1iv:2510. 102000 2025.
Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatiics 57(4):702-710, October 2004. ISSN 1097-0134. doi: 10. 20.104 URL hdtt::/ dei. wtt.::i.....1111.11...00000..000 prot 20264
Zhuoran Zhang, Yongxiang Li, Zijian Kan, Keyuan Cheng, Lijie Hu, and Di Wang. Locate-then-edi for multi-hop factual recall under knowledge editing. arXiv preprint arXiv:2410.063)) 2024.
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
To complement the main related work in Section 2 provide additional discussions that span three interconnected areas: (i) explainable Al (XAI) techniques for interpreting and steering protein language models (PLMs); (ii) repetition and degeneration in natural language generation, including decoding and representational interventions.
XAI for protein language model. Recent studies have begun to apply explainable AI (XAI) meth- ods to uncover how protein language models (PLMs) encode biological knowledge. Attribution and attention-basee analyzes demonstrate that PLMs highlight residues at functional motifs or binding sites, offering interpretable evidence of biologically meaningful signal capture (Sun et al. 2025). To probe model internals more systematically, researchers have leveraged sparse autoencoders to decompose PLM representations into human-interpretable features corresponding to structural or functional categories (Simon & Zou| 2024 Parsan et al.) 2025), and neuron-level studies reveal "knowledge neurons" associated with enzyme classes and other biochemical properties (Nor et al, 2023). Beyond post-hoc interpretation, intervention-based strategies emerging: recent work shows that editing activations can steer PLMs toward desired sequence properties (Huang et al.) 2025), while concept bottleneck architectures introduce interpretable intermediate variables that able controllable protein design (Ismail et al. (023). These advances illustrate growing toolkit for interpreting. probing, and steering PLMs, laying the groundwork for more transparent and biologi- cally informed protein design.
Repetition and degeneration in NLP generation. While not discussed in detail in the main text, repetition has long been recognized as failure mode in natural language generation. Here, we provide a more focused overview of this literature, which informs our approach to repetition trol in protein sequence generation. Repetition in large language models often leads to low-entropy loops or degenerate text with reduced diversity (Holtzman al.. 2020 Welleck et al., 2020). To mitigate these issues, the NLP community has explored decoding strategies such as top-k or nu- cleus sampling, n--gram blocking, and temperature scaling (Graves) (014]] Fan et al. 2018 Kulikov et al. 2019). More recent work investigates internal mechanisms of repetition and proposes neuron- level interventions (Yao et al.) 2025). Yet these techniques were developed in the context of natural language, where repetition mainly reduces readability. In proteins, by contrast, uncontrolled repe- tition can collapse structural folding and render sequences biologically non-functional. While most NLP work has focused on decoding-levar strategies, another line of research investigates steering generation through internal representations, which inspires our approach.
B NOTATION
We summarize here the notations used throughout the paper.
C REPETITION IN PLM: EMPIRICAL OBSERVATION
C.1 RAW POS vs. PLM RAw NEG
As previewed in Table of the Introduction, here we provide a more detailed analysis of raw positive pool P+ (natural proteins) and negative pool P (PIM --gnerrret proteins). Natural proteins ex- hibit rich compositional balance and non-trivial motif structures, while PLM generations frequently collapse into homopolymers (e.g., long stretches of alanine or glycine) or short repetitive motifs. Structural prediction with AlphaFold further reveals that natural proteins achieve high pLDDT con- fidence, whereas PLM outputs often result in unstable folds with poor confidence, confirming their limited biological plausibility.
C.2 TOKEN FREQUENCY DISTRIBUTIONS: PLM vs. NATURAL PROTEINS
To further support our observation of degeneracy in PLM outputs, we compare the overall amino acid frequency distributions between natural proteins (P+) and PL.--generate sequences (P-).
15

---

Published as a conference paper at ICLR 2026
Symbol
Description
Basic objects
Amino alphabet
r (
Protcin soquence of length T. with at
T
Sequence length
p(a)
Empirical unigram distrihution of residue E A in a
Models and generation
M
Pretrained protein language model (PLM)
Prediction of masked token (MLM setting)
M(a, x2))
Prediction of the token (AR setting)
p
Prefix used for conditional generation
f
Modification function applicd to M for repetition control
Optimization problem
problem
E
Tolerance for utility degradation miny (()(M,))) p)) s.t. U(f(M. p)
Objective
Metrics
R(x)
Repetition score integrating Hamem (2r,, Distinct-2/3, and Rapoty (2)
((r)
Utility score combining (LDDT((r) and pTM((r
Representations and steering
(a(a)) e Rd
Hidden representation layer L for token
(*))
Sequence-level summary of hidden states (mean for MI.,, last token for AR)
Contrastive steering voctor, EzeD- - EreD+ (x))
Steering strength coefficient Steered hidden state: 1. o
(E()))
Datasets and construction
pt
Raw positive (low-repctttio) and negative (high-repetition) candidate pools
D+. D
Filtered contrastive datascts from p+. P
AR, AU
Differences in repetition/utiiity between D and D
Table 3: Summary of notation used in the paper.
Figure presents grouped bar charts of amino acid compositions, where the horizontal axis enumer- ates amino acid types (standard 20 amino acids, represented by their single-letter codes), and, for each amino acid, different colored bars correspond to different datasets (POS vs. NEG), Natural pro- teins show a relatively balanced usage of amino acids across different physicochemical categories, while PLM generations are strongly skewed. In particular, in ESM3 outputs alanine (A) accounts for more than 40% of residues and leucine (L. is also overrepresented, whereas in ProtGPT2 out- puts alanine and proline (P) are highly enriched. This compositional collapse corroborates the repetition phenomena reported in the main text, and motivates the definition of our repetition-related evaluation metries (Hnorm. Distinct-n, Rapoly)
Prgueettppreennts
D STRUCTURAL UTILITY METRICS
D.1 PER-RESIDUE LDDT AND PLDDT.
Following AlphaFold Jumper et al. 2021), we measure local structural accuracy using the noon-superposttionbbaeee based) local Distance Difference Test (IDDT) MMariani et al., (013) score. For residue i, let Nr be the set of residues whose C. atoms lie within a cutoff rab in the reference structure (and jl > A to ignore trivial neighbors along the chain). Let dæ and d' denote the predicted and reference inter-residue distances. The IDDT residue averages tolerance tests
16

---

Published as a conference paper at ICLR 2026
Compasition by Dataset
0.
ProtGPT2
CATH
Uniref5o
SCOP
0.3
0.1
0.0
Figure 6: Grouped bar charts of amino acid frequency distributions across natural proteins (CATH, Uniref50, SCOP) and PLM-generated proteins (ESM3, Pro1GPT2). Natural datasets show balanced usage across amino acids, whereas PLM generations are dominated by a few residues (e.g., alanine (A) and leucine (L) in ESM3; alanine (A) and proline (P) in ProtGPT2).
of thresholds T (0.5, 1.0, 4.0, A:
IDDT; = INIT
1
ΣΣ FET
1(d), - -Idyy--d <T).
The chain-level IDDT the mean over residues, IDDT + Σ, IDDT, AlphaFold's predicted IDDT (pLDDT) is obtained by confidence head that outputs a categorical distribution over binned IDDT values; PLDDT the (scaled) expectation:
1 Σ PLDDT, 100). PLDDT T.
pLDDT, 100 I
D.2 GLOBAL TOPOLOGY TM-SCORE AND PTM.
Global fold accuracy is summarized by the TM-score (Zhang & Skolnick 2004) for an optimal alignment between predicted and reference structures:
1 1 L
1
TM(A)
do(T.) 1.24 VT. 1.8,
2 do(L)
where d,(,A) is the aligned C. distance at residue and (L)) is the TM normalization. AlphaFold's predicted TM-score (pTM) uses the predicted alignment error (PAE), i.e., distribution pair- wise alignment deviations. Let [[d,] denote the expected aligned error at residue i under the PAE. Then pTM is expectatioo/relxxation of TM-score:
TM
Σ
[0, 1].
1
(
(() do(L)
Intuitively, PLDDT captures local geometric confidence while pTM captures global topology confi- dence.
D.3 OUR OPERATIONAL UTILITY.
In our experiments use
U(x) (PLDDT(()) / 100 + PTM(2)).
ie., the mean of (scaled) PLDDT and pTM, to summarize both local and global structural plausibility for generated sequence
17

---

Published as a conference paper at ICLR 2026
E
METRIC DETAILS
This appendix provides detailed definitions and motivations for the metrics used throughout the paper. While the main text only presents their compact forms (Section 4). here we explain the construction rationale and discuss why they provide reliable signals for evaluating repetition and biological plausibility protein sequences.
E.1 REPETITION SCORE CONSTRUCTION
We define the overall repetition score R(x) for protein sequence æ as normalized aggregation of three complementary measures:
R(x)= = Hnorm (2) (Distinntn22(() Distinct-3(x)) + Rapoty(x))
Entropy normm) Normalized unigram entropy captures the global residue distribution. Low tropy indicates collapse into a small subset of amino acids, hallmark of low-complexity sequences.
n-gram diversity (Distinct-n). Distinet-2/3 measures local motif diversity. Recurrent dipeptide or tripeptide fragments (e.g., AGAGAG, PQAPQA) lead to low Distinct-n, signaling motif-level repe- tition.
Homopolymer diversity RRapoly). This score penalizes runs of length 2 4 for any residue, detect- ing extended homopolymerization (e.g., AAAAAA). Such runs are a canonical form of pathological collapse in PLM generations.
Why an average? Each component captures a different failure mode: entropy (global collapse), Distinet-n (motif recurrence), and Rhpaly (long runs). By averaging, we obtain a balanced metric that is sensitive to diverse artifacts without overemphasizing any single mode. Empirically, we observed that sequences with high R(x) align with natural proteins, while PLM-generated sequences with low R(x) exhibit visible collapse patterns (see Appendix F.2
E.2 UTILITY SCORE CONSTRUCTION
To measure biological plausibility, we define the utility score as
(iz) (PLDDT(2)/100- + 1010000-pMM// pTM(()))
where PLDDT is the residue-level local confidence, and pTT' estimates global fold topology.
Why combine PLDDT and PTM? Relying on a single metric can be misleading:
PLDDD is highly sensitive to local residue environments but blind to global misfolding.
pTM captures overall fold similarity but may overestimate local confidence.
By averaging them, obtain score that balances local accuracy and global topology.
E.3 SUMMARY
In summary:
The repetition score R(x)) integrates entropy, motif-level diversity, and homopolymer penalties, offering a comprehensive view of pathological repetition.
The utility score Uzx) combines local (pLDDT) and global (pTM) structure metrics, pro- viding a balanced estimate biological plausibility.
Together, these scores allow to construct contrastive datasets that differ primarily along the rep- ctition dimension while remaining matched in structural plausibility, enabling effective application of our proposed ttllit--Controlled Contrastive Steering (UCCS).
18

---

Published as a conference paper at ICLR 2026
F DATASET CONSTRUCTION DETAILS
We describe the preprocessing steps, filtering rules, and sampling strategies used to construct the datasets evaluated Section 5
F.I DETAILS OF RAW DATASET
F.1.1 POSITIVE DATASET
We randomly sample 10,000 sequences from widely used protein databases including CATH (Knudsen & Wiuf, 2020) Non-redundant Subset), UniRef50 (Consortium) 2024), an SCOP (Murzin et al., 1995). All sequences are annotated along two dimensions: (I) Repetition, measured by R(x) and its components (see Section 3.3) and (2) Biological Utility, measured by
U(r), combining pLDDT and pTM((r (Appendix)
F.1.2 NEGATIVE DATASET
Negative examples synthetic rather than natural proteins. Two independent generation proce- dures were used:
ESM-3 unconditional sampling: sequences are generated by unconditional sampling from the pretrained ESM-3 model using the official inference client, with random target lengths (50 T < 1024).
ProtGPT2 autoregressive sampling: sequences are generated using the ProtGPT2 lan- guage model, from empty prefix, with random target lengths (50 < T 1024).
Together, these sources provide a diverse negative pool with low or ambiguous biological plausi- bility. Filtering ensures all negatives pass minimal length and pLDDT constraints while exhibiting low entropy. Notably, the synthetic negatives tend to display much higher repetition levels than natural positives, making them complementary counterparts for balanced contrastive evaluation.
F.2 STATISTICS
To ensure that our repetition and structure metrics are biologically meaningful rather than ad-hoc definitions, we compute them on both natural proteins (positive p+) and PLM-generated quences (negative sets P-). This analysis serves a dual purpose: (i) characterizing the distributional properties our datasets, and (ii) validating that Hnorm. Distinct-2/3, Rapoly. and structure-based scores (pLDDT, pTTM reliably distinguish repetitive, degenerate PLM outputs from diverse, fold- able natural proteins.
Tabular Summary.
We report the following metrics consistently with Appendix
Hnorm (r): normalized entropy.
Distinct-2/3- proportion of unique n-grams,
Rnpoty (z): 4-gram polynomial entropy,
PLDDT(2)) (x) and pTM((r): structural confidence scores.
Density distributions. For each metric, we plot kernel density estimates (KDEs) for both positive (natural proteins) and negative (PLM generations) datasets. To enable comparison across disparate scales, we report relative density, i.c., each KDE normalized such that its maximum equals These visualizations highlight clear contrasts between natural and synthetic sequences: (i) repe- tition metrics (Hoorms Distinct-2/3, Rapoly: see Figure D indicate that negatives exhibit higher redundancy and lower lexical diversity; and (ii) structure utilities (pLDDT, pTM; see Figure 7 cleanly separate natural sequences shifted toward high-confidence regions from PLM outputs with degraded or unstable folds.
19

---

Published as a conference paper at ICLR 2026
Table 4: mean and variance of repetition and bio-utility metries across pos (natural proteins) and neg (plm generations).
metric
ESM (neg)
ProtGPT2 (neg)
CATH (pos)
Uniref50 (pos)
SCOP (pos)
Haorm
0.436 + 0.149
0.711 0.182
0.917 0.041
0.926 0.042
0.924 0.036
Distinct-2
0.144 + 0.092
0.389 + 0.243
0.760 + 0.111
0.612 0.150
0.727 + 0.125
Distinct-3
0.272 + 0.156
0.625 0.313
0.974 + 0.031
0.949 0.045
0.973 + 0.024
Unpoly
0.516 + 0.252
0.939 + 0.128
0.994 + 0.022
0.996 0.016
0.999 + 0.006
pLDDT (0-100)
63.45 + 19.75
75.51 + 20.83
89.94 + 8.56
90.14 10.15
91.62 1 9.22
рТM
0.427 + 0.194
0.514 0.269
0.763 + 0.154
0.767 0.197
0.811 + 0.146
natural proteins; neg: plm generations. repetition metrics (c.g., Haorm- distinct-n, Rapoly) consis- tently separate pos vs neg, validating their effectivvness.
notes, pos:
Interprefation of Results. The quantitative results in Table 4 reinforce the patterns observed in the density plots. Synthetic negatives generated by PLMs (ESM-3, ProtGPT2) exhibit substantially lower values across all repetition setrriss-includinn Hnorm- Distinct-2/3, and R inniiiatin higher redundancy and reduced sequence diversity. In contrast, natural datasets (CATH, UniRef50, SCOP) consistently score higher entropy and diversity, reflecting richer lexical and structural complexity.
From structural perspective, natural proteins achieve significantly higher pLDDT and pTM scores, reflecting greater foldability and evolutionary plausibility. Meanwhile, synthetic negatives not only show lower means but also greater variance, suggesting instability and reduced structural confidence.
Figure I demonstrates that our repetition metries not only distinguish natural versus synthetic pro- teins, but also reveal distinct failure modes across PLM architectures. Hnorm globally separates both PLMS from natural proteins, confirming the presence of pathological repetition in general. Distinet- 2/3 expose local motif-level redundancy specific to autoregressive models like ProtGPT2 ("AGA- GAG patterns), whereas Ahpoly captures long-run homopolymer collapse ("AAAAAA... unique masked LMs like ESM3. Overall, both PLMs diverge sharply from natural proteins in global repetition metrics, underscoring that pathological repetition is a shared phenomenon mani- fested differently across model classes.
F.3 DETAILS OF THE RANKING ALGORITHM
Defining metries. In our dataset construction, we adopt the repetition score R(x) as the primary metric for distinguishing between positive and negative samples. Unlike raw repetition measures (e.g., homopolymer runs), the repetition score is composite metric that integrates three comple- mentary components:
Normalized sequence entropy H norm (z);
Average 7-gram diversity 1 (Distinct-2(xx) + Distinct-3(x)):
Homopolymer diversity score R (x), which penalizes runs of length 2 4.
Formally,
R(x)= = (norm + (Distinct22(x) Distinct-3(x)) + Ripoly (z))
A higher R(x) indicates greater sequence diversity and reduced redundancy, which aligns with bi- ological plausibility. Positive samples are therefore expected have higher R(x), while negatives tend to exhibit low diversity and strong repetitive patterns.
The utility score (xx) reflects structural plausibility, computed as:
U(x) (PLDDT(ss//10 + pTM())))
20

---

Published as a conference paper at ICLR 2026
Structure utilities: KDE + JS divergence
CATH
UniRe150
SCOP
ESMB
Pro1GPT2
1.0
Relative density
0.5
0.0
20
80
80
100
0.0
0.2
0.4
0.6
0.8
1.0
PLDDT
рTM
Jensen-Shannon divergence
CATH
0.3
UniRef50
Dataset A
0.2
SCOP
ESM3
0.1
ProtGPT2
0.0
CATH
UniRelGo
SCOP
ESM3
PratGPT2
CATH
UniRet50
SCOP
ESM3
ProtGPT2
Dataset B
Dataset
Figure 7: Structure utilitics: KDE + JS divergence. Top:
Kernel density estimates (relative den- sity, peak = 1) of structure-based metries (pLDDT, pTM) across natural (CATH, UniRef50, SCOP) and PLM-generated (ESM3, ProtGPT2) datasets. Bottom: Pairwise Jensen-Shane
divergence heatmaps between datasets. High divergence between PLM and natural sets confirms that struc- ture metrics (U(x) effectively separate foldable natural proteins from unstable PL.M generations. Natural datasets cluster tightly, while PLM sets enverge-indicate reduced structural reliability. Together with repetition metrics, these results demonstrate that our metric suite captures both struc- tural and lexical degradation in PLM outputs.
where PLDDT reports residue-level local confidence and pTTM measures fold-level topology (Jumper ct al. 202I)
F.3.1 LENGTH AND BIO-UTILITY BINNING
Before ranking, bucketize sequences along orthogonal axes:
Length bins: determined by pre-defined thresholds (50, 129, 257, 513, 1025). Each se- quence is assigned to the corresponding length interval.
2. Quantile-based bio-utility bins: within each length bin, sequences are further split into Q bins by quantiles of (xx (using either pTM pLDDT depending on availability).
This two-level bucketing ensures that positives and negatives are compared on matched subsets, removing trivial confounders such sequence length.
21

---

Published as a conference paper at ICLR 2026
F.3.2 PARETO-BASER BASED SELECTION
We use an approximate two-objective selection that operates within each length-utility bin. Let +1 for positive candidates and & = -1 for negative candidates. Inside the current bin, we robustly scale scores using
mediangeB 2(z)
[S max 2xx), $}
with small for stability. Define oriented objectives:
(repetition-ooiinted)) A(x) = s-19I( [x)))
B(x) - (1))((
(utility alignment).
Here Une is the mean (-) of the opposite side the same bin.
Front extraction (hin quota o). Rather than running full nondominated sort, we apply single- pass frontier approximation: (i) sort all candidates in the bin by the pair (A(x), B(x)) in descending lexicographic order; (ii) scan once in that order, keeping any point whose Bxx) is within e of the running maximum Buax (updating Bmax as we go), which yields high- envelope under high A; (iii) if the envelope exceeds the quota, keep the top q by A(x) + B(x); if it falls short, fill from the remaining candidates in sort order until reaching q. After selecting the positive side, we update Üze using the selected positives and then select negatives (and vice versa). This procedure repeated until per-bin quotas are satisfied.
F.3.3 COMPOSITE SCORE SELECTION
As simpler alternative, we form a weighted composite score:
S(x) = a (+R(=)) + 3 G(x),
where R(x) the robustly scaled repetition score, (xx) is the scaled deviation term (1) and (a, (3) balance the trade-off. This produces single scalar ranking per bin.
- Unerl.
F.3.4 BASELINE: RANDOM SAMPLING
We also consider random baselines:
Simple random: global uniform sampling, ignoring bins;
F.3.5 SUMMARY
In summary, ranking enforces matched distributions in length and bio-utility, while maximizing contrast in repetition. The Pareto strategy provides a principled multi-objective approximation, the composite score offers a tunable scalarization, and random sampling serves as a control. We report dataset-level statistics and violin plots of the three repetition components and two bio-utility metrics Section
5
F.4
PSEUDO ALGORITHM OF DATASET CONSTRUCTION
For completeness, provide pscudocode for the dataset construction procedure described in Sec- This algorithm implements the optimization objective
tion 4.2
(D+ DT) arg max AR s.t. AU S.
with multiple selection strategies (Pareto frontier, composite scoring, and random selection base- line).
G IMPLEMENTATION DETAILS AND HYPERPARAMETERS
This section provides additional details on model setups, baseline implementations, and hyperpa- rameter choices.
22

---

Published as a conference paper at ICLR 2026
Algorithm Dataset Construction for Contrastive Steering
1: Inputs:
Raw positive pool p+ (low repetition, unconstrained utility)
Raw negative pool p (high repetition, unconstrained utility)
Repetition scorer R(-), Utility scorer U-)
Tolerance e, target sizes n4, n-
2:
Outputs: D (filtered positives), D (filtered negatives)
3: Compute (R(x), (x)) for all sequences in P+ U UP-
4: Filter out sequences with (x) U1 > c
5: Partition candidates into bins B by sequence length and bio-utility quantiles
6: if method = Pareto then
7: for each bin B with quota k do
8: Scale R(z), (xx) within
9: Select positives maximizing R(x)) while aligning (xx) to offt) (Pareto frontier)
10: Update ('(=)
Select negatives minimizing R(x) while aligning Uxx) o(-)) (Pareto frontier)
12: end for
else if method = Composite then
14:
for each bin B with quota do
Scale R(x), U (2x) within
Positives: score S(x) +aR(x) - (z) 0,(1), take top
16:
Update et(-)
17:
Negatives: score S(z) ax)() - o(ا)), take top A
19: end for
else
Random baseline
21:
Uniformly sample n. positives and n negatives from global pools
22: end if
23: Return: D+, D
Table hyperparameter settings for ESM-3 sequence generation.
parameter
default value
schedule (schedule)
cosine random
unmasking strategy (strategy)
decoding steps (num_steps)
20
temperature (temperature)
1.0
temperature anncaling (cemperature.anneallng
True
top-p (tep-p)
1.0
G.1 MODEL CONHIGURATIONS
We summarize the default configurations for ESM-3 (masked LM) and PROTGPT2 (autoregressive LM). Unless otherwise specified, all other hyperparameters follow the official implementations.
G.1.1
ESM-3
Sequence generation.
The default iterative unmasking settings are given in Table 5
Structure folding, on usage 6T
For structure prediction tasks, we vary the number iterative steps depending
23

---

Published as a conference paper at ICLR 2026
Table 6: settings for structure folding with ESM-3.
usage
value of num_steps
dataset construction
main experiments
Table 7: hyperparameter settings for PROTGPT2 sequence generation.
parameter
default value
temperature (temperature)
1.0
top-p (top-p)
1.0
repetition penalty (sspetition.penaltt)
no-repeat n-gram size (no.repeat..graamsszz))
G.1.2 PROTGPT2
Sequence generation. We use HuggingFace's generate () API with the parameters listed in Table
To strictly control sequence length, we set
max manghength min aaiinlende length target sequence length.
G.2
DECODING STRATEGY BASELINES
G.2.1
ESM DECODING STRATEGY
Unless otherwise noted, parameters follow Appendix G.I.I with only the sampling policy varied.
Generative unmasking (default). Random positions are unmasked with annealed temperature: schedule cosine,
strategy random, temperature annealing True.
Entropy-basse based unmasking. Lowest-entropy positions are unmasked first. To isolate this effect, annealing is disabled:
schedule cos cosine, ine, strategy entropy,
temperaturee.mmelllng False.
Temperature sampling.
We sweep the sampling temperature:
TE {0.7, 1.0, 1.3).
Top-p sampling.
We vary the nucleus probability cutoff: E {0.80, 0.85, 0.90, 0.95, 0.98, 1.00}.
G.2.2 EFFECT OF DECODING STEPS ON REPETITION
To examine whether the observed repetition behavior of ESM-3 arises from an insufficient number of decoding iterations, conducted ablation varying the number of unmasking steps. Note that ESM-3 performs parallel unmasking rather than stepwise autoregressive decoding; the default configuration of 20 steps follows the official recommendation balancing quality and efficiency, We nonetheless tested wide range of decoding depths to verify the robustness of this choice.
Specifically, we varied the number of iterative unmasking steps
num_steps E {10, 20, 40, 100, 200),
under unconditional generation (target length = 200 aa, 100 samples, random seeds). Table reports the mean and standard deviation of structure and diversity metrics.
24

---

Published as a conference paper at ICLR 2026
Table Ablation on the number of decoding steps (num (cumssteps) for ESM-3 sequence generation. eepetition-relaterr metrics noomm Distinct-2/3, Rupoly) remain stable, indicating that repetition is intrinsic to the model rather than caused by limited decoding depth.
Steps
PLDDT
PTM
Haorm
Distinct-2
Distinct-3
Ripoly
10
10.95
0.607=0.117
0.454
0.282=0.086
0.371+0.160
0.559:1-0.377
20
80,41=13.89
0.581=0.187
0,442#0.164
0.217=0.10
0.373=0.182
0.557=10.270
40
12.42:+1..39
0.598=0.177 179
0.429=0.153
0.198=0.089
0.344+0.16 167
0.555:10.279
100
83.35
0.628.10.204
0,460.1.113
0.222-0.122
0.381
0.560110.300
200
10.30.114.99
0.608.10.194
0.454.1..199
0.221
0.381-00.22
0.563110.302
Table 9: Composite scores versus decoding steps.
Steps
Repetition Score
Biological Utility
10
0.435
0.707
20
0.431
0.692
40
0.419
0.711
100
0.440
0.732
200
0,439
0.706
Summary. Across decoding depths, repetition metrics remain nearly constant, while structure quality (pLDDT, pTTM) varies only within normal statistical noise. The composite repetition-utiliy scores (Table show no consistent improvement as the number of steps increases, confirming that repetition primarily determined by decoding depth.
Overall, increasing the number of unmasking iterations does not meaningfully reduce repetition, suggesting that the observed degeneracy is inherent property of the pretrained model rather than a consequence of limited decoding steps.
G.2.3 PROTGPT2 DECODING STRATEGY
PROTGPT2 is an autoregressive protein LM. The official example command includes non-trivial repetition penalty
sequences protgpt2
"<I entoftext >>", max_length=100, do_sample-True
top_k=950, wwwn_reeturnsseueneeeeeeeee
jos_token_id=0)
Our goal is to study repetition without decoding-specific heuristics. repetition.penalty--l in all baselines and compare standard decoding methods:
Thus, we set
Top-p (nucleus) sampling.
We truncate to the top cumulative probability mass p. sweeping p E (0.80, 0.85, 0.90, 0.95, 0.98, 1.00].
Temperature sampling. We vary the softmax temperature:
E {0.7, 1.0, 1.3).
Repetition penalty (for comparison). For completeness, also include ablations with trivial repetition penalties:
repetit iopetition.penaltyy E (1.1, 1.2, 1.3).
These settings are reported only for comparison and are not part of our main baselines.
n-gram blocking. We additionally evaluate n-gram blocking, where repeated n-grams disal- lowed during generation:
E (2, 3, 4).
25

---

Published as a conference paper at ICLR 2026
G.3 MECHANISM INTERPRETABILITY BASELINES
We provide implementation details for two mechanism-level baselines: Neuron-based Deactivation and Probe-based Steering. Both operate on sequence-level hidden activations h h²(x) R of transformer language model. where L denotes the layer index and d the hidden dimension. For masked language models (MLMs), h (x) is obtained via mean pooling across tokens, while for autoregressive models (AR-LMs) we use the last-token embedding (see Appendix B
G.3.1 NEURON-BASED DEACTIVATION
Definition of neuron. At layer L, the activation for sequence is 1c E Rd. A neuron corresponds to a single coordinate j E (1....,d, d), iec the scalar value (z)
Supervision signal. We construct contrastive sets D (low repetition) and D (high repetition). For each sequence æ, we compute the normalized entropy Hnorm (z) (Section 4.1 Appendix as
scalar supervision signal: lower entropy indicates higher repetition.
Scoring. For every layer L and neuron j. we collect activations {h (z)} across sequences, align them with supervision values (Horm (z)) and compute the Pearson correlation coefficient:
(1-ا) = cor(h)((1)) (x)) Hnorm (n))
We then rank all neurons globally by |s(1.5)).
Deactivation.
We select the top-K K most correlated neurons and set their activations to zero: hx() + O, j E Sk.
Configuration. We sweep K across {8, 64, 256 1024, 4096J.
Algorithm Neuron-based Deactivation
Require: Activations (x:), supervision (n)om()))
1: for each layer L and neuron do
2: Collect (hi (r) over dataset
3: g(Iná) + corr(h)(x)) (x), Hnorm (I))
4: end for
5: Rank neurons by |s(L,)) select top-K indices SK
6: for each (L,j) E SK do
7: Set h (x) + 0 Vr
8: end for
Summary. This baseline evaluates how strongly neurons correlate with entropy based repetition signals Hnorm (x), and ablates the most predictive dimensions by zeroing them out.
G.3.2 PROBE-BASED STEERING
Setup. Given contrastive sets Dt (low repetition) and D (high repetition), we train a binary probe at chosen layer L to separate the two sets. The probe's learned weight vector is then used as a steering direction.
Model. Let the activations at layer L (1" C Rd. We form a labeled dataset:
X
[x (x+) (x ]]
y [1p++] 0 Op--1
A logistic regression probe is trained with binary cross-entropy:
b. f BCFWithToggtssz.v))
26

---

Published as a conference paper at ICLR 2026
Steering. After training, extract the weight vector R Optionally normalize it as v w/|[w|le At inference, activations are updated as:
ú if normalize=True,
h' = h + w.
w. otherwise.
where E R controls the steering strength (Appendix B
Configuration. We control the probe-based steering through three hyperparameters:
Layer L: the layer at which activations are probed and edited.
Normalization: whether to L2-normalize the weight vector w (default: False).
Steering strength a: the magnitude of the applied edit (default: 1.0).
Algorithm Probe-based Steering
Require:
Activations (h (x:),, }. contrastive sets D* D
1: Build dataset (X, v) from D and D
2: Train logistic regressor (w,, b) (X,y)
epochs=200, 1r=10-2, weight,decay=0
3: w f w/ ||/2 if normalize=True else w
4: inference: h +
Summary. This baseline learns a linear probe distinguishing high- vs. low-repetition activations, and uses its weight vector as steering direction reduce R(x) while aiming preserve (xx).
G.4 HOW SELECT THE BEST PARAMETERS
In this section, we describe how we select the optimal hyperparameters for each method. Recall from Section 4 that we evaluate generated sequences using the repetition score R(x) and the utility score (xx (see Appendix B). For each method, we choose the hyperparameters that maximize harmonic mean of R(x) and U(x):
Score(x:) 2- R(x) U(x) R(x) + (())
This criterion balances between reducing repetition and maintaining biological plausibility, ensuring that the selected parameters yield sequences that both diverse and structurally sound.
H DETAILS OF EXPERIMENT RESULTS
H.1 MAIN RESULTS
We provide complete results across all three datasets (CATH, UniRef50, SCOP), both generation
tasks (conditional, unconditional), and both model backbones (ESM3, ProtGPT2). Figures 8
present R-U seatter plots, and Tables 12 13 report the full set of metrics underlying R(x) and Uxx). Specifically, each table provides values for pLDDT, pTM, Hnorm, Distinct-2, Distinct-3, and Rapolly
H.2 ABLATION STUDY RESULTS
ablation experiments probe the two main components of the method (i) contrastive dataset construction and (ii) steering vector injection- -along three axes: dataset sam- pling, steering strength a, and steering layer L..
As introduced in Sec. 5.3
27

---

Published as a conference paper at ICLR 2026
Table 10: ESM3 results for two tasks with utility constraint: (a) unconditional generation, (b) con- ditional generation.
CATH
UniRef50
SCOP
Method
R
U
R
Unconditional generation
Original Model
20.044
0.576
0..24400004
0.576
4..23
0.576 00.00
Temperature Sampling
0.751 20,006
0.502
0.700
0.566
0. #0.00
0. 0.566
Top-p
Top-p-Sampling Sampling
0.419 00,00 +0,000
0.590
+0.37* 0..90
40.00
0.590
+0,077 00.007
0.419 #0.m0
0.590 +0,00
Entropy Based Sampling
40,022
0.477 20.44
0.904
0.477=0mm
0.904+00002
-044
Neuron Deactivation
0.551 10,001
0.507 1.507 1.0.488
0.448 10:00
0.594
10,022
0.452
0.597 0.597 10,005
Probe Steering
0.587 +0.011
0.416 20.00
0.586
20,010
0.415
0.585
UCCS()
0.645
0.65210.020
0.631 10.016
0.64610.00 10.011
0.641
0.646 20.012
(b) Conditional generation
Original Model
0.54210.00 10.004
0.686 10.018
10.014
0.686 10,018
0.54210.00 10,004
0.686 10,018
Temperature Sampling
0.792 10,006
0.65210.021
10:00
0.652_o.021
0.79210.000
0.65210.01
Top-p Sampling
0.539 10,016
0,681 1..01
0.539 10.016
0.681 1..81
0.539 100006
0.681
Entropy Based Sampling
0.923.g0.003
0.378g0.0
0.923 20.00
0.578_0011
0.923.20.00
0.578gn01
Neuron Deactivation
0.637 10.004
0.63610.00
0.590 100
0.638
0. 0.537 20,013
0.673
Probe Steering
0.5:06g0.005
0.688g00044
0.546
0.685_om22
0.5 0.5:00
0.686_om23
UCCS(ours)
0.707 20,019
0.720
0.707 20.002
0.726
0. 0.707 40.00 40.001
0.717 20.11
Notes. R: repetition score; U: biological utility. annotate cells with v when U satisfied).
Unni (utility constraint
CATH - Cond.
UNIPROT Cand.
SCOP Cond.
0.725
0.700
0.700
0.675
0.675
0.650
0.050
0.625
0.635
0.000
0.600
0.600
R(O)
ROO
R(X)
(6)
[4)
(6)
CATH Gnoond.
UNIPROT Uncond,
SCOP Unooodd
0.85
0.60
0000
0.55
0.55
0.55
0.50
0.50
0.00
0.6
0.7
0.4
0.5
0.9
0.7
0.5
0.9
1030
R(x)
(b)
(d)
Entropy Dased Sampling
Original Model
Temperature Sampling
UCCS
Neuron Deactivation
Probe Steering
Topep Sampling
Figure 8: Main results of ESM3. R-U scatter plots across three datasets (CATH, UniRef50, SCOP) and two generation conditions (conditional and unconditional), summarized into single 2x3 panel.
Setup. Unless otherwise noted, we use UniRef50 (Consortium) 2024). For each ablation we con- struct contrastive sets D and 'D- (100 sequences each) from natural proteins and PLM generations with high repetition. Results are averaged over five random seeds and evaluated under both uncon- ditional and conditional generation. We report repetition score R (higher is better) and bio-utility score U (higher is better). Shaded areas and error bars indicate variability across seeds.
Below present detailed visualizations and tabular results for each axis of ablation.
28

---

Published as a conference paper at ICLR 2026
CATH - Cond.
UNIPROT Cond.
SCOP - Cond.
0.73
0.73
0.72
0000
0.71
9.21
0.70
0.70
0.20
0.89
0.86
0.85
R00
R00
R60
(8)
(c)
(c)
CATH Unoodd
UNIPROT Unooedd.
Unnond
0.72
0.70
0.68
0.70
0.68
0.68
0.99
0.66
0.64
0.64
0.64
0.70
0.62
0.725
0.750
0.775
0.800
0.825
0.850
0.72
0.74
0.76
0.78
0.80
0.82
0.72
0.74
0.76
0.70
0.80
0.82
0.84
(b)
(4)
83
Neuron Deactivatio No Repeat N-grum
Ociginal Model Probo Stawring
Repetition Ponalty
Top-p Sampling
Temperature Sampling
CUCS
Figure 9: Main results of ProtGPT2. R-U scatter plots across three datasets (CATH, UniRef50, SCOP) and two generation conditions (conditional and unconditional), summarized into a single 2x3 panel.
Table 11: ESM3 Results: Unconditional Generation
Metrics
Datuset
Method
Parameter
PLDDT
pTM
Haurm
Distinct-2
Distinct-3
Rhpuly
CATH
Original Model
0.686
0.466
0.445
0.200
0.347
0.549
CATH
Temperature Sampling
Temperature 1.3
0.687
0.444
0.763
0.416
0.739
CATH
Top-p Sampling
Top p= 0.95
0.699
0.482
0.440
0.193
0.339
0.550
CATH
Entropy Based Sumpling
0.596
0.358
0.923
0.626
0.955
0,998
CATH
Neuron Deactivation
Top K 1024.0
0.643
0.372
0.545
0.216
0.429
0.785
CATH
Probe Stecring
47
0.692
0.482
0.442
0.196
0.342
0.535
CATH
UCCS(ours)
Layer =
0.524
0.632
0.296
0.549
Uniref50
Original Model
0.686
0.466
0.445
0.200
0.347
0.549
Uniref50
Temperature Sampling
Temperature 1.3
0.687
0.444
0.763
0.416
0.739
0.913
Uniref50
Top-p Sampling
Top 0.95
0.95
0.699
0.482
0.440
0.193
0.339
Uniret50
Entroyy Based Sampling
0.596
0.358
0.923
0.626
0.955
0.998
Uniref50
Neuron Deactivation
Top K 256.0
0.700
0.487
0.474
0.204
0.353
0,592
Uniref50
Probe Steering
Layer=
0.691
0.480
0.442
0.196
0.343
0.536
Unire150
UCCS(ours)
Layer 47
0.778
0.513
0.619
0.286
0.531
0.866
SCOP
Original Model
0.686
0.466
0.445
0.200
0.347
0.549
SCOP
Temperature Sampling
Temperature = 1.3
0.687
0.444
0.763
0.416
0.739
0.913
SCOP
Top-p Sampling
Top p= 0.95
0.699
0.482
0.440
0.193
0.339
0.550
SCOP
Entropy Based Sampling
0.358
0.626
0.955
SCOP
Neuron Deactivation
Top 256.0
0.707
0.487
0.479
0.202
0.351
0.602
SCOP
Probe Stecring
Layer 47
0.691
0.479
0.441
0.196
0.342
0.535
SCOP
UCCS()ur))
Layer=
0.773
0.519
0.628
0.294
0.546
0.876
29

---

Published as a conference paper at ICLR 2026
Table 12: ESM3 Results: Conditional Generation
Metrics
Dataset
Method
Parameter
pLDDT
PTM
Hmorm
Distinct-2
Distinct-3
Rhpuly
CATH
Original Model
0.808
0.565
0.545
0.339
0.515
0.655
CATH
Temperature Sampling
Temperature 1.3
0.785
0.519
0.768
0.549
0.824
0.921
CATH
Top-p Sampling
Top p=
0 0.95
0.807
0.554
0.538
0.330
0,505
0.660
CATH
Entropy Based Sampling
0.720
0.435
0,909
0.755
0.974
0,996
CATH
Neuron Deactivation
Top K 1024.0
0.782
0.491
0.601
0.349
0,570
0.850
CATH
Probe Stecring
47 Layer=47
0,813
0.558
0,548
0.339
0.517
0.663
CATH
UCCS(ours)
0.851
0.589
0.669
0.438
0.691
0.886
Uniref50
Original Model
0.808
0.565
0.545
0.339
0,515
0.655
Uniref50
Temperature Sampling
Temperature = 1.3
0.785
0.519
0.768
0.549
0.824
0,921
Uniref50
Top-p Sampling
Top p= 0.95
0,807
0.554
0.538
0.330
0.505
0.660
Uniref50
Entropy Based Sampling
0.720
0.435
0.909
0.755
0.974
0.996
Uniref50
Neuron Deactivation
Top K 1024.0
0.785
0.491
0.573
0.339
0.539
0.758
Uniref50
Probe Steering
Layer=47 47 Layer
0.813
0.557
0.548
0.339
0.517
0.663
Uniref50
UCCS(ou
0.857
0.595
0.672
0.439
0.691
0.884
SCOP
Original Model
0.808
0.565
0.545
0.339
0.515
0.655
SCOP
Temperature Sampling
Temperature = 1.3
0.785
0.519
0.768
0.549
0.824
0,921
SCOP
Top-p Sampling
Top p 0.95
0.807
0.554
0.538
0.330
0.505
0.660
SCOP
Entropy Based Sumpling
0.720
0.435
0.909
0.755
0.974
0.996
SCOP
Neuron Deactivation
Top 64.0
0.799
0.547
0.559
0.335
0.506
0.692
SCOP
Probe Steering
=ayer=47
0.814
0.558
0.547
0.339
0.516
0.663
SCOP
UCCS(ours)
Layer= = 47
0.849
0.585
0.671
0.440
0.695
0.884
Table 13: ProtGPT2 Results: Unconditional Generation
Metrics
Dataset
Method
Parameter
PLDDT
pIM
H Hgorm
Distinet-2
Distinct-3
Papply
CATH
Original Model
0.747
0.495
0.721
0.406
0.647
0.936
CATH
Temperature Sampling
Temperature
0.746
0.478
0.745
0.430
0.707
0.954
CATH
Top-p Sampling
Top p = 0.98
0.736
0,479
0.706
0.391
0.624
0.929
CATH
No Repeat N- gram
no repeat 3.0 gram
0.741
0.485
0.727
0.404
0.663
0,947
CATH
Repetition Penalty
repetition penalty 1.2
0.753
0.492
0.777
0.458
0.759
0.955
CATH
Neuron Deactivation
Top K= 8.0
0.734
0.487
0.708
0,393
0.627
0.940
CATH
Probe Steering
Layer = 30
0,739
0.475
0.711
0,408
0.649
0.926
CATH
UCCS(ours)
Layer 30
0.820
0.602
0.855
0.553
0.826
0.991
Uniref50
Original Model
0.747
0.495
0.721
0.406
0.647
0.936
Uniret50
Temperature Sampling
Temperature 1.3
0.746
0.478
0.745
0.430
0.707
0.954
Uniref50
Top-p Sampling
Top p = 0.98
0.736
0.479
0.706
0,391
0.624
0.929
Unire150
No Repeat N- N-gram
no repeat 3.0 gram
0.741
0.485
0.727
0.404
0.663
0.947
Uniref50
Repetition Penalty
repetition penalty 1.2
0.753
0.492
0.777
0.458
0.759
0.955
Uniref50
Neuron Deactivation
Top 8.0
0.746
0.481
0.705
0.389
0.632
0.940
Uniref50
Probe Steering
Layer=
0.749
0.498
0.723
0.411
0.649
0.945
Uniret50
UCCS(ours)
Layer
0.807
0.599
0.828
0.510
0.788
0.994
SCOP
Original Model
0.747
0.495
0.721
0.406
0.647
0.936
SCOP
Temperature Sampling
Temperature 1.3
0.746
0.478
0.745
0.430
0.707
0.954
SCOP
TTo-- Sampling
Top p 0.98
0.736
0.479
0.706
0.391
0.624
0.929
SCOP
No Repeat N-gram
repeat 3.0 gram
0.741
0.485
0.727
0.404
0.663
0.947
SCOP
Repetition Penalty
repetition penalty 1.2
0.753
0.492
0.777
0.458
SCOP
Neuron Deactivation
Top K
0.742
0.487
0.727
0.412
0.664
0.938
SCOP
Probe Steering
Layer =
0.746
0.492
0.728
0.415
0.661
0.939
SCOP
UCCS(ours)
Layer = 30
0.823
0.621
0.844
0.530
0.805
0.993
H.2.1
DATASET SAMPLING ABLATION
This section expands the summary in Sec. 5.3 by detailing the effect of different sampling strate- gies (Simple random, Pareto, and Composite) for constructing contrastive datasets. We report both unconditional and conditional results for ESM3 and Pro1GPT2.
30

---

Published as a conference paper at ICLR 2026
Table 14: ProtGPT2 Results: Conditional Generation
Metrics
Dataset
Method
Parameter
PLDDT
pTM
Distinet-2
Distinet-3
Repoly
CATH
Original Model
0.836
0.572
0.800
0.620
0.850
0.973
CATH
Temperature Sampling
Temperature 1.3
0.829
0.571
0.811
0.634
0.872
0,978
CATH
Top-p Sampling
Top p 0.98
0.837
0,583
0.794
0.612
0.833
0,966
CATH
No Repear N-gram
repeat 2.0 gram
0.839
0.577
0.793
0.612
0.840
0.965
CATH
Repetition Penalty
repetition penalty
0.591
0.807
0.632
0.871
0,971
CATH
Neuron Deactivation
Top 64.0
0.581
0.626
0.847
0.971
CATH
Prohe Steering
Layer =
0.834
0.569
0.795
0.608
0.833
0.970
CATH
UCCS(ours)
Layer = 30
0.851
0.635
0.860
0.672
0.891
0.991
Uniref50
Original Model
0.836
0.572
0,800
0.620
0.850
0.973
Uniref50
Temperature Sampling
Temperature Top p = 0.98
0.829
0.571
0.811
0.634
0.872
0.978
Uniref50
Top-p Sampling
0.583
Uniref50
No Repeat N- gram
no repeat 2.0 gram
0.839
0.577
0.793
0.612
0.840
0.965
Uniref50
Repetition Penalty
repetition penalty 1.1
0.841
0.591
0.807
0.632
0.871
0.971
Uniref50
Neurnn Deactivation
Top K 8.0
0.834
0.577
0.793
0.603
0.837
0.973
Uniref50
Probe Steering
Layer 30
0.828
0.552
0.793
0.614
0.837
0.968
Uniref50
UCCS(aurs)
Layer
0.834
0.616
0.859
0.675
0.895
0.995
SCOP
Original Model
0.836
0.572
0.800
0.620
0.850
0.973
SCOP
Temperature Sampling
Temperature 1.3
0.829
0,571
0.811
0.634
0.872
0.978
SCOP
Top-p Sampling
Top p 0.98
0.837
0.583
0.794
0.612
0.833
0.966
SCOP
No Repeat N-gram
repeat 2.0 gram
0.839
0.577
0.793
0.612
0.840
0.965
SCOP
Repetition Penalty
repetition penalty 1.1
0.841
0.591
0.807
0.632
0.871
0.971
SCOP
Neuron Deactivation
Topp 64.0
0.831
0.571
0.794
0.616
0.839
0.962
SCOP
Probe Steering
Layer=
0.821
0.551
0.789
0.828
0.966
SCOP
UCCS(ours)
Layer
0.846
0.627
Table 15: dataset sampling ablation: per-task performance across seed runs (mean + std), model:
ESM3, dataset: Uniref50.
dataset
R
U
HM(R, U)
unconditional
pareto
0.636 10.013
0.674
0.655
composite
0.62340012
0.637
0.639
simple (random)
0.618
0.61810000
0.634_0.021
0.626
5
conditional
pareto
0..00s
0.715
0.709
composite
0.693 16022
1..012
0.705
simple (random)
+0.07
0.706
0.699
notes. R denotes the repetition score R(x), and U. denotes the bio-utility (xx).
Visualization. We visualize the repetition score (R(x)) score (U))) trade-off under three dataset sampling strategies (Simple random, Pareto, and Composite) for both tasks (unconditional vs. conditional) and both models (ESM3 vs. ProtGPT2). Each marker corresponds to seed, and shaded ellipses indicate the 95% covariance region across seeds for each method.
Results. Table 15 and Table 16 report the per-task mean for repetition and utility metrics, together with the harmonic mean HM (R, U). These tables complement the visualizations by pro- viding robust averages and variability across seeds.
Findings. Overall, our Pareto and Composite strategies outperform simple random sampling. This especially evident in the unconditional task, where the error ellipses tighter (indicating greater
31

---

Published as a conference paper at ICLR 2026
unconditional: R(x) U(x)
conditional: R(x) U(x)
0.70
0.76
0.68
0.74
0.66
A
composil
0.64
simple
Ubd
@mp
0.70
0.62
composite
0.68
composite
pareto
simple
0.66
simple
0.58
R(x)
R(x)
(a) ESM3, Unconditional
(b) ESM3, Conditional
unconditional: R(x) vs. U(x)
conditional: R(x) vs. U(x)
0,74
0.78
composite
0.73
0.76
simple
0.72
Do with
0.71
simple
UD0
0.74
simple
pareii
0.70
0.69
0.68
composite pareto simple
0.70
0.67
0.68
0.60
0.81
0.82
0.83
0.84
0.95
0.96
0.86
0.87
0.88
0.89
0,90
R(x)
R(x)
(c) Pro1GPT2, Unconditional
(d) ProtGPT2, Conditional
Figure 10: Dataset sampling ablation for ESM3 and ProtGPT2 under unconditional and conditional generation. Each point a seed, with ellipses showing 95% seed covariance.
Table 16: dataset sampling ablation: per-task performance across seed runs (mean = std), model: PROTGPT2, dataset: Uniref50.
dataset
R
U
HM(R, U)
unconditional
pareto
0.835 10.007
0.722 10.007
0.775
composite
0.844 10.009
0.714410.007
0.774
simple (random)
0.825
0.704_00.06
0.760
conditional
composite
0.800
10.014
0.808
pareto
0.889 10.007
00.013
0.805
simple (random)
0.878 10.000
0. 0.726
0.795
notes. R denotes the repetition score R(x), and U denotes the bio-utility Uxx).
stability across seeds) and both and U values are consistently higher. In conditional generation, differences smaller but Pareto/Composite still maintain clear edge.
H.2.2 STEERING STRENGTH ABLATION
This section provides the full results for the steering strength(a in section4.3) ablation summarized in Sec. [53] We investigate how injecting the steering vector at different strengths affects R and U in ESM3 and ProtGPT2.
32

---

Published as a conference paper at ICLR 2026
Visualization. We provide three complementary plots per model: (i) log2 vs. repetition R(x), (ii) log2 a vs. utility Ux), and (iii) the ((z)-U(x) trade-off scatter. Shaded bands indicate 95% confidence intervals across seeds, and markers highlight the selected best operating points.
logg (a) vs. R
017-00717
log2 (a) vs. U(X)
1.5
0.7
106666
0.725
0.0
0.700
0.5
0.675
R(x
Ulx]
0.650
0.4
0.625
0.3
0.600
0.2
conditional
0.575
conditional
unconditional
0.550
unconditional
0.1
-2
logo (a))
logo (a)
(a) ESM3: R. vs. logg a
(b) ESM3: U vs. log, a
Rix) vs. UOx
0.725
0.700
0.675
0.625
0.575
unconditional
0.550
0.2
0.3
0.4
0.5
0.6
0.7
Rix
(c) ESM3: R vs.
Figure 11: Steering-strengee (a) ablation on ESM3. Conditional sampling consistently traces the upper envelope in the R-U plane.
Results. Tables 17 and 18 summarize the per-task means (# std) across seeds for each tested o, reporting atomic metrics (e.g., Hnorm, Distinet-n, Rapoly, pLDDT, P'TM) together with the aggregates R and U.
Observations. Across both backbones, logg(a) exhibits unimodal trend: increasing from neg- ative values to roughly 0.5 (a Re 1.4 ~ 2) improves both and U; further increases cause marked drop in the unconditional setting, while the conditional setting is comparatively robust (Figs.
ESM3. As shown in Table 17 unconditional performance peaks around 1.5 (log, 0.585), where R = 0.636 and U 0.677. Conditional sampling attains its joint optimum near a = (R=0.717, U = 0. 708). Very large a (e.g., 20) yields imbalanced trade-offs: some structure metrics increase, but R collapses.
ProtGPT2. When selecting the operating point based on the harmonic mean of R and U(eection G.4). the optimal setting is a 1.0. In the unconditional case, this yields 0.83.33 and U 00776, while for conditional sampling we obtain Rt 0.891 and U 0.742. Although larger values of a (e.g.. = 1.5) further increase It this comes at the cost of a noticeable drop in U. Overall, conditional curves dominate the Pareto frontier, and a 1.0 offers the most balanced trade-off between repetition reduction and bio-utility (Figs.
12 b)
Recommendation. Both models exhibit stable trade-offs when a lies in the range 1-2. In practice, we recommend selecting a = 1.0, which maximizes the harmonic mean of R and U (Appendix G.4) and thus provides the most balanced operating point. For conditional sampling, values in [1, 2|
33

---

Published as a conference paper at ICLR 2026
logz (at) R(x)
log2 (a) vs. U(x)
R0x 00591
0.9
050-007022
RUXT
0.75
a=1 Nk)=0.706
0.8
0.70
0.65
R60
U(x)
0.60
0.55
conditional
conditional
0.50
unconditional
unconditional
logg (a)
log2
(a) PROTGPT2: R vs. log2
(b) PROTGPT2: U vs. log2
R(x) vs. UIX)
Red 201 Uni
0.75
unconditoonl
conditional
a 700
0.70
0.53
0.50
0.6
0.7
0.8
0.9
R(x)
(c) PROTGPT2: R vs. U
Figure 12: Steering-strength (a) ablation on PROTGPT2. Shaded areas denote seed variability. Markers highlight the selected operating points discussed below.
Table 17: Alpha ablation: repetition and bio-utility metrics per a (mean + std across seeds). Model: ESM3, Dataset: Uniref50
(a)
Haorm
D
Dist-3
Ragoly
Pl.DITT
pTM
R
Unconditional
0.05
-432
0.402
0.207 20.014
0.366
0.586go.002
0.71220012
0.403
0.448 20,00
0.603
0.25
-2
0.540
0.23910.009
0. 0.689 0.401
0. 0.737_ 10.000
0.203
0.521 10011
0.615
0.5
-1
0. 10.001
0.265
0.487 0.487100.00
0. 0.775
0, 1..66
0.508
0.580
0.637 10.00
00000
0.281 20,004
0.528 2.000
0.847 20.15
0. 0.781 20.12
0.525
0.653 0.622
1.5
0.585
0.616
20,012
00.14
0..83
0.538
+0.22
0.657 2.636
2
0.588 20.044
0.20940.011 20.011
0.303
0. 0.759 2.017
0.877
0.515
20014
0.637
20.077
5
2.32
0.34440.00
0.158
0.267200022
0.267
1.456
0. 1.707 20.01
0.39440010
0.337
0.337
0.551
20.000
20
4.32
0. 0..04
0.073
0.115
1..03
0.711 10.025
0.40410017
0.14440021
0.557 10.019
Conditional
0.05
.4.2
0.566
0.349 10.004
0.535_00009
0.702_0.00
0.81410.11
0.54810022
0.570
0.681 10.0s
0.25
-2
0.605
0.376 10,005
0.58410.011
0.768 10.017
0.8301o.0
0.557
10017
1..10
0.69410.013
0.5
10.014
0.403
0.40310,000
10,009
0.627 10.15
0.811 10.011
0.849 1001
0.567
0.56710022
10,25
0.65410011
0.70810.01
0
0.637
10,000
0.42610.000
0.66710007
0.865
10:05
0.84210012
0.56610017
0.69010019
0.704
1.5
0.585
0.667 10.015
0.439
0.43910.007
10.007
0.684100088
0.901 10.019
0.8391o013
0.563
0.710
0.701 10o0s
2
1
0.666
0.437 20.010
0.6862001
0.924_2001
0.841gn012
0.5744n011
0,717
0.708
20.00
0.315
0.463
0.615go019
0.84920018
0.51440016
0.498
200r7
20
4.32
0.312
0,253
0.291
0.875
0.568
0,275
0.722
main competitive: a= favors higher utility U, whereas a 1.5-2 emphasizes repetition reduction R at the cost of some U. Very small or large values of a lead to inferior performance in both settings.
34

---

Published as a conference paper at ICLR 2026
Table 18: Alpha ablation: repetition and bio-utility metrics per a (mean = std across seeds). Model: ProtGPT2, Dataset: Uniref50
log2(a)
Haurm
Dist-2
Dist-3
Rapaly
pLDDT
PTM
R
Unconditional
0.05
4.32
0.719
0.407
0.407
20,03
2.64920.08
0.937 20:012
0.744
0.494
20.00
0.6192000s
0.25
-2
0.756100005
20.002
g0.06s
2.908
0.776
0.535
0.65620.015 0.705
0.5
0.8 0.814
0.506
0.981 20.00
0.814
0.598
0.81210013
0.706
1
0
0,838 0,838 20,011
0.534100.00
0.807 10.018
0. 0.992.20005
0,811 20.015
0.60110021
0.833 10001
0.706
1.5
0.585
0.841 10,000
0.5-18 10,007
0.815
0.997 10:001
0.803
10005
1.812
1000s
0.816
40.00
0.777
0.991
0. 0.705
0.4722u022
0.818
0.613
5
2.32
0. 00.30
0.311
002222
0. 0.453
0.656
0.310
0.
0. g0.09
20
4.32
0.697 40.051
00.07
0.587 40.002
0.997 #0.007
0, g.030
4.000
0.7 40.00
0.345
Condtionnl
0.05
4,32
0. 0.80220.001
0.61 0.611
0.841
0.974
0.8 0.833
0.577
0.834
0.706
0.25
-2
0,821 +0.010
0.639 0.639 20.008
0.866200009
0.979.20.00
0.587 0.587 0.833
+0018
0.851 0.851 20:005
0.710
0.5
0.8.93.00.09
0.670 40.000
0.89620000
0.990.00002
0.619 0.848440015
0.87220003
0.731
0.868 +0,011
40,014
0.920
2.900
0.803
0.63 49.021
0.891 0.891 +0.00
0.742
1.5
0.585
0.874
0.7 0.710
0.913
0. 2.900
0.827
0.580
0.891
0.701
2
1
0. 0.706 20.044
0.910
0. g0.05
0. 0.792
0.099
0.889
0.610
2.32
0. 0.798
0..95 40,055
0.409 20.150
0.493 20233
g.495
40.03
0.346
0.581 20.103
0.525 20.07
0.748 40,044
0.468 20.058
0.705
20000
0.690
0.456
0.778
0.573
H.2.3 STEERING LAYER ABLATION
We further study the effect of inserting the mean difference vector at different transformer layers. Results reported for both ESM3 (48-layer auto-encoding) and PROTGPT2 (36-layer autoregres- sive) backbones.
Visualization. We provide three complementary plots per model: (i) layer index L vs. repetition R(x). (ii) layer index vs. utility (z), and (iii) the (x) trade-off scatter. As before, shaded bands indicate 95% confidence intervals across seeds, and markers denote the best-performing lay- ers.
19 19 report, for the unconditional setting, the per-task averages (# standard deviation across seeds) of each steering layer. The tables include both the atomic metrics nHorrms Distinct-n, Repoly, pLDDT, p'TM) and the aggregate scores R and U. The corresponding results for the conditional setting are presented Tables 22 20
Results. Tables
Observations. For ESM3, steering at early or middle layers (L < 40) has little effect on either R or U. marked improvement appears only in the final layers, peaking at L 46 where = 0.652 and U = 0.660 in the unconditional aetting(Table, 21] and R = 0.723, U = 0.721 in the conditional (Tabee 20
This indicates that late-layer intervention is most effective for the auto-encoding backbone.
For PROTGPT2, repetition control is already strong overall. Steering in very early layers (e.g., L 2) or around mid layers (L 18) tends to exacerbate repetition and slightly reduce U. In contrast, steering after 20 gradually improves both metrics, with a clear optimum around
L == 30 (R 0.888, U 0.730 in the conditional setting, see Table
Overall, both models demonstrate that effective steering occurs in the later portion of the network, but the precise "sweet spot" depends on architecture depth: the last few layers for ESM3, and the last third of the stack for PROTGPT2. Conditional sampling consistently traces the upper R-U envelope across layers.
35

---

Published as a conference paper at ICLR 2026
layer vs. R(X)
L=40
layer vs. U(X)
Ladd
conditional
0.725
unconditional
RDd1-D.652
0.65
0.700
Lodd
0.675
0.80
0.650
conditional
0.55
unconditional
0.625
0.600
0.45
0.575
0.40
0.550
20
30
layer
layer
(a) ESM3: across layers
(b) ESM3: across layers
0.725
cumditinn unconditiona
LedG
0.675
0.650
0.625
0.525
0.550
0.45
0.50
0.55
0.50
0.65
0.70
RG1
(c) ESM3: R vs.
U
Figure 13: Layer-wise steering ablation on ESM3. Conditional sampling tends trace the upper envelope in the R-U plane.
layer vs. R(x)
L=30
layer vs. U(x)
0.90
RDX1=0.888
0.75
6M-88:28
0.85
0.70
0.80
0.75
0.65
Ub0
0.60
0.55
0.60
0.35
conditional
0.50
conditional
unconditional
unconditional
0.50
10
15
20
25
30
35
0
5
10
15
20
25
30
35
layer
layer
(a) PROTGPT2: across layers
(b) PROTGPT2: across layers
R(x) U(x)
L-30
1-:30
conditionnl
0.70
incuncitional
0.05
0.60
0.55
0.50
0.5
0.7
0.8
0.9
Rix)
(c) PROTGPT2: R vs. U
Figure 14: Layer-wise steering ablation on PROTGPT2. Shaded areas denote seed variability.
36

---

Published as a conference paper at ICLR 2026
Table 19: Layer ablation (unconditional) result of ProtGPT2 on Uniref50: repetition and bio-utility metrics per layer (mean + std across seeds).
layer
Hnorm
Dist-2
Dist-3
Hepoly
PLDDT
pTM
R
0. 0.728 40020
0.410
0.656
0.942
0.743
0.488
0.734
0.616g0022
0.670
0.34
g.065
000200
0.721
0.472-00.22
0.596 0.082
2
0. 10015
0.167 0..155
0.286
0.857 20.022
0.624
0.350
0.487=00019 0.527
0. 0.730
0.41620.039
0.658
0.920
0.735
00000
0.739 200003
0.612_00013
4
0. 20012
0.42020.014
0.669 20014
0.961
0.747
0.497 0.499220.02
0. 0.746g.0.011
0.622200199
0.751
0.445
0..
0.965_g0.05
0.754
20.027
0. 0.763
0.637
6
00011
0.445_20019
0.698
0.958.0.009
0.765
0.520
0. 0.762
0.642200021
0.066
20.23
20023
0.97220.005
0.757 2001
0.517 20.000
0.772 0.702
0.637
8
0.764
0.45920.026
0.7 0.715
0.964_0:000
0.001 g.r
0.772
0.650_0.024
0.750
0.436
0.68310001
0.965_1.008
0.749 10023
0.498 0.498 10.037
0.758 0.75810.023
1.0023
0.623
0. 10
1001
0.443
0.692 10.02
0.967 10.007
0.754
0.508
0.765
0.631
11 1.71
0.418
0.656100119
0.972
0.758 10016
0.516.2.019
10.00s
0.607
12
0,008
0.434
0.682 10015
0.978
0.764
0.50220.031
0. 0.765
0.633
13 0.769
0.442200221
0.686_00031
0.979
0.501 0.023
0.771 200014
0.632_0.021
14
0.75
00010
0.410=0.012
0.639
0.988
0.755
0.497
0. 00.07
0.626_0.010
15 0.019
0.401
0.614
20.004
0.758
0.501
0.752200000
0.629_60016
16
0.738
0.311
0.45610017
0.995
0.025
00.00
0.603 0.705
17
10.02
0.453
0.082
0.991 0.091
10.003
0.755
0..01
0.704
0..03
18
10015
0.275
0.388
100029
0.995
0.729
0.526
0.688
10.01
0.627
19
0.78610012
0..19
0.655
1.01710017
0.989 10.003
0.74510011
10.018
0.771 100009
0.622 0..02
20
0.
10009
0.530
0
10,02
0.988 10.003
0.754
0.494
0.82610.009
0.624_0.014
21
1o0u
0.559
0.843
10.002
0.055
0.010
0.845
0. 0.,03
22 10008
0.51620.133
0.804
0.987 10.005
0.785
0.518
0.818
0.652_0.000
23
0.515 40003
0.805 20012
0.987=0.005 20.005
0.987
0.785
20.04
0.81740.000
0.655_0..008
24
0.81840020
0.818
20020
0.533
0.819 40.02
0.986
0.770 400000
0.513
0.827 0.827 +0.055
0.642_0.022
25
0.542
0.827 10.007
00.003
20012
0.544
0.804
0.666
26
0.821
10010
0.533
0.814 1..00
0.989
0.791 10000
0.537 10.016
0.537
0.828
0.82840005
0.664_0.012
27 1.024
0.545
0.821 100027
0.991 10.004
0.991
0.799 0.79910016 20016
0.577 10.016
10.05
0.688
28
0
0.821
10008
0.515
0.794
0.991 10.004
0. 1.8051000
0.563
0.822 10.27
0.684_0.022
29
0. 0.837 10007
0.545
0.81910.020
0.991 10.002
0.8 0.810 10000
0.584
10.00
0.697
30
4.012
0.54620.013
0.815
0.991 20.002
0.828 40017
0.613
0.839
0.83940007 40007
0.720
31
0.829
4000s
0.521 20.013
20012
0.991 20.004
0.8 40000
40.022
0.826 20004
0.698_2.026
32
0.530 40010
0.793 20018
0.988
0.817 40011
0.828 0.583
0.700
33
0.524 00013
0.788
20.003
g0019
40.026
20022
0.688_
34
0.
0.823
0.518
0.77 0.774
0.984_p.003
0.794
40012
0.556
0.818 40001
0.675
35
0.
40011
0.388
0.627 20017
0.941 0.941 10.011
0.742 40.22
0.480
0.719 20,100
0.611 20.024
T MECHANISTIC ANALYSIS OF REPETITION IN PROTEIN LANGUAGE MODELS
To address why certain protein PLMs suffer more severely from repetitive degeneration, and to pro- vide concrete hypotheses for the community, we conduct systematic mechanistic analysis at two complementary levels: (i) layer-level probing, to trace how repetition-related information emerges and evolves across depth; and (ii) neuron-level correlation analysis, to examine whether repetition is localized or distributed, and how its strength differs across architectures. Together, these analyses reveal which models struggle most with repetition and offer insights into the underlying representa- tional geometry.
I.1 LAYER-LEVEL EVIDENCE OF HODEL-DEPENDENT REPETITION DYNAMICS
We begin by examining how repetition-relater information is represented throughout the network using layer-wise linear probes. We construct balanced dataset of positive (low-repetition) and
37

---

Published as a conference paper at ICLR 2026
Table 20: Layer ablation (conditional) result of ProtGPT2 on Uniref50: repetition and bio-utility metrics per layer (mean + std across seeds).
layer
Hoorm
Dist-2
Dist-3
Hopely
PLDDT
pTM
R
0.805
0.610
0.836
0.972
0.577 0.838
0.708 0.833
0.774
0.5
0.791 4000:
0.954
40000
0.546-
0.676 0.002
2
10021
0.425
0.635
0.918
0.467 0.769
0. 1.0016
0.61810019
0.806
0.613
0.046
0.827 0.900
0.835 0.572 20009
100011
0. 0.602
4
20000
0.62420.014
0.857 20014
0.978
0.835 0.835
0.585
0. 0.8422.00009
0. 0.710
0. 0.012
0.623g0.115
0.85220.009
0.974
0.000
0.576
100007
0. 0.703_00005
6
00007
0.623g0.318
0.85220014
0.972:0.007
0.576g0.021 0.830
0. 0.703 0.841
20013
0.642g0.020
0.86-
0.977
0.831 +000
0.851 0..01
0.711 0..00
8
0.002
0.624g0.014
0.803
0.974
0.827
0.581
0.841
0. 0.704_00020
10.009
0.632
0.816100011
1.007
10007
0.576_10017
100008
0.699
O. 10
10008
0.640
0.867 10001
0.98110.003
0.827
0.679
0. 0.703 0.852
11 0.807
10012
0.604
0.828 10006
0.975
0.023
0.572
0.697_0.018 0.833100000
12
10011
0.618
0.838
0.98220.004
0.8181001
0.562200031
0.8 0.842
0.690_0021
13 0.807
0.603g0.115
0.818 200009
0.985.20003
0.014
0.545_20.013
0.834420008
0.678_
0.818 14
0.604
0.8 0.81610021
0.902
00005
0.540
0.8 0.000000.000
0.677=0018
15
0.003
0.509
0.764
0.993
0.007
0.538
0.824
0.607_00016
16
0.775
0.432
0.56510.029
0.991 00.004
0.758 10000
1..09
0.75510001
0.628
17
0.839
10003
0.612
0.799
10.024
10.00
10000
20.006
0.846 0.006
0.603
18
0.7891000
0.411
0.529
10.001
0.991
0. 0.755 100009
0.528 10.025
0.75010000
0.641
19
10.08
0.086
0.79010.229
10,099
0.99310.004
0.791 10019
0.522
0.807100012 100012
0.657 0.657
20
0.869
0. 0..00
0.925
10010
0.994
0. 1..00
0.514
0. 0.003
0..02
21 0.87710006
0.932 20.15
10.007
0.900
0.787 10:00
10.09
10.01
0.661_0.009
22
10011
0.91210.001 0..88
0.99210.005
0.82910005
0..05
0.88210005
0.607
23
4.004
0.689
0.914 g0.014
0.000
00.00
0.829
400mt 0.82940011
0..09
0.699 0.883
24
0.869
10008
0.715
0.937 10.006
0.991 10.006
0 0.818 400044
0..09
0.693_a.015 0.896
25
0.060
0. 0.709
0.927 10.00
0.991
10.000 10.006
0.829
20018 4.029400188
0..05
1.88940.00
0.697
26
0.8584000
10012
0.691
0.91 10,022
0.990
0.832
0.584
0.883
0.708
27
0.070
20.008
0.927 10.006
0.993
10013
40.025
0.894 10,00
00:01
28
0.858
0.681
0.907 100016
0.992
0.840
1..05
0.881 10,012
0.717
29
0.861 10015
10.021
10.14
0.99210.004
0.02810000
10000
0.587
10.000
0.708 20:11
30
1.871+000
0.69220.013
0.904
20014
0.994_n.002
0.847 +0018
0.613
0.888 20007
0. 0.730
31
0.859
40006
0.685
0.902
+0014
0.993_n..02
20013
0..09
20.0s
0.727
32
0.862 40004
0.691 20.012
0.899
+0.008
0.993
0.842
40,00
0.601 20.018
0.883 40004
0.721
33
0.857
0.681 20.016
0.009
0.836+0000 0.992_..003
40009
20.031
0.881
40006
0.714
34
0.857
40010
0.684
0.893
0.990
0.843 40.005
0.597
0.879
0.720
35
0.789+0012
0.590
0.815
0.963
0.821 40.00
0.562
0.692
negative (high-repetition) sequences, extract per-layer sequence representations, and train a logistic regression classifier at each layer.
Across both ProtGPT2 and ESM3 (Fig.
15 Fig 16
we observe a consistent three-stage pattern:
Early layers. Probe Accuracy and AUC rise sharply, approaching = 1.0 within only few lay- ers. This indicates that cues predictive of downstream repetition- such as short-range motifs, local copy patterns, homopolymeric tendencies-aa are encoded extremely early and become globally accessible with little depth.
Middle layers. A shallow U-shaped dip appears: AUC remains high, but Accuracy decreases. This suggests that the repetition signal persists but becomes temporarily entangled with other semantic and structural features, reducing its linear margin while retaining rank-based separability.
Final layers. Probe performance recovers near the output, consistent with repetition being sharp- ened into logit-aligned attractor state that directly influences next-token prediction.
38

---

Published as a conference paper at ICLR 2026
Table 21: Layer ablation (unconditional) result of ESM3 on Uniref50: repetition and bio-utility metrics per layer (mean + std across seeds).
layer
Hnorm
Dist-2
Dist-3
Hepoly
PLDDT
pTM
R
0. 0.459
0.202
0.024
0.561 40.029
0.477 4.003
0.433
0.586_0001
0. 0.457
0.204
0.325
0,182 0.700 0.006
0.601 0.000
2
0. 10016
0.2042
0.304
0.561.00022
0.475 0.695
0. 0.433
0.585
0. 20410
0.349 0.200
0.349
20.018
0.557 200023
00003
0,478 20.009
0,429 2:0014
0.586_20010
4
0. 20013
0.205
0.357 200022
0.564_00022
0. 0.696
0. 0.478
0. 0.406
0.587_00014
0. 20060
0.203
0.35-420027
0.000
0..01 40009
0. 0.030
0.59120008
6
0. 20019
0. 0.203g0.013
0.355_2002
0.567 20.035
0.697
0. 0.482200021
0. 0.435
0.580
20018
0.204_0.013
0.358 g0027
20.077
0.488 0,488
20.020
0.595_00019 0.438
8
0. 40010
0.207
0.360
0.576g0.018
0.480 0.704
0.442
0.593_00017
0.
1..00
0.204
0.356100017
0.576_1.024
0.685 10.000
0.476 10.022
0.440
0.581
10
0.461
0.
10009
0.204
0.35610015
0.57210.026
0.687
0.476
0.438
0.082_00022
11 0.
10064
0.206
0.000_00020
0.582
0.404 0.000
0.444100015
0.594_0.014
12
0.
0.404
0.206g0.012
0.060
0.572
0.699
0.484
0. 0.440
0.501
13
20011
0.206_0.000
0.065
0.576_0.030
0000
0.489
0.440
0.597
14
0.
0.207 20014
0.361 10001
0.58220.039
0.702
0.488
0. 0.444
0.505_00020
15 40009
0.206
0.360_20016
0.575
0.005
0.490
0.440 20.14
0.597
16
0.461
0.206
0.359 10.05
0.564
0..00
0.484
0.436
0.593
17 10.01
0.206_o.013
0.359 10.05
0.561 0.501
10,030
1.000 10023
0.483 20.025
10.02
0.592
18
0.199 1.002
1.044
0.546
0.092100012
0.475
0.423
0.
19
10012
0.005
0.35210002
0.352
10.026
0.504
0.69510001
0.477
10.022
0.404
0.586
20 0,198 1.024
0.343
1.00s
0.561
0.69210002
0.471
0.429
0.42910.029
10.029
0.581
21
0.449
0..092 10021
0.342
100020
0.537
0.537
10.032
0.681
0,460 20.099
0.419
0.670 10002
22 10010
0. 0.208
0.362 10016
0.56810.025
0.696
0.486
0.440
0.001
23
0.200 40014
0.346
20017
0.558_0.030 00000
0.683
40.021
0.428 20,019
0.575
24
0.457
20025
0.201 10,016
0.346 10030
0.580
0.691 40.009
0.476
0.437
0.437
+000:
0.584
25
0.449
0.194 20010
10016
0.567
20.028
0.670
20.013
0.426
0.557 20.10
26
0.449
0.197 0.107
0.338
10.018
0.679 0.570_0.036
0.455
0.429
4..20
0.567
0..01
27 0.448
10.09
0.335 0..05
20012
1..04
0.673
0.456
10.011
0.504
28
0.
0.199 10015
0.344
0.590
0.68 1 10012
0.460
0.43910.015
0.439
10.015
0.000
29
10012
0.202
0.351
100022
0.57210.029
0.677
0.461 10.023
0.434 10.022
0.569
30
0.
40013
0. 0.208
0.361
+0.000
0.590
0.677 20044
0.463 0.463
0.570 4.446
31
40007
0.193
0.335 20013
0.525_20.024
0.675
0.457 20.010
20012
0.566_0.00
32
0.44740022
0. 0.199
0.348
40.300
0.574
0.676 40.29
0.472
0.431
4000:
0.574_0.024
33
0.205 000
0.362
20018
0..05
20013
0..07 20.011
20017
0.591
34
0.203 0 40015
0.358
40001
0.599
0.691 40014
0.482
0.448
40.19
0.586
35
40022
0.200
0.351
10.024
0.562
0.677
20008
0.42940000 20.024
0.429
20,00
0.576
36
0.479
0.
10011
0.214
0.3794a000
0.379
1.0000
0.602
0. 0.700
0. 4..00
0 0.459 20007
0.505
37
0.215g0.000 0.47440013
0.380
2002
0.588_20023
0.693
20.01
0.453g0006
0.502220024
38
0.4
0.479
0.214_0.112
0.376
20023
0.593gn.052
0.694
0.487 20,020
0.456
0.591
39
0.205 40013
Lan19
0.594_n.052
40016
20.022
0.449
0.581
40
0.478
0.21 0.211 40011
0.382
20019
0.683 0.622_.0.017
0.470
0.466
0.576_p0018
0.548
4000s
0.252_0.010
0.468
0.4684a014 20014
0.749 0.678_n.018
4001s
0.555
0.529
20011
0.652_2.018
42
0.598
40018
0.273
0.497
+0027
0.724
0.747 40011
0.502
0.569
+00es
0.624_0.019
43
0.575
00.002
0.477 #0014
20.099
0.707
0.461 0.707 +0008
0.570
0.584_0.008
44
0.514
0.230_2..007
0.41611011
0.651 20.02
0. 106000
0. 0.446
0. 20.0s
0.571
45
0.537
20007
0.225
0.422
0.785
0.699
0,434
0.549
0.567
46
0.
0.64440016
0.300_m..14
0.5621a000
0.882_n.01
0.77440044
0,546
0.652.o004
0.660_p.016
47
0.6
0.615
0.290g00014
0.543
0.866_p.011
0.795
0.554
0.636go013
0.674
Importantly, this pattern is stronger in ESM3: its mid-layer dip is more pronounced and its late-layer sharpening more extreme. This aligns with our main experiments showing that ESM3 exhibits more severe repetition than ProtGPT2.
39

---

Published as a conference paper at ICLR 2026
Table 22: Layer ablation (conditional) result ESM3 on Uniref50: repetition and bio-utility metrics per layer (mean + std across seeds).
layer
Hnorm
Dist-2
Dist-3
Hopoly
PLDDT
pTM
R
40009
0.343
0.020
0.684_0..27
0.555 4.013
0.55840013
0.685_0:008
0.554
20.022
0.020
0.683
0.812
40.022
0.536
0.681
2
0.009
0.340
0.620,20006
0.677
0.814
0.554
0.534110014
0.684_00014
0.557 1.000
1.0000
0.343 20.003
2.0037
0.682
0.809 20000
0..50
10001
0.680_6.016
4
0.340
0.519 20010
0.676g0.022
0.807 20012
0.543320022
0 0.533
0.675_0..17
0.555
0.342200005
0.523.20.00
0.012 0.680
0.549
0.536
0.68120012
6
20012
0.337
0.517 20010
0.675.0:024
0.811 2000
0.548
0.531
0.67920014
0.558
0.345g0.09
0.52920004
0.529
20014
0.687
g.01
0.561 0.556
0.685_00.14
8
0.008
0.342g0.000
0.522 20017
0.68620.036
0.812
0.049
0.68120002
1.001
00.00
0.025100016
0.525
10.016
0.680_0.029
0.012
0.547 10.024
0.557 100077
0.680
10
10000
0.308
0.532
10012
0.68910.018
0.015
0.55410.017
0.561
0.684_0.011
11
1.000
0.347 20.005
10.000
0.693
0.810 10005
0..52
0.565
0.681_0.012
12
00007
0.342
0.524
0.68610.023
0.811
0.552.00020
0.539
0.682_00013
13
0.559
0.345
20.099
0.69220.023
00015
0.557 20.018
0,563
0.686_e.15
14
0,00
0.345
0.52920010
0.601
0.818
0.560
0.563
0.680
15
0.558
20008
0.34220000
0.526
20.22
20003
0.552220.22
0.060
0.681200018
16
0.559
10014
0.345
10077
0.687 20.031
0.819
0.500
0.001
0.610
17
0.553
10010
0.340_0.008
10.14
0.683_0.026
10017
10.02
0.056
0.687
18
10014
0.341
0.52010017
0.67810.025
0.815
0.504
0.003
0.080
19 1.003
0.310
0.52010001
0.520
1001s
0.67310.030
1.014
0..50
10.019
0.682 0.682_0.016
20
1001
0.337
0.513
0.676
0.81010011
0.543
0.03010.013
0.676
21
0.541
0.000
0.508
0.50810012
10012
0.662.0.019
10013
0.543
20.000
100011
0.676
22
0.34 10014
0.522 10014
0.681 10.020
0.812
0.547
0.555
0.679
23
0.552
40002
0.338
0.51 00.08
0.687
40000
0.539 20.032
0.555
40,00
0.669_0.026
24
0.554
+0009
0..09
0.5 1.512 10,009
0.802 0.690
0.543
0.557
0.55740006 40006
0.672_0.016
25
0.542
10010
0.328
0.497 10.009
0.673
0.797 40000
20010
0.542 40.018
0..63
26
0.543
0.498 0.49810013 0.330
0.680
0.802
0.532
0.546
0.667
27
10.00
0.330
0.508
0.50810.00
10.005
0.689 10.024
1000s
0.539 20.019
0.554
0.672
28
0.559
10012
0.345
0.523 10013
0.700
1.005
0..00
0.56510001
0.672
29
0.561
10014
0.352
0.53410011
1..0910.015
0.804
0.537 10.019
10.012
0.670
30
0.561
40006
0.351 20.007
0.535
40007
0. 0.704
0.810
0.549
0.570
20006
0..00
31
0.004
0.337 20.008
0.517 20011
0. 0..09
0.797 4000r
0..06
20,02
0.
0.666_0.008
32
0.04
0.341 20.009
0.520
#0.009
0 0.680 2.001
0.806
0.552 0.552
20,100
0.679
33
40014
0.347 20.012
0.531
+0013
0.704 000022
0.806
0..03
0.568
0.680
34
0.561 40000
0.530 0.346g0.007
40,005
0. 0.709
4.002
0.547
0.569
20.0s
0.678
35
00017
0.348g0.012
0.530 20016
0.681 10.030
0.681
4..21
20.013
0.557 +0,00
0.663
36
0.009
0.358
0.550
+0004
0. 0.716
0.806
0.556
1.580
0.681
37
0.568go010
0.5
0.552 00.008
20
0.6 0.690_a.019
0.801 goots
0.548
20.084
20.024
0.675 0.072
38
0.577
+0009
0.362_0.006
0.558 10010
0. 0.709
0. 0.807 20044
0.548
0.582 20.08
0.677
39
4000
0.357 20.09
0.552 zo011
0.707 20.21 20.21
0.799
0.535 20.022
0.578
0.667
40
0.568
40010
0.358_0.006
0.555
+000
0. 0.702_m.017
0. 0.798
0.535
0.576 20.00
0.667
41
0.580
20012
0.369
0.574 20015
0.709
0.807 0..29_0.026 n.026
20012
+0.020
0.676 0.587
42
0
0.633
40004
0.397 20.00s
0.614
0.770=m.m1
0.823 40003
0.543
0.636+0000
0.683
43
0.012
0.393
0.618
0.796
0000
0.528
0.6422a000
0.665
44
10009
0.569 0.56910000 0.362.20.009
10000
0.717 20.014
0.808 20000
0.526g0.011
0,587 20.009
0.667
45
00017
0.357220.04
0.559200033
0.783_20031
20019
20.012
0.608
0.660-20015
46
0.
1.084
0.455
0.706
0.903_n.000
0.851
0. 0.723 0.501
0.721
47
0.
0.666
0.434200003
0.67620011
0.887200011
0.850go012
0.580_a0023
0.715 0.703
1.2
NEURON-LEVER EVIDENCE FOR
REPETITION SUBSPACES
To investigate whether repetition arises from localized malfunctioning units or distributed represen- tational modes, we compute Pearson correlations between each neuron's 's activation and a continuous repetition score across all layers (Fig. 17 Fig. 18
40

---

Published as a conference paper at ICLR 2026
1.0
0.8
Probe performance
0.2
Accuracy
AUC
0.0
10
20
30
40
Layer index
Figure 15: Layer-wise probe performance on ESM3. Accuracy and AUC rise rapidly in early layers, exhibit mid-layer dip, and sharpen again near the output. The strength of this pattern reflects ESM3's stronger repetition tendency.
1.0
Prabe performnnce
0.4
0.2
Accuracy
AUC
5
10
15
20
25
30
35
Layer index
Figure 16: Layer-wise probe performance ProtGPT2. The same qualitative pattern appears, but with overall weaker magnitude than in ESM3, matching its milder repetition behavior.
Across both models, the correlation distributions are centered near zero and lack isolated high- correlation outliers. This indicates that repetition driven by a small set of specialized neurons but reflects distributed representational pattern spread across many dimensions. Repetition there- fore aligns more naturally with a direction or subspace activation space than with single-unit activations.
However, the strength of the repetition- related subspace varies strongly across architectures. Prot- GPT2 exhibits narrow, unimodal correlation distributions, whereas ESM3 shows noticeably broader and, in several layers, bimodal patterns. This indicates that ESM3's representations form a more polarized repetition-related subspace-strongly separating neurons that increase versus decrease tivation during repetitive degeneration. This architectural difference mirrors our empirical finding that ESM3 struggles significantly with repetition.
41

---

Published as a conference paper at ICLR 2026
1eer12
Liperts
Neuron COUnT
Leerta
Correlation with repetition
Neuron Neuron-repetition correlation distributions for ProtGPT2 across all layers. The distribu- tions are narrow and unimodal, indicating a weaker and more diffuse repetition-relate subspace.
Figure 17:
1.3 INSIGHTS AND RECOMMENDATIONS FOR THE PLM COMMUNITY
Our analyses reveal that repetition in protein PLMs should be viewed globul and distributed representational phenomenon rather than a localized failure. Repetition emerges early, becomes most entangled mid-network, and is ultimately sharpened into logit-aligned attractor state near the output. Architectural choices modulate the strength this subspace: ESM3 exhibits a broader and more polarized repetition-related subspace than ProtGPT2, explaining its more severe repetitive degeneration.
These results suggest that future PLM development may benefit from diagnostics that track the geometry of repetition-related subspaces across layers, and from interventions that modulate repre- sentation trajectories rather than individual neurons or decoding heuristics. Understanding repetition at the level of activation space geometry provides a more principled foundation for mitigating de- generation in next-generation protein language models.
J FURTHER EVALUATION ADDITIONAL PROTEIN LANGUAGE MODELS
We additionally include experiments on two representative backbones beyond those reported in the main paper: ProGen2 and DPLM. These models span distinct architectures (autoregressive diffusion language models), training scales, and inductive biases, thereby broadening the diversity of PLM paradigms assessed in our study.
Consistent with the main experiments Sec. 5] we evaluate each model under both unconditional and conditional generation, using the repetition score R(x)), utility score U(x). and structural prediction protocol.
Overall findings. Across all addtional evaluated models, UCCS consistently:
42

---

Published as a conference paper at ICLR 2026
Lowelt
Lewil
Sew
Neuron count
Low Utmaat
Correlation with repetition
Figure 18: Neuron-repetition correlation distributions for ESM3 across all layers. The wider, and in several layers bimodal, distributions reveal polarized and substantially stronger repetition- related subspace, consistent with ESM3's ' more pronounced repetition tendencies.
1. reduces both motif-level and homopolymer-level repetition;
2. preserves or improves structural plausibility (pLDDT, pTM);
3. outperforms decoding heuristics and mechanism-leve baselines.
J.1 EVALUATION ON PROGEN2
We further validate UCCS on ProGen2-base Madani et al. 2023)
an autoregressive PLM available on HuggingFace Since ProGen2 follows the same generation paradigm as ProtGPT2, we adopt the identical baselines and decoding settings used for ProtGPT2, including temperature sampling, top-p sampling, no-repeat-n-gram blocking, and repetition penalties.
Across all three datasets, UCCS provides consistent improvements over ProGen2's default sam- pling behavior. As shown in Table 23 decoding heuristics heuristiss-including temperature scaling, top-p sampling, and no-repeat-ne n-ggea constraints produce nearly identical repetition scores in the conditional setting and offer limited control over pathological motifs; stronger constraints (c.g., no-repeat-nn-gamm) may increase R but at the cost of reduced utility. In contrast, UCCS yields mod-
43

---

Published as a conference paper at ICLR 2026
Table 23: ProGen2 results for two tasks with utility constraint: (a) unconditional generation, (b) conditional generation.
Method
CATH
UniRel30
SCOP
R1
U1
R1
U
R
U/ t
(a) Unconditional geueration
Original Model
v0.493 0.010
0.890
0.002
<0.195
0,890 + 0.002
/0.190 0.009
Temperature
0.890 0.002
v0.495
0.016
0.890 1: 0.002
0.492 0.014
0.890 1: 0.002
v0.496 1 0.011
Top-p Sampling
0.890 0.002
/0.191 0.011
0.890 + 0.002
0.191
0,890 + 0.002
/0, 0.011
No Repeat N-gram
0.989 0.001
0.189 : 0.011
0.914 = 0.001
v 0.013
0,959 + 0.001
0,488 0.011
Repetition Penalty
0.002
v0.495
0.016
0.002
0.014
1: 0.002
<0.491
0.017
Neuron Deactivation
0,803 0.013
0.490 0.025
0.911
0.008
0.486
0.030
0.876 4: 0.005
0.478 0.010
Prohe Steering
0.806 0.002
00.501
+ 0.006
0.894 + 0.003
v0.503
0.016
0.894 + 0.003
.0.503 0.019
UCCS
0.897 0.009
v0.511 0.014
0.912 + 0.004
v0.499 + 0.014
0.906 + 0.002
.0.500 0.007
(b) Conditional generution
Original Model
0.919 0.003
v0.601 0.015
0.919 0.005
<0.607 0.014
0.924 + 0.002
<0.804 + 0.014
Temperature
0.919 = 0.003
/0.606=
0.919 + 0.005
v0.007
0.011
0.924 + 0.002
.0.610 0.019
Top-p Sampling
0.919 = 003
/0.601 0.010
0.913 0.005
v0.625
+ 0.012
0.918
.0.613 0.014
No Repeat N-gram
0.932 = 0.001
v0.601 0.010
0.922 = 0.001
/0.615
0.935 + 0.002
/0.611 # 0.017
Repetition Penalty
0.944
0.582
0.003
0.010
0.944 0.003
0.589 0.027
Neuron Deactivation
0.912 D002
0.599 = 0.017
0.913 0.006
0.598 0.008
0.916 + 0.001
.0.605 + 0.012
Probe Steering
0.920 0. 0002
/0 = 0.020
0.921 + 0.001
70.618
0.925 + 0.001
/0.621 + 0.019
UCCS
0.924 0.003
v0.610 1: 0.005
0.921 1 0.005
v0.611 4 0.013
0.933 1 0.003
/0.610 1 0.015
Notes. repetition score; U: biological utility. Cells marked with v satisfy the utility constraint relative to the Original Model.
est yet reliable gains in repetition reduction while maintaining or improving biological utility under both unconditional and conditional generation. Notably, UCCS is the only method that simulta- neously increases R and satisfies the utility constraint across all evaluation conditions, confirming that its latent repetition direction generalizes robustly ProGen2 despite its distinet autoregressive architecture and training corpus.
J.2 EVALUATION ON DPLM
We next evaluate UCCS on the diffusion-based protein language model DPLM-650M (Wang et al., 2024) (airkingbd) wwwpiinnnnddddmmmmmmmmmmmm dp1m_650m), Because DPLM employs a fundamentally different genera- tion mechanism compared to autoregressive or masked PLMs, we adopt decoding configurations tailored its diffusion sampling pipeline. Specifically, we sweep four classes of baselines: (i) tem- perature scaling (0.7, 1.0, 1.3); (ii) sampling strategies {gumbel laarellaaaaaax ar gma, vanilla]; (ii) disabling the model's internal resampling module; and (iv) resample-ratin sweeps (0.10-0.25). All experiments follow the protocol in Sec. reporting repetition R(x:) and biological utility (xr)
across CATH, UniRef50, and SCOP.
Results in Table 24 show that DPLM achieves strong structural utility under default decoding, yet still exhibits notable repetition. Decoding heuristics such as temperature scaling and alternative sampling strategies introduce only minor changes to I and generally leave U unchanged. Disabling resampling substantially reduces repetition but induces pronounced degradation in utility, whereas adjusting the resampling ratio increases R while yielding unstable or inconsistent U. In contrast, UCCS achieves the best overall trade-off across both unconditional and conditional settings, de- livering substantial improvements in repetition reduction (e.g.. R 0.863 0.879 unconditional; R = 0.811-000888 conditional) while simultaneously improving structural plausibility relative to the original model. Notably, UCCS is the only method that consistently satisfies the utility constraint across all datasets and tasks, despite the distinctive diffusion diffusion-generaaionn dynamics of DPLM. These findings demonstrate that UCCS generalizes robustly beyond autoregressive and masked PLMs, ef- fectively steering diffusion- based models as well.
K
CASE STUDY: VISUALIZATIONS OF GENERATED PROTEINS
To complement the quantitative results in the main text, provide qualitative studies of gen- erated protein structures. Specifically, we visualize generations from both ESM3 and PROTGPT2, evaluated on the UniRef50 dataset under unconditional and conditional sampling. For interpretabil-
44

---

Published as a conference paper at ICLR 2026
Table 24: DPLM results for two tasks with utility constraint: (a) unconditional generation, (b) conditional generation.
Method
CATH
UniRe130
SCOP
R1
U1
R1
UT
T
(a) Unconditional geueration
Original Model
v0.811
0.016
v0.835
0.016
0.805
Temperature
0.806 0.007
0.837
0.022
0.805 1: 0.015
v0.840
0.019
0.805 1: 0.015
v0.840 1: 0.012
Gumbel Sampling
0.806 0.007
0.833 0.011
0.805 + 0.015
/0,838 0.018
0,805 + 0.015
00.810 0.012
Disable Resample
0.734 0.019
0.820 = 0.017
0.734 = 0.019
0.827 1.277:0000
0,734 +: 0.019
0.520 4 0.020
Different Resample Ratio
0.006
0.831
Neuron Deactivation
0.696 0.061
0.704 0.069
0.678 0.015
0.695 0.028
0.701 4: 0.052
0.713 4. 0.067
Prohe Steering
0.811 0.005
v..843 + 0.011
0.815 + 0.015
v0.847 0.008
0.815 + 0.006
<0.842 0.010
UCCS
0.863 0.013
000800 :: 0.008
0.878 + 0.004
v 0.010
0.879 + 0.006
20.875 0.008
(b) Conditional generution
Original Model
0.794 0.017
v0.782 +: 0.012
0.809 0.009
v0.796
0.011
0.806 + 0.007
.0.791 + 0.007
Temperature
0.797 0.014
v0.782 + 0.008
0.812 + 0.000
v0.797
+ 0.010
0.807 + 0.008
v0.792 + 0.009
Gumbel Sampling
0.789 0.016
0.776=00000
0.812 = 0.009
00,796
+
0.804 + 0.011
.0,793 = 0.007
Disable Resample
0.670 0.038
0.746 :: 0.021
0.702 = 0.018
0.776 0.003
0,701 + 0.012
0.781 : 0.016
Different Resample Ratio
v0.785
0.901
0.007
0.005
<0.791 0.015
Neuron Deactivation
0.745 0.044
0,736 0.054
0.731 + 0.006
0.715
0.761 + 0.025
0.746 = 0.033
Probe Steering
0.815 0.010
/0,796 :: 0.015
0.823 = 0.013
00,809 + 0.011
0,819 + 0.011
/0.812 + 0.011
UCCS
0.881 0.012
v0.821 1: 0.013
0.893 1 0.008
v0.845 0.008
0.894 1: 0.009
v0.840 1 0.008
Notes. repetition score; U: biological utility. Cells marked with v satisfy the utility constraint relative to the Original Model.
ity, we select representative sequences with lengths roughly in the ranges of 64, 128, and 256 amino acids (not strictly exact) to illustrate how different intervention methods affect the generation pro- cess. These visualizations highlight structural diversity and foldability improvements under UCCS, compared to generations and bascline decoding strategies. Color indicates pLDDT confi- dence: blue = high (> 90), cyan/green = medium (70-90), orange/red = low (< 70).
our UCCS method consistently produces structures that are not only less repetitive but also exhibit higher biological utility compared to the Original Model. In particular, generations steered by UCCS display stronger foldability signals, with a higher proportion of blue regions (high PLDDT), while maintaining structural diversity. This demonstrates that our method avoids the degeneration artifacts common raw generations and achieves a more favorable trade-off than baseline interventions.
As shown in Fig 25
28
L THE USE OF LARGE LANGUAGE MODELS (LLMs)
In preparing this manuscript, we made use of a large language model (LLM) as a general-purpose assistive tool. Specifically, the LLM was used for polishing the English presentation of the text (e.g., improving grammar, clarity, and flow), and for generating draft suggestions for sections such as the Discussion and Conclusion. All scientific ideas, methods, experiments, and analyses were conceived, designed, and conducted entirely by the authors. The LLM did not contribute to research ideation, experimental design, data analysis, or the generation of scientific claims.
The authors take full responsibility for the entire contents of the paper, including any text generated with LLM assistance. The use LLMs does not imply authorship. and the LLM is not listed as a co-author.
45

---

Published as a conference paper at ICLR 2026
Table 25: ESM3 Unconditional. Rows methods; columns approximate sequence lengths. 255
64 aa
128
VIV
UCCS
QULAA mim
Original Model
Probe Steering
Entropy-based Sampling
NeuronnDaattivation (TopK=256)
TemperatureSSamllng (T=1.3)
S JAM
TopPSampling (p=0.95)
46

---

Published as a conference paper at ICLR 2026
Table 26: ESM3 Conditional. Rows methods; columns approximate sequence lengths.
64 aa
256 128
CHODD LAAAN
UCCS
Original Model
Probe Steering
Entropy-based Sampling
NeuronDDeattivation (TopK=1024)
MAA
TemperatureSSapllng (T=1.3)
mummn
TopPSampling (p=0.95)
47

---

Published as a conference paper at ICLR 2026
Table 27: ProtGPT2 Unconditional. Rows methods; columns approximate sequence lengths.
64 aa
~ 128 aa
255 aa
UCCS
TIMA
Original Model
Probe Steering
NeuronDeactivation (TopK=8)
NoRepeatNgram
RepetitionPenalty (penalty=1.2)
TemperatureSSamllng (T=1.3)
TopPSampling (p=0.98)
48

---

Published as a conference paper at ICLR 2026
Table 28: ProtGPT2 Conditional. lengths.
Rows are methods; columns are approximate sequence
64 aa
~ 128 aa
~55 aa
UCCS
Original Model
Probe Steering
NeuronDeactivation (TopK=8)
NoRepeatNgram
RepetitionPenalty (penalty=1.1)
CANTEN ERAAAE
TemperatureSSmmling (T=1.3)
IAAAI
TopPSampling (p=0.98)
49


Reviews:
[{"review_id": "HGRegLvh9Q", "reviewer": "Reviewer_kZvX", "summary": "The paper presents the first systematic study of pathological repetition in protein language models (PLMs), proposing a novel method called Utility-Controlled Contrastive Steering (UCCS). The method effectively reduces repetitive amino acid sequences during generation, critical for biological viability, without sacrificing structural plausibility. The approach is elegant, practical, and clearly advances both the theoretical framing and engineering solution space for PLMs.", "questions": "a.\tDid you experiment with combining UCCS with decoding heuristics (e.g., top-p + UCCS)? Is there synergistic gain?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "a.\tThis paper is the first to clearly define and systematically address the problem of repetition in protein language models. This is done while understanding biological consequences. Rather than borrowing loosely from NLP, the authors develop a domain-aware framework that captures why repetition is especially damaging in proteins. This makes framework, truly domain specific.\nb.\tThe proposed method, UCCS, stands out for its simplicity. It doesn’t require retraining the model or altering its architecture, just a novel use of internal representations to steer generation away from pathological repetition. That makes it both practical and applicable to different PLMs.\nc.\tThe technical work is robust and well-grounded. The authors design meaningful metrics, like Rhpoly, to detect homopolymer collapse, and run thorough experiments across two leading PLMs (ESM-3 and ProtGPT2), three canonical datasets (CATH, SCOP, UniRef50), and multiple generation settings. Their method reduced repetition while preserving structural plausibility, as measured by AlphaFold confidence scores.", "weaknesses": "a.\tWhile UCCS performs well on ESM-3 and ProtGPT2, it’s still an open question how well the approach would transfer to larger or more diverse protein models like ProGen2 or ProteinMPNN? It would be valuable to see whether the same steering technique holds up as models scale or shift in architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T6ms7JKt77", "reviewer": "Reviewer_nKVd", "summary": "The paper systematically studies pathological repetition in protein langauge model. \nFirst, the authors formally define repetition in PLM and introduce quantitative metric that capture both motif-level and homopolym er repetition. These metrics are strongly correlate with structural utility measured by AlphaFold pLDDT.\nSecond, the authors introduced Utility-Controlled Contrastive Steering. They propose a representation-level intervention that disentangles repetition from structural utility. And inject such representation during inference stage to generate less repeated sequence.\nThe experiments show that UCCS reduces repetition in both MLM and autoregressive PLM.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper first systematically identify and formalize pathological repetition in PLM, with quantitative metric. The metrics are well-motivated and interpretable.\n2. The UCCS approach is simple, training-free and model agnostic. It greatly reduces repetition while maintaining foldability.\n3. The experiments cover both MLM and autoregressive models, both unconditional and conditional generation settings with multiple datasets.", "weaknesses": "1. Limited Novelty. I'm not familiar with steering methods during inference time with LLM, but according to the cited papers, it looks like deriving the difference vector and adding it during inference has been greatly explored in LLM. The adaptation to PLM is incremental rather than fundamentally new.\n2. Comparison with learning based methods. Similarly, I have no idea about if people reduce repetition with learing-based method in LLM. If so, is it possible to compare UCCS with these learning based methods?\n3. Lack of interpretation. While UCCS greatly improve the results, the paper would be stronger if it included analysis and visualization about the steering direction about biological meaning. For example, the model was about to generate AA...AA, and how steering method \"pull\" it back to a normal protein and why.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nkB1nmzA1A", "reviewer": "Reviewer_45fe", "summary": "This paper studies the phenomenon existing in the modern protein language models (PLMs) that the generated protein sequences usually exhibit  ill repetitions as a negative effect regarding the utility like foldability. The authors define the repetition score, utility score, and propose the UCCS, as a remedy strategy to help PLMs have improved repetition score while keeping utility. Experiments shows the UCCS can help ESM3 and ProtGPT2 reduce repetition and achieve comparable or higher plddt scores.", "questions": "- Though the paper has already included three small structure and sequence-only dataset, I think the “sensitivity” of steering datasets should be more systematically studied. Recent protein design models may involve structure data other than PDB (eg. AFDB or SwissProt) into training, Could the authors try quantitatively studying the dataset sensitivity of involved steering methods (such as temperature sampling and the proposed UCCS)? This may help the readers understand which strategies work consistently in reducing the repetitions.\n- (referring to one of the concerns above) Could the authors additionally consider benchmarking broader families of PLMs into the result table?\n- There are existing in-practice strategies to reduce sequence repetition (such as the remasking in MultiFlow[3] and gumbel+resampling in DPLM[2], to name a few). Could the authors also incorporate  these strategy as practical and competitive “heuristics and interventions” by showing similar benchmark and visualize the pareto front of R and U?\n\nMinor typos:\n\n- Ubiquitous UUCS -> UCCS?\n- Column 1 in Table 1 seems not balanced in the cell\n- The text in Figure 1 is too small to read\n\nReferences:\n\n[1] Nijkamp, Erik, et al. \"Progen2: exploring the boundaries of protein language models.\" Cell systems 14.11 (2023): 968-978.\n\n[2] Wang, Xinyou, et al. \"Diffusion language models are versatile protein learners.\" arXiv preprint arXiv:2402.18567 (2024).\n\n[3] Campbell, Andrew, et al. \"Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design.\" arXiv preprint arXiv:2402.04997 (2024).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The problem studied is relevant and interesting. The repetition problem in PLMs deserves more attention for the related research efforts\n- The proposed method, UCCS is model-agnostic and plug-and-play, making it easy to implement and adapt for existing PLMs\n- The experimental results support the claim that the proposed UCCS “mitigates degeneracy while preserving foldability”, yet I believe it can be further improved (refer to cons/questions).", "weaknesses": "- This study still does not explicitly answer (or explore further) why common PLMs (or some of the PLMs) intrinsically exhibit repetition in their generated samples, which I think is more important to guide the PLMs design for the community; furthermore, AlphaFold-like confidence score (aka foldability) only serve as a probe and can hardly counted as strong indicators for good design, broader metrics should be considered in the main results to let this paper become a comprehensive study. I do not think the current shown results are convincing enough to make UCCS really useful in the real design cases.\n- Given the authors claimed \"model-agnostic\" and “plug-and-play” steering method, benchmarking broader families of PLM should be considered, such as Progen[1] and DPLM[2] series. Improve the limitation in base model selections shall make the study more complete and resutls convincing and have larger impacts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8a3OZwwbw5", "reviewer": "Reviewer_g59x", "summary": "This paper introduces the problem of pathological repeats in sequences sampled from protein language models. The authors devised metrics (token entropy, n-gram diversity, homopolymer diversity) to quantify this phenomenon and leverage contrastive steering at inference time via a dataset of utility matched pairs to alleviate this issue. I appreciated the motivation of this work and the thorough investigation of failure modes in the design of the metrics. I also appreciate the simplicity of the contrastive steering solution and its broad applicability among protein language models. However, there are a few concerns with the current paper that make me lean towards rejection. I am open to increasing my score if these concerns are well addressed.\n\n1) While the metrics to capture pathological repetitiveness seem well justified, the claim that they *sharply separate natural proteins (POS) from PLM generations (NEG)* seems overstated as the distribution plot of $R_{hpoly}$ for ProtGPT2 samples is actually quite similar to natural datasets.\n2) I believe that the poor performance of ESM3 is related to the extremely low number of decoding steps (20) used in the experiment. Could the authors provide results with higher number of decoding steps (ideally the same number as the protein length)?\n3) Given the generality of the framework, could the authors expand the scope of PLM models they evaluate? ESM3 also models structure and function which could confound some of the results here. There are many popular and performant PLMs such as the ProGen family and ESM2. \n4) While the authors provide thorough results on the utility and repetitiveness of generated samples, they did not report the impact of each method on sequence diversity and novelty. I am concerned that the steering could restrict sample diversity and bias generations towards positive samples in the contrastive set.\n\nComments to improve the paper, not related to score\n- The plots in the paper are all quite small and difficult to read without zooming in\n- I would move the biological discussion supporting pathological repetition in proteins from the Appendix A to the main text", "questions": "See summary points 1-4", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- well motivated and tackles an under-researched problem \n- thorough and sound experiments\n- positive results for contrastive steering", "weaknesses": "- did not evaluate enough PLMs\n- sampling choices for ESM3 are too simple\n- did not report effect on sequence diversity and novelty\n- some results are overstated (see summary point 1)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)