Reverse Distillation: Consistently Scaling Protein Language Model Representations
Authors: Darius Catrina
Christian Bepler
Samuel Sledzieski
Rohit Singh
Venue: ICLR 2026 Poster (2026)
Keywords: Protein language models
model scaling
Representation learning
Subspace decomposition
interpretability
Model distillation
Matryoshka embeddings
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Protein language model performance plateaus at larger scales. Reverse Distillation decomposes them via smaller models, improving scalability.

Abstract: Unlike the predictable scaling laws in natural language processing and computer vision, protein language models (PLMs) scale poorly: for many tasks, models within the same family plateau or even decrease in performance, with mid-sized models often outperforming the largest in the family. We introduce Reverse Distillation a principled framework that decomposes large PLM representations into orthogonal subspaces guided by smaller models of the same family. The resulting embeddings have a nested, Matryoshka-style structure: the first $k$ dimensions of a larger model's embedding are exactly the representation from the smaller model. This ensures that larger reverse-distilled models consistently outperform smaller ones. A motivating intuition is that smaller models, constrained by capacity, preferentially encode broadly-shared protein features. Reverse distillation isolates these shared features and orthogonally extracts additional contributions from larger models, preventing interference between the two. On ProteinGym benchmarks, reverse-distilled ESM-2 variants outperform their respective baselines at the same embedding dimensionality, with the reverse-distilled 15 billion parameter model achieving the strongest performance. Our framework is generalizable to any model family where scaling challenges persist. Code and trained models are available at https://github.com/rohitsinghlab/plm_reverse_distillation.

Published as conference paper ICLR 2026
REVERSE DISTILLATION: CONSISTENTLY SCALING PRO- TEIN LANGUAGE MODEL REPRESENTATIONS
Darius Catrina'
Christian Bepler*
Department of Computer Science
Department of Computer Science
Duke University
Duke University
Durham, NC. USA
Durham, NC. USA
darius. catrinataaaaa.. na nnstrina@dukeer wew.eaaeeeee.mmmee edu
christian bep www.beppppeeee...o wwweereeeeeeeem edu
Samuel Sledzieski!
Rohit Singh
Center for Computational Biology
Depts of Biostats. and Bioinformatics; and Cell Biology
Flatiron Institute
Duke University
New York, NY, USA
Durham, NC, USA
ss ssledzieski www.latiooonnntttttttttttooootee
rohit. singtt.singhdduue..eee
ABSTRACT
Unlike the predictable scaling laws in natural language processing and computer vision, protein language models (PLMs) scale poorly: for many tasks, models within the same family plateau even decrease in performance, with mid-sized models often outperform- ing the largest in the family. We introduce Reverse Distillation, a principled framework that decomposes large PLM representations into orthogonal subspaces guided by smaller models the same family. The resulting embeddings have nested, Matryoshka-sty/ structure: the first & dimensions of larger model's 's embedding exactly the representa- tion from the smaller model. This ensures that larger reverse-distille models consistently outperform smaller ones. A motivating intuition is that smaller models, constrained by capacity, preferentially encode broadly-shared protein features. Reverse distillation iso- lates these shared features and orthogonally extracts additional contributions from larger models, preventing interference between the two. On ProteinGym benchmarks, reverse- distilled ESM-2 variants outperform their respective basclines at the same embedding di- mensionality, with the reverse-distille 15 billion parameter model achieving the strongest performance. Our framework is generalizable to any model family where scaling chal- lenges persist. Code and trained models are available at
https: http://ggtttubb..o/oooiiii
httpaaabbbpp__eeeeeeeeeee reverse rww.erse_dsttllllaeeeee
INTRODUCTION
Protein language models (PLMs) have emerged as powerful representation learners, capturing evolutionary patterns from millions of sequences and enabling unprecedented capabilities in structure prediction (Lin
et al.
2023), function annotation (Yu et 2023), and protein design (Ferruz et al. 2022 Devkota al."
2024
These models learn rich protein representations through self-supervised training on vast sequence
databases. However, unlike the predictable scaling laws observed in natural language processing (Kaplan) al. 2020 Hotfmann et al. (022), PLMs-and and more broadly, biological foundation models-exhibil coun-
terintuitive scaling behavior: smaller models often outperform larger ones on functional prediction tasks
Li
"Equal contribution.
Co-corresponding authors.

---

Published as a conference paper at ICLR 2026
al.
2024
For example, on deep mutational scanning (DMS) benchmarks from ProteinGym Notin et al., the ESM-2 family peaks at 650M-3B parameters, with the 15B model showing degraded performance.
2023
This unexpected scaling behavior creates fundamental challenges. Given models M,, M2 with |M2] > Mi] parameters, we observe non-monotonic performance: we often find that smaller models outperform larger models on downstream tasks. Moreover, we cannot reliably predict which biological tasks will ex- hibit poor scaling behavior, leading to difficulties in model selection for any specific task. A related lim- itation of PLMs is that embeddings across model scales are disconnected. In contrast, Matryoshka- embeddings (Kusupati et al. 2022) in natural language processing are structured such that their prefixes are also directly usable. With these embeddings, prefixes of an overall embedding are themselves functional, albeit with some performance degradation. This enhances computational and storage efficiency, enabling a paradigm of "embed once, reuse prefixes as needed.". However, current PLM representations do not offer this advantage: representations of dimension & cannot be truncated to dimension k' < l while maintaining smooth performance degradation.
A motivating intuition for our approach comes from the bias-variance tradeoff. Small PLMs, constrained by capacity, are biased toward encoding frequent, broadly-shared biological regularities- -eeaarttttisssssrrrrrrrra struc- ture propensities, hydrophobicity patterns, or conserved structural motifs. Larger models have the capac- ity to additionally represent rarer, higher-order phenomena: family-specifue patterns, epistatic interactions, allosteric signals. But this additional capacity also introduces variance: when higher-order features are entangled with simpler ones in a single representational space, downstream linear predictors struggle to iso- late task-relevant signal. The result is that task-irrelevant features effectively add noise the fundamental patterns that drive most benchmark performance.
We introduce Reverse Distillatior a principled framework that decomposes large PLM representations into orthogonal subspaces anchored by smaller models. Unlike traditional knowledge distillation, which compresses large models into small ones, our method identifies the unique information contributed by each model scale. Our key insight is structural: by treating smaller model representations as a basis and extracting orthogonal residuals from the larger model, we prevent destructive interference between features at different scales. In doing so, reverse distillation introduces a Matryoshka-see -tyle relationship between embeddings of different model sizes, enabling predictable scaling behavior as model size increases.
Formally, given models M, and M, where |Mrl < [Mpl). with embedding dimensionality k, < kip we decompose representations as Hp ~ [H,, Hres] where H, E B captures more fundamental patterns learned by the smaller model and Hres E (kp-k)) captures unique information from the larger model, orthogonal to H,, We prove that this decomposition MSE-optimal among all k--dim representations that fully encompass M, (i.e., the cylinder set of M, RP). Our contributions are:
Hierarchical Decomposition: We show how to transform family of PLM models such that they follow a hierarchical structure where each higher scale adds orthogonal information. Our decom- position is also guaranteed to approximate the original representation space well.
Matryoshka-stylee Embeddings and Monotonic Improvement: constructed such that nested embeddings of dimensionality d contain prefixes of sizes dj < d2 < dy < d such that each prefix is the reverse-distilled embedding of the corresponding dimensionality. Our decomposition thus provides controlled performance degradation as a function of embedding size.
embeddings are
Scaling Consistency:
Reverse distillation scales nearly always, i.e., larger eeverse-distille models consistently perform better than smaller ones.
Improvement over Baseline: For the ESM-2 family, reverse- distilled models of the same embed- ding size (e.g., 1280 for ESM-2 650M) generally outperform their corresponding baselines.
"The term "reverse distillation" has appeared in certain teacher-student architectures in ML literature. Our usage- decomposing large models using smaller ones a basis-is distinct and believe intuitive from context.
2

---

Published as a conference paper ICLR 2026
Computee
linear mapping between representations
Summarize residual subspace
Iteratively construct Matryoshka style embedding
NEKTMLNA...
MEKTNLNA..
ESM2- 1 50M residual
Each RD model contans
scale modelo an
ESM2-0M
ESM2-1 50M
U
Σ
X
ESM2-15B
ESM2-BM
ESM2-1 50M
ESM2 8M
X
H0-158
Redundant, entangled features
Onthogonalized feature decompoattion
Figure 1: Overview of Reverse Distillation Large protein language models (e.g., ESM-23a) entangle representations diverse features single representational space, hindering the performance downstream linear probes. Reverse distillation constructs product space by preserving the smaller model's representation (capturing more conserved fea- tures) and extracting orthogonal residuals via SVD (capturing features unique to the larger model), Iterating this process across model family yields Matryoshka-stye embeddings where each prefix corresponds to valid wverse-distilled representation that scale. Figure created using biorender.io
METHOD
The ESM-2 family spans embedding dimensions from 320 (8M parameters) to 5120 (15B parameters), providing systematic testbed for analyzing PLM scaling behavior. Each model learns residue co-evolution patterns through self-attention mechanisms, but capacity constraints induce dif- ferent feature distributions across model scales.
Motivation and Intuition
Small models operate under severe capacity constraints. To minimize loss training time, these models must prioritize the most frequently occurring co-evolutionary patterns- that maximize compression across the protein sequence distribution. These patterns likely correspond to general protein properties: secondary structure propensities, hydrophobicity patterns, and conserved structural motifs. The smallest ESM-2 model (8M parameters) lacks sufficient capacity to represent rarer patterns; its limited parameters are allocated to features with the highest marginal utility across the training distribution.
Large models possess capacity to represent more diverse functional features. They can encode enzyme- specific catalytic motifs, protein specif- allosteric couplings, and higher-order interaction patters.. However, as asLLi et al. 2024) demonstrated, downstream performance often relies on early, low-level features rather than additional capacity; consequently, linear probes larger, mixed representations frequently fail to isolate task-relevant signal from task-irrelevant variance.
Our key intuition is that smaller models provide a natural basis for decomposition (Fig. D A smaller model from the same family-trained trained identical data with the same architecture pronucccseepuuuussssseee representations biased toward broadly-shares features due to capacity constraints. By computing the orthogonal complement of the smaller model's is subspace within the larger model's representation, we achieve partial separation of features without requiring explicit feature identification.
We employ linear decomposition throughout our framework to maintain interpretability. Linear meth- ods reveal directly accessible information in representations without confounding from nonlinear prediction heads, and enable precise attribution of information to specific model scales. While nonlinear methods might achieve higher downstream performance, linear decomposition provides the analytical tractability necessary to characterize how biological information distributes across the model hierarchy.
2.1 PROBLEM FORMULATION
Consider hierarchy of protein language models M {M1, M2,..., Mm) ordered by parameter count. Each model M, maps sequences to embeddings;
M, P Bunk
3

---

Published as a conference paper ICLR 2026
where is the amino acid alphabet, n varies per sequence, and k < h2 ... < km are embedding dimensions. For the ESM-2 family, this hierarchy exists naturally, with model scales at 8M (k) = 320), 35M (k2 = 640), 150M (k33 480), 650M (k₃ 1280). 3B (ks 2560), and 15B (hg = 5120) parameters.
While our framework does require monotonically increasing dimensions appropriatee dimensionality reduction via PCA or variational autoencoders could enable reverse distillation between arbitrary model pairs- the ESM-2 family's 's architecture provides this structure directly, simplifying our implementation. For protein sequence dataset D {s, with sequence lengths (n.), the total amino acid positions = -=>' n2 represents our effective sample size for learning linear relationships. This formulation (treat- ing all positions samples) enables data-efficient subspace learning. For training, sequences sampled randomly from UniRef50; all sequences had 30% or lower sequence identity to datasets used in Section 3. Due to the simple linear transforms involved in this work, we found this N to be sufficient.
used N 1, 000
Reverse Distillation Decomposition
Definition (Reverse Distillation Decomposition). Given models M, and Mp where r < p, we decompose the representation space Ba*k, into orthogonal subspaces:
Sp S $ Sres
where S, = Br preserves M,'s 's representations and Sres = (nx((ky-kk) captures orthogonal residual
information.
We express any representation Hp E Buxkp as:
Hp [Hr, Hres]
where H, E Ra*k comes directly from the smaller model My, and Hyes E (nx((t- represents the unique contribution of the larger model.
We preserve entire smaller models rather than selecting subsets of their dimensions. This choice, abled by the natural progression of embedding sizes 1220=660011200-2550--
in ESM-2), maintains interpretability know H, represents the complete feature space learned by the smaller model, making the residual space Sres directly interpretable as features unique to the larger model.
2.2 ALGORITTMS
Algorithm D presents training procedure. Phase comprises forward passes of the M, and Mp. comput- ing representations of the same sequences at different scales.
Refinements the linear mapping. In Phase 2 of Algorithm 1, we use principal component regression (PCR) rather than ordinary least squares. Since some dimensions of H, are likely noise, we project onto its principal components and apply the Johnstone threshold (Johnstone 2001) from random matrix theory to discard likely-noise components before regression (Appendix A.I)) We also verified that reverse distil- lation performs comparably to a stronger but non- Matrryssk baseline: concatenating all model scales and reducing via PCA (Appendix A..2).
Algorithm shows inference. The decomposed representation Hyd = [Hr, Hres] is Matryoshka by construction prefixes correspond to valid smaller model outputs, enabling adaptive compute at deployment. Algorithm extends reverse distillation from pair of models to a whole family of models. Chaining reveals hierarchical structure where each scale contributes orthogonal information that cannot be linearly predicted from smaller models.
Let M, C be the manifold spanned by embeddings of Mp. To enable scala- bility and flexibility in our representation space, it is desirable to enforce the Matryoshka property on our embeddings. Thus, we consider the set of all dimeennnio representations that preserve M, 's embeddings their first k, coordinates:
Theoretical Analysis
C, ((H,, X]: H, E M,, X E
4

---

Published as conference paper ICLR 2026
Algorithm 1 Reverse Distillation Algorithm (Pre-training)
Require: Dataset D )s where |sy| R4, models My. Mp with r p
Ensure: Subspace decomposition matrices W Vre
1: Phase 1: Compute Representations
2: for i 1 to N do
3: H (s:): E *R {Variable length n
4: H = (x)))) E RR.kk,
5: end for
6: Phase 2: Learn Linear Mappings
7: Define total length: I. = Σi=1 f4i
8: Stack representations: H, = vstack HH,0) .., H, E LL
9: Stack representations: HI vstack 11, ..... -111) E L2
10: Solve: W arg minw ||,,, H, W W 1}
Via prin. comp. reg. (PCR); noise PCs of H, removed Compute residuals: R Hp H, W E RLLx,
12: Phase 3: Subspace Identification
13: Apply SVD: R UEV*
14: Select top (kp kp) components: Vres V(t1 (kp kz))
15: return W Vees
Our decomposition H,d [H,, Hree minimizes reconstruction enror within this constrained space:
Theorem 1 (Optimal Constrained Approximation). Let Hp E RLL and H, E LRL be stacked represen- tations from models Mp and M, respectively, where p. Among all representations of the form [il, X] where X TR. (kbpbe)),, the representation Hrd [H,, Hres) with Hres derived from the top (kip k
singular vectors of the residual Hp H,W minimizes:
min xE
HH, [H,, X]AI2
Proof. The proof directly follows from the Eckart- Young theorem. The optimal linear predictor W fi,)- Hp. minimizing the reconstruction error over all W E RR.xk, The residual Hp H, 4 contains information orthogonal to H,. For R = UΣVT, the optimal lank-Kk;- - approximation uses the top (k, k,) singular vectors.
Algorithm 2 Reverse Distillation Inference
Require: New sequence with |s| n. learned matrices W*, Vress models M,, Mp
Ensure: Decomposed representation Hra Ra*k,
1: H, = Ms) E nx (Smaller model embedding)
2: Hp MM(s) Raxk, [Larger model embedding)
3: Hpred H Baxk, (Predicted large model embedding)
4: R = Hp Hpred Bank, (Unexplained residuals}
5: Hres RV res (nnx(kp-k.)) (Projected residuals)
6: Hnd = [Hr, Hyea] E Raxk, [Concatenate reference + residual)
7: return Hyd
5

---

Published as conference paper ICLR 2026
Algorithm 3 Chained Reverse Distillation
Require: Dataset D, model hierarchy [M1,..., 3m
Ensure: Decomposition components (W,, Vi)
1: Initialize: Hace = Mi (D), 1.1 = kj
2: for i 2 m do
3: H; = M,(D)
4: Learn predictor: W; = arg a minw || H,
5: Compute residuals: R: H; 716-1)
6: Apply SVD: R, V
7: Select components: V; Vi(:. 1 (ke kacz!)))
8: Update: Hull = [Hlce)) R4 V
9: Update: [i) K;
10: end for
11: return (W;, V.
3 EXPERIMENTS
3.1 INITIAL EXPLORATION OF MODEL CHAIN CONFIGURATION
We began by investigating the optimal chaining of small models into larger models. For three DMS datasets from ProteinGym (Notnn et al. 2023
evaluated range of chain configurations. Let
.....,,, ,,,, denote the n + model sizes from a model family (for ESM-2: K {8M, 35M, 150M, 650 3B, 15BJ). For a target embedding kn with [0,n], nl.. the chain configuration was defined as follows:
K = (kg, ki.....
1. for each & [0, with < t, direct chain k; le
2. longest chain: kg- + kj > l
As shown in Table consistent trend emerged in which longer incremental chains yielded improved performance. Consequently, we concentrated our comprehensive experiments on the results obtained from reverse distillation of the largest models.
In the rest of the paper, we denote the chain K&M ..... Resoss rd.650, the chain keas - KaB as rd.3B, and the chain KgMs + + kisB as rd.15B.
3.2 PROTEINGYM DMS ANALYSIS
For comprehensive analysis, obtained datasets from ProteinGym with at least double- multi- mutation variant. To ensure that our evaluation estimates were reliable, we excluded datasets with fewer than 100 singlc-mutation variants. Given an embedding scheme, for each protein in the dataset, we loaded the embedding of the wild-type sequence and the embeddings of the mutated sequence. For each mutation, we computed the difference vector between the embeddings the mutated sequence and the corresponding wild-type sequence at the mutated position, feeding it into a ridge regression classifier. For variants with multiple mutations, we first average the differences across mutated positions. We fit the ridge regression on 80% of the single-mutational variants using leave-one-out cross-validation. The trained model was used and remaining single-mutant cases. Note that since rd.650M (rd.3B)
predict for all multiple-muten a prefix of rd.3B (rd. 15B) by construction (the first 1280 (2560) dimensions are identical), any cases where rd.650M (rd.3B) outperforms rd.3B (rd.15B) likely reflect ridge regression artifacts rather than representa- tional limitations. For each DMS dataset, computed the Spearman correlation between scores and the ground truth; we followed the original ProteinGym manuscript in our choice of the Spearman metric. We report estimated per-dataset measure of improvement, asking "in how many datasets does
predicted
6

---

Published as conference paper ICLR 2026
Table I: Progressive chain vs. direct chain. Our approach supports reverse distillation of any smaller model into any larger model. However, we find empirically that the best performance comes from progressively distilling up a "chain" of models. the remainder this paper, we refer to these progressive chains as rd vd.650M, rd.3B, and rd. 15B.
ARGR ECOLI Tsuboyama 200Y
DN7A SACS2
ILF3 HUMAN Tsuboyama 2023 2133
ESM Models
Tsuboyama. 20IC.IJIC
8M
0.771
0.746
0.670
35M
0.767
0.806
0.692
rd: 8M-+35M
0.776
0.793
0.701
150M
0.799
0.786
0.760
rd: 15M-+150M
0.811
0.791
0.772
rd: 8M-+150M
0.820
0.792
0,779
650M
0.834
0.868
0.712
rd: 8M-650M
0.849
0.878
0.765
rd: 35M->6500 >650
0.835
0.881
0.759
rd: 150M->650M
0.845
0,866
0.751
rd: 8M 8M-+35-00
~650 (rd.650M)
0.858
0.867
0.786
3B
0.845
0,880
0.749
rd: 8M-+3B
0.852
0,898
0.780
rd: 35M-+3B
0.844
0,894
0.777
rd: 150M-+3B
0.853
0,886
0.775
rd: 650M
0.859
0,880
0.751
rd: --5 11.50-6650-3BB (rd.3B)
0.873
0.890
0.801
along with the mean and standard deviations of these correlations for both the ESM-2 family of models and their reverse-distilled version (Table
model Mj outperfom M,?" (Table 2
3.3 ADDITIONAL PROTEIN PROPERTY PREDICTION
We evaluated our everre-distilled models on several downstream protein property prediction tasks where the prediction task directly corresponded to protein structural, functional, and dynamic features. We utilized the 3-class secondary structure prediction (SSP Q3), 8-class secondary structure prediction (SSP Q8), metal ion binding (MIB) and localization prediction (LOC) benchmarks from Chen al. (2024), well the R2/R1 prediction task from Wayment-Steele et al. (2025). Training was performed analogously to the previous
Table Reverse Distillation restores scaling on ProteinGym benchmarks. We show that not only do rd. dd.650M, rd.3B, and 15.15B consistently outperform their baseline counterparts, but that reverse distillation preserves expected scaling, i.e. larger models more frequently outperform smaller models.
% ProteinGym m DMS Datasels where one model outperforms another
ProteinGym
DMS Datasets
650M
15B
15B
3B 650M
15B
ProteinGym DMS Datasets with and mutations
mut
71.43% 53.57%
85.71%
86%
53,57% 46.43%
92.86%
92.86%
85.71%
53.57%
57.14%
85.71
67.86%
75.00%
ProteinGym ym DMS Datasets with :>2 mutations
mut
83.33%
16.67%
66.67%
100.00%
33.33%
100.00%
66.67%
2 mul mot
33.33%
66.66%
83.33%
33.33%
83.33%
100.00%
100.00%
4 mul 66.66%
66.66%
66.66%
66.67%
83,33%
100.00%
100.00%
50.00%
50.00%
50.00%
50.00%
66.67%
100.00%
100.00%
7

---

Published as a conference paper ICLR 2026
Table 3: Spearman correlation of predicted mutational effect on ProteinGym benchmarks. We report the performance of classifier trained embeddings from various models to predict variant effect on deep mutational scanning (DMS) data from ProteinGym. rd. 15B achieves the strongest performance out of any model tested, and the reverse distilled models generally outperform their baseline counterparts.
Test Spearman correlation
# ProteinGym DMS Datasets
(mean + std)
650M
650M
rd 15B
ProteinGym DMS Datasets with mutations
28
0.879 0.040)
0.885 (#: 0.038)
0.881 (# 0.043)
0.893 (+: 0.039) 0.697 (+ 0.115)
0.899 (#: 0.036)
0.904 (=: 0.037)
2 mut
28
0.672 (+ 0.140)
0.688 (+ 0.133)
11672 (+ 0.128)
0.714 (+ 0.115)
0.720 (+ 0.120)
ProteinGym DMS Datasets with 2-2 mutations
mut
6
0.534 (#: 0.192)
0.550 (::: 0.187)
0.591 (# 0.189)
0.574 (#: 0.165)
0.579 (:: 0.180)
0,580 (= 0.189)
mut
0.608 0.131)
0.600 (d: 0.137)
0.621 (# 0.146)
0.643 0.132)
0.653 (:: 0.156)
0.677 (= 0.134)
0.578 (+: 0.056)
0.577 (+ 0.068)
0.583 (+ 0.103)
0.617 (+ 0.082)
0.607 (+ 0.183)
0.652 (- 0.084)
4 mut
0.555 (+: 0.067)
0.539 (d: 0.089)
0.552 (# 0.145)
0.576 (+: 0.090)
0.566 (:: 0.171)
0.615 (= 0.086)
setting. The reverse-distilled models frequently outperformed the base models (Table 4 and demonstrated consistent scaling, with rd.15B nearly always outperforming rd.3B and the baseline ESM models.
3.4 PROBING EMBEDDINGS WITH SPARSE AUTOENCODERS
Finally, we sought to explore whether our reverse-distilldd embeddings could disentangling representations of biological features by training sparse autoencoders (SAE). We followed the analysis of Gujral et al. 2025). training an SAE on embeddings from both the base ESM-2 35M and our rd.35M model. Using set of 18,142 proteins from the UniProt database, we identified the proteins significantly associated with each sparse feature, then performed a GO enrichment analysis using annotations for the proteins (Ashburner et al, 2000 to identify the set of GO terms associated with each sparse feature. We found that the SAE trained on rd.35M embeddings contained more enriched GO terms (40 GO terms for the average rd.35M SAE feature, vs. 32 for the average ESM-2 35M SAE feature) indicating that these embeddings capture more functional features than those from the base model (Figure 2h).
In addition, the DAG relationship between GO terms allows us to probe the relationships within each functional set. Following Gujral et al. (2255) we compute "monospecificity." the inverse of average shortest pairwise distance between GO terms in the same set (Figure 2p), and "generality," the depth of the least common ancestor (LCA) of each pair of terms in the set (Figure 2P). A high monospecificity means that SAE features capture a more consistent set of functional features, while a high generality means that SAE features
Table 4: Reverse distillation of ESM-2 improves performance on downstream protein property pre- diction tasks. We evaluate ESM-2 650M, 3B and 15B and their corresponding reverse-distilles versions on four data sets from Chen et al. (2024: 3-class secondary structure prediction (SSP Q3), 8-class secondary structure prediction (SSP Q8), metal ion binding (MIB), and localization prediction (LOC)) We also report Across nearly all data sets, we find that rd. 15B
on R2/R1 prediction from Wayment-Steele et al. 2025
achieves the strongest performance.
Dataset
650M
rd 650M
3B
rd 3B
15B
rd 15B
SSP Q3 (aupr 1)
0.831
0.833
0.791
0,816
0.845
0.861
SSP Q8 (aupr 1)
0.365
0.369
0.379
0.395
0.418
0.431
MIB (aupr f)
0.881
0.855
0.893
0.891
0.900
0.901
LOC (aupr 1)
0.709
0.669
0.705
0.708
0.745
0.743
R2/RI (aupr 1)
0.343
0.405
0.369
0.425
0.368
0.468
8

---

Published as conference paper ICLR 2026
a
100
35M
115
Number of sparse fealures
102
Dietiled 35M
Weighted Monaspechilly
Weightes Gemeraiity
80
10a
40
104
100
250
500
750
1000
1250
1800
E9M2 35M
Reverse Distiled 8M 35M
ESM2 35M
Rervrrse Dislited 8M 35M
Number enriched GO
Figure 2: Reverse Distillation embeddings capture GO terms. (a) SAE features from the rd.35M model are enriched for more GO terms than those from the base model, indicating that they contain more functionally relevant information. (b) The sets of GO terms for each model are equally compact, measured by pairwise shortest path on the GO tree. (c) The sets of GO terms for rd.35M SAE features are significantly less general, measured by the depth of the pairwise least common ancestor on the GO tree. While features from the base model capture high-level terms, reverse distillation enables model features to represent distinct functions.
capture less specific biological function. While rd.35M and ESM-2 35M embeddings have similar levels of monospecificity, rd.35M embeddings are significantly less general, lending support to our hypothesis that reverse distillation helps to disentangle representations of biological features within the model.
3.5 INFERENCE TIME
On Nvidia A6000 GPU, embedding protein sequence (mean length 536) took 0.09s and 0.249s for the ESM-2 650M and 3B models respectively. Even though rd.650M involves four ESM model-invocations (8M, 35M, 150M and 650M) it only took (66x(=00.12s) the time the smaller models have faster infer- ence. Similarly, rd.3B makes five model-invocations but took only 1.53x (=0.380s) the time compared to baseline ESM-2 3B, and the six model-invocations of rd. 15B take only 1.70x as long the base model. Thus, reverse distillation does not have a prohibitive inference overhead. We also note that the prefix struc- ture the embeddings enhances reusability.
4 RELATED WORK
The PLM scaling problem is well-documented. The ProteinGym benchmark shows that performance gains plateau around 1-4B parameters, with hybrids leveraging MSAs or structure often winning on zero- shot fitness (Noinn 2025). Multiple lines of evidence reinforce this ceiling: improvements with scale are largely deseneepnn,nnt, with linear probes on large representations struggling isolate task-relevant sig- nal (Li et al. 2024h; the ESM-2 3B model delivers contact-recovery signals comparable to the 15B vari- ant Zhang et al. 2024p: medium-sizze models perform competitively realistic transfer settings (Vieira et al. 2025): and variant-effect accuracy peaks at intermediate model perplexity, with both under- and over- confident models degrading discrimination (Hou et al., 2025). Complementary work on representational be substantially compressed along both sequence and feature dimensions (Lu et al. 2025 Devkota et al. 2024), suggesting that large models devote significant capacity redundant entangled features. Collectively, these findings characterize the scaling problem; reverse distillation offers a solution by systematically decomposing representations across scales.
redundancy shows that PLM embeddings
Relationship to distillation, model combination, and continual learning. Traditional knowledge distil- lation (Hinton et al., 2013) compresses large teacher into a smaller student; model soups (Wortsman et al., 2022 average weights across fine-tuned variants. Both aim to consolidate information into single model.
9

---

Published as a conference paper ICLR 2026
Reverse distillation instead decomposes representations across model scales, identifying each scale's unique contributions. Our use of orthogonal subspaces is related to continual learning methods like o-LoRA (Wangl al. p023p and Adaptive SVD (Naya et al. 2025), which maintain orthogonal weight subspaces when adapting to new tasks. However, these methods are task-specific; reverse distillation is task-agnostic, ex- tracting representations from a multi-scale model family by maximizing residual information gain.
Relationship to interpretability methods. Attention-head analyses ("BERTology") probe what individ- ual heads encode (Rogers et al. 2020 Clark et al. 2019 Vig et al.] 202T). while sparse autoencoders (SAEs)
enumerate latent features explicitly (Gujral et al. 2025; Adams et al. 2025 Both inspect representations but neither decomposes them across model scales. Moreover, mapping SAE latents to biologically relevant features demands extensive manual annotation. Reverse distillation sidesteps this by operating implicitly- separating what different model scales contribute without pre-defining or cataloging individual features.
5 CONCLUSION
The success of reverse distillation suggests that scaling challenges in PLMs stem from inefficient use of representational capacity rather than fundamental limits model expressiveness. A purely linear decomposition-
no model retraining overhead-reesores monotonic scaling and improves over baselines at the same embedding dimensionality. This indicates that the information needed for consis- tent scaling is already present in large models; the challenge is extracting it. By reframing the question from "when do large models help?" to "how can we systematically combine contributions across scales?", reverse distillation opens new avenues for representation analysis and more effective scaling strategies.
Limitations and Future Work. Our current framework relies on linear decomposition, which provides theoretical guarantees and interpretability but may leave nonlinear relationships between model scales unex- ploited. Initial explorations of nonlinear mappings show improved reconstruction (8M -> 35M R² 0.528 vs. 0.422; 650M + 3B R = 0.400 vs. 0.261), suggesting room to further extract unique features from larger models. Nonlinear dimensionality reduction (e.g., UMAP (Meinnes al.) 2018)) could also more effectively separate residual features, and would be particularly valuable for model families where different- sized models share the same embedding dimension.
complementary direction is to use parameter-efficient fine-tuning (e.g.. LoRA) to produce reverse- distilled embeddings directly at the last layer of a large model. This would enable generative use-cases and likelihood-basee probing while requiring only single forward pass, eliminating the current multi-model inference overhead.
Finally, we plan to test reverse distillation on other biological foundation mooelss-ncludiie autoregres- sive PLMs like ProGen, as well as models for genomics and drug discovery- and on foundation models outside biology. We anticipate that the core principle of leveraging smaller models to effectively decompose larger ones may apply broadly wherever scaling challenges persist.
ACKNOWLEDGMENTS
R.S. acknowledges the support of the Whitehead Scholarship the Duke University School of Medicine. D.C. acknowledges the support of the Karsh International Scholars Program from Duke University. S.S. acknowledges support from the Simons Foundation.
6 REPRODUCIBILITY STATEMENT
To facilitate easy verification and replication of our results, we have modularized our source code, added README documentation and provided information on how to train the reverse distillation mod- www.iitsiighhapppmmmm...
els. Code is available on GitHub at ltttt::// pettps://qithub. distillation, and pre-trained models are available to download via the package.
10

---

Published as a conference paper ICLR 2026
REFERENCES
Etowah Adams, Liam Bai, Minji Lee, Yiyang Yu, and Mohammed AlQuraishi. From mechanistic inter- pretability to mechanistic biology: Training, evaluating, and interpreting sparse autoencoders on protein language models. In ICML, 2025. URL htttp:://ppereeiiiieit wttp::.eee//oouueeeeeeeeeee forr//ooommmddde 2
Michael Ashburner, Catherine A Ball, Judith A Blake, David Botstein, Heather Butler, J Michael Cherry, Allan P Davis, Kara Dolinski, Selina Dwight, Janan T Eppig, et al. Gene ontology: tool for the unification of biology. Nature Genetics, 25(1):25-299 2000.
Bo Chen, Xingyi Cheng, Pan Li, Yangli-ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, et al. xtrimopglm: unified 100b-scale pre-trained transformer for deciphering the language of protein. arXiv preprint arXiv:2401.066 19949999.999..... 2024.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What Does BERT Look At? An Analysis of BERT's Attention. In Proceedings of BlackboxNLP, 2019.
Kapil Devkota, Daichi Shonai, Joey Mao, Young Su Ko, Wei Wang, Scott Soderling, and Rohit Singh. Miniaturizing, Modifying, and Magnifying Nature's Proteins with Raygun. bioRxiv, pp. 2024-08, 2024.
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. ProtGPT2 is a deep unsupervised language model for protein design. Nature Communications, 13(1):4348, 2022.
Onkar Gujral, Mihir Bafna, Eric Alm, and Bonnie Berger. Sparse autoencoders uncover biologically inter- pretable features in protein language model representations. PNAS, 2025. doi: 10. 10.10733pnas.255666 25063 1616621612..
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0253) 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute optimal large language models. arXiv preprint arXiv:2203.15556 2022.
Chao Hou, Di Liu, Aziz Zafar, and Yufeng Shen. Understanding language model scaling on protein fitness prediction. bioRxiv, 2025.
Jain M. Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of Statistics, 29(2):295-327) 2001.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.00 003361, 2020.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder. Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233-30249, 2022.
Francesca-Zhoufann Li, Ava P Amini, Yisong Yue, Kevin K Yang, and Alex X Lu. Feature reuse and scaling: Understanding transfer learning with protein language models. bioRxiv, pp. 2024-02, 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction atomic-level protein structure with language model. Science, 379 1130, 2023.
11

---

Published as a conference paper ICLR 2026
Amy X Lu, Wilson Yan, Kevin K Yang, Vladimir Gligorijevic, Kyunghyun Cho, Picter Abbeel, Richard Bonneau, and Nathan C Frey. Tokenized and continuous embedding compressions of protein sequence and structure. Patterns, 6(6), 2025.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.
Nikhil Shivakumar Nayak, Krishnateja Killamsetty, Ligong Han, Abhishek Bhandwaldar, Prateek Chanda, Kai Xu, Hao Wang, Aldo Pareja, Oleg Silkin, Mustafa Eyceoz, et al. Sculpting subspaces: Constrained full fine-tuning in llms for continual learning. arXiv preprint arXiv:500.070097, 2025.
Pascal Notin. Have we hit the scaling wall for protein language models? substack www.c//cc/////// htaplppaaveweee wittt sca
2025. Accessed: 2025-09-25.
Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al. ProteinGym: Large-scale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing Systems, 36:64331-64379, 2023.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Computational Linguisties, 2020. doi: 10.1162/ t0034.
Luiz C Vieira, Morgan L Handojo, and Claus O Wilke. Medium-sized protein language models perform well at transfer learning on realistic datasets: Le vieira al. Scientific Reports, 15(1):21400, 2025.
Jesse Vig, Martin Jaggi, and et al. BERTology Meets Biology: Interpreting Attention Protein Language
Models. In ICLR, 2021.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan- Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10658-10671, 2023.
Hannah Wayment-Steele, Gina Nesr, Ramith Hettiarachchi, Hasindu Kariyawasam, Sergey Ovehin- nikov, and Dorothee Kern. Learning millisecond protein dynamics from what is missing in nmr spectra. bioRxiv, pp. 2025-03, 2025.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: eraging weights multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965-23998. PMLR, 2022.
Tianhao Yu, Haiyang Cui, Jianan Canal Li, Yunan Luo, Guangde Jiang, and Huimin Zhao. Enzyme function prediction using contrastive learning. Science, 379(6639): 1358-991133811330 1363, 2023.
Zhidian Zhang, Hannah K Wayment-Steele, Garyk Brixi, Haobo Wang, Dorothee Kern, and Sergey Ovehin- nikov. Protein language models learn evolutionary statistics of interacting sequence motifs, Proceedings of the National Academy of Sciences, 121665))e2442285121,, 2024.
12

---

Published as conference paper ICLR 2026
A APPENDIX
A.1 ABLATION ON LINEAR MAPPING DURING REVERSE DISTILLATION (PHASE 2 ALGORITHM
I)
To map the smaller model embeddings H, to the larger model embeddings Hp. we explored two approaches: a) an ordinary least-squares regression (OLS), and b) prineiple-component regression (PCR). Since there may be "noise" features in H,, we hypothesized that using PCR would let utilize only the informative features of Hr for predicting Hp. Towards that, we first mapped μr, to its principal components, and then applied principles from random matrix theory (specifically, the Johnstone threshold (Johnstone) 200T)) to select the top r; eikely-nonnooie PCs, and used only these to predict Hy. We compare the performance of PCR and OLS in Figure showing that PCR consistently yields stronger downstream performance across multiple mutation data sets.
Ablation Linear Mapping: ProteinGym performance of PCR vs OLS
Model Size: 158
Model Size: 38
Model Size: 650M
0.5
0.9
ProtennGym Connlettor OL5
C.
ProteinGym Comnuatton:
0.7
0.7
R.R.
0.8
0.6
ProteinGym
0.6
Tesls
Texts
hols
Luner
0.5
1
4,
0.4
0.4
0.7
0.5
0.5
0.7
0.8 0.9
0.6
0.5
0.2
0.9
ProteinGly Correlation: PCR
ProtainGym Corre ation: PCR
Proteindyy Corretation: PCR
Figure 3: Ablation Linear Mapping The plots show the performance of the rd.650M, rd.3B, and rd.15B models using both PCR and OLS and evaluated on their Spearman Correlation (p) across 37 datasets from the Protein Gym benchmark. They show that PCR consistently outperforms OLS, with overall win rates 58.8%, 60.3%, and 63.2% respectively. This suggests that isolating signal from noise in the lower-scale representations is critical for effective cross-scale distillation
A.2 ABLATION ON SCALE-PRESHRVING FEATURE CONCATENATION
An alternative to Matryoshka approach-whil while still preserving linear subspace-decomposice approach-is to simply concatenate embeddings models and then perform dimensionality reduction via principal component anlaysis (PCA). Thus, distill embeddings H, into Hp. would concatenate the two, perform PCA, and take the top components. In a sense, this represents the upper bound of rep- resentational orthogonality achievable by any linearly-construutee p-dimensional subspace that invokes all model scales up to Hp. At the same time, these representations are not Matryoshka-style: when graduating higher model scale, the resulting embeddings (since they will be produced by fresh PCA) may be entirely different. As we show in Figure A.2 our reverse distillation representations performs similarly to this representation while preserving the Matryoshka property.
13

---

Published as conference paper ICLR 2026
Ablation Matryoshka property: ProteinGym performance of method concatenation + PCA
Model Sizes
Model Size:
Model Size: 650M
ProteinGym Corrsstion Concatenation RCA
ProtetrOym Comeestionn Conratenstion PCA
0.9
Proteindpm Carelaztonn Concaltration
08
0.0
0.7
0.6
Taits Line 2umut
0.5
1urur
Lesa
7,mu
2
Zumat
y-x
0.4
0.7
0.8
0.9
04
0,5
0.6
0.7
0.4
0.3
0.6
0.9
ProteinGym Correlatinn Reverse Distilacion ladien
PreteinGym Correlation: Reverse Distilation
ProteinGym Correction: Reverse latiination
Naive PCA concatenation scalability ProteinGym DMS Datasets
naive 3B naive 650M naive 15B naive 3B
rd 3B
rd 650M
rd 15B > rd 3B
ProteinGym DMS Datasets with 1 and 2 mutations
mut 85.71%
71 71.43%
92.86%
85.7 1% 75.00%
2 mut 76.19%
57.14%
67.86%
ProteinGym DMS Datasets with >2 mutations
mut 85.71% 1%
14.29%
100.00%
100.00%
2 mut 85.71% %
71.43%
100.00%
66.67%
mut 85.71% %
71.43%
100.00%
100.00%
mut
85.71%
71.43%
100.00%
100.00%
Figure 4: Ablation on Matryushka property: (Top) Comparison of Spearman correlation on ProtcinGym datasets
between our Reverse Distillation approach and PCA+ + concatenation baseline. While the baseline provides thcoretical upper bound for linear subspace decomposition, (Bottom) highlights its critical limitation: unlike our method, relies on fresh projections each scale therefore cannot preserve structural consistency or scalability across 650M, 3B, and 1SB models.
14


Reviews:
[{"review_id": "x1NnOBwJYE", "reviewer": "Reviewer_XboP", "summary": "The paper addresses a significant problem in biological foundation models: they scale poorly compared to models in natural language processing. Specifically, larger Protein Language Models (PLMs) in families like ESM-2 often underperform smaller ones on key benchmarks, a phenomenon known as non-monotonic scaling. The authors hypothesize this is because small models capture \"universal\" features (like secondary structure), while larger models add \"specialized\" features (like protein-family specific functions). When these features are entangled in a single representation, the specialized features can act as noise, degrading performance on tasks that rely on universal patterns. To solve this, the paper introduces a method that decomposes large protein language model representations into orthogonal subspaces guided by smaller models. On benchmarks like ProteinGym and BioMap, the reverse-distilled ESM-2 models (e.g., rd.650M) broadly outperform their corresponding baselines (e.g., 650M).", "questions": "1. The paper hypothesizes that $H_{r}$ captures \"universal\" features and $H_{res}$ captures \"specialized\" ones. Beyond downstream task performance, did you conduct any qualitative analysis to verify this? For example, could you use feature attribution or probing to show that $H_{res}$ contains information about specific protein-family motifs or epistatic interactions that are demonstrably absent when probing $H_{r}$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper establishes the central problem: PLMs \"scale relatively poorly\" , with the ESM-2 family's performance plateauing. The authors' core hypothesis is highly intuitive that this is due to larger models \"entangling\" universal (low-level) and specialized (high-level) features, which increases variance.\n\n2. This work introduces high-performing and efficient embeddings. The resulting models outperform baselines at the same size. They also feature a \"Matryoshka-style\" structure, which allows smaller prefixes of a single embedding to be used as valid, lower-dimensional representations, saving computation and storage.\n\n3. The experimental design is comprehensive. The authors test their method on standard, challenging benchmarks, including ProteinGym DMS and BioMap. The inclusion of practical analyses, such as an ablation on the training data size and a measurement of inference overhead, further strengthens the work's quality.", "weaknesses": "1. The authors should at least attempt a reverse distillation of 3B $\\rightarrow$ 15B (or the full chain up to 15B). This experiment is critical. If rd.15B outperforms rd.3B, the paper's core thesis is validated. If rd.15B still underperforms, it would suggest the scaling problem is more complex than just feature entanglement, fundamentally weakening the paper's conclusion.\n2. This linear-only approach may be restrictive. The paper itself hypothesizes that larger models encode rarer, higher-order phenomena. These complex, higher-order features may not be neatly separable from the universal features via a simple linear projection. The authors' method might only be extracting the linearly predictable component, leaving a \"residual\" that is still a mix of true novel features and non-linear transformations of universal features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5e8djWvgT3", "reviewer": "Reviewer_akEp", "summary": "This paper improves PLMs through a model distillation process which decomposes large models into smaller sub-models with disentangled residuals. The resulting embeddings also enjoy the Matryoshka property which allows for slices of embeddings to remain informative. These models also recover better scaling properties, allowing for more efficient parameter use. Benchmarking was done on ProteinGym and BioMap, showing good predictive performance as well as scaling with model size.", "questions": "n/a", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper is quite strong in my opinion. The proposed methods are a clear improvement on current approaches and is a valuable contribution to the protein representation field.", "weaknesses": "* Additional inference time though not prohibitive could still limit adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SRgG85Lyek", "reviewer": "Reviewer_nDBC", "summary": "This paper addresses the well-known \"counterintuitive scaling\" problem in Protein Language Models (PLMs) like ESM-2, where larger models often perform worse than medium-sized models on downstream benchmarks.1 The authors hypothesize this is due to \"feature entanglement,\" where larger models mix \"universal\" features (from small models) with \"specialized\" features, and this mixture acts as noise for standard linear probes.2 The authors propose \"Reverse Distillation\" (RD), a novel and elegant post-hoc framework. Instead of compressing, RD uses a smaller model's representation ($H_r$) as a basis and decomposes a larger model's representation ($H_p$) into an orthogonal combination $[H_r, H_{res}]$, where $H_{res}$ captures the new, orthogonal information from the larger model.2 The method is theoretically grounded (Theorem 1) 2 and empirically shown to restore monotonic scaling (i.e., the rd.3B model consistently beats the rd.650M model) on ProteinGym and BioMap benchmarks.2", "questions": "see weakness", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper tackles a critical, well-documented problem (PLM scaling failure ) with a highly novel solution. The idea of using smaller models as a basis for post-hoc orthogonal decomposition is elegant and new.\n2.  The experiments persuasively demonstrate that RD works. It not only improves baseline performance (e.g., rd.3B > 3B) but, more importantly, it restores monotonic scaling (rd.3B > rd.650M wins 96.4% of the time, vs. 53.6% for the baseline).\n3.The BioMap experiment (Table 4) provides strong evidence for the \"feature entanglement\" hypothesis. RD specifically fixes the scaling failure on \"universal\" features (like secondary structure) that the paper hypothesized were \"drowned out\" in larger models.\n4. The method is post-hoc, requiring no model retraining. The \"Chained\" version provides a practical, novel way to create Matryoshka-style nested embeddings from an existing model family.", "weaknesses": "The paper's primary weakness is the exclusion of the ESM-2 15B model.2 The most severe example of scaling failure is the performance degradation from 3B to 15B.2 The paper only demonstrates fixing the 650M-to-3B plateau. Without testing the 15B model, the central claim of \"solving\" the scaling paradox is incomplete.\nThe core idea of using orthogonal subspaces to separate/disentangle knowledge, while novel in this application, is conceptually similar to methods in continual learning (e.g., O-LoRA), which should be cited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Pci3sPTUz", "reviewer": "Reviewer_xk7u", "summary": "This paper tries to investigate why larger Protein Language Models such as ESM-2 fail to exhibit the expected scaling gains.\nThe authors attribute this to the entanglement of general and specialized representations within large models, which increases the variance of linear probes.\nTo address the above issue, this paper proposes Reverse Distillation, a linear subspace decomposition method that uses smaller models to define a general feature subspace and extracts orthogonal residuals from larger models to represent specialized knowledge.\nExtensive experiments on ProteinGym and BioMap show that RD consistently improves predictive performance.", "questions": "All my concerns about this paper are stated in the weakness section. Please refer to the weakness section for rebuttal/discussion.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper tries to tackle an important issue in large Protein Language Models (PLMs): the unexpected degradation of scaling behavior\n2. The proposed Reverse Distillation method is computationally lightweight and purely linear, involving only least-squares fitting and SVD decomposition\n3. Despite the method’s simplicity, RD exhibits stable and monotonic performance gains across multiple datasets and scaling levels.", "weaknesses": "1. The paper builds its entire motivation on the “general vs. specialized representation” hypothesis but does not provide a quantitative or qualitative analysis to validate it\n2. The proposed Optimal Constrained Approximation theorem only guarantees minimal reconstruction error under a prefix constraint: a standard property of linear least squares combined with SVD. However, this result does not theoretically justify why the assumed decomposition is needed or effective.\n3. Since RD performs a chain-wise representation enhancement, a straightforward baseline naturally arises: direct representation fusion across the same model chain. It remains unclear whether RD’s improvement comes from its “distillation mechanism” or simply from aggregating multi-scale features. A comparison against naive fusion, or fusion with simple KD objectives, is essential to establish the method’s actual contribution.\n4. The experiments demonstrate improvement within the ESM-2 family, but it is uncertain whether the observed scaling restoration generalizes to other architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)