Next Token Perception Score: Analytical Assessment of your LLM Perception Skills

## Abstract

Abstract Autoregressive pretraining has become the de facto paradigm for learning general-purpose representations in large language models (LLMs). However, linear probe performance across downstream perception tasks shows substantial variability, suggesting that features optimized for next-token prediction do not consistently transfer well to downstream perception tasks. We demonstrate that representations learned via autoregression capture features that may lie outside the subspaces most informative for perception. To quantify the (mis)alignment between autoregressive pretraining and downstream perception, we introduce the Next Token Perception Score (NTPS)—a score derived under a linear setting that measures the overlap between autoregressive and perception feature subspaces. This metric can be easily computed in closed form from pretrained representations and labeled data, and is proven to both upper- and lower-bound the excess loss. Empirically, we show that NTPS correlates strongly with linear probe accuracy across 12 diverse NLP datasets and eight pretrained models ranging from 270M to 8B parameters, confirming its utility as a measure of alignment. Furthermore, we show that NTPS increases following low-rank adaptation (LoRA) fine-tuning, especially in large models, suggesting that LoRA aligning representations to perception tasks enhances subspace overlap and thus improves downstream performance. More importantly, we find that NTPS reliably predicts the additional accuracy gains attained by LoRA finetuning thereby providing a lightweight prescreening tool for LoRA adaptation. Our results offer both theoretical insights and practical tools for analytically assessing LLM perception skills.

## 1 Introduction

Figure: Figure 1: 2D projections of the representations learned through perception $U^{\top}X$ (left) or autoregressive $V^{\top}X$ (right) tasks based on OpenELM-450M and Emotion[1] dataset. In the perception space, joy and sad words are clearly separated, illustrating label-relevant semantic encoding; in the autoregressive space, these two emotion classes overlap, but syntactic categories become clearly clustered.
Refer to caption: x1.png

The success of GPT-1 [2] demonstrated the effectiveness of autoregressive pretraining—training a model to predict the next token given preceding context—for learning transferable language representations. This autoregressive training paradigm quickly became the standard for building large language models (LLMs), leading to increasingly capable successors such as GPT-2/3/4 [3, 4, 5] and LLaMA-1/2/3/4 [6, 7, 8, 9]. As model capabilities scaled, the expectation evolved: rather than merely serving as a source of transferable features requiring task-specific fine-tuning, a foundation model is now expected to perform well on downstream perception tasks straight out of the box, without modifying its pretrained weights.

Under this expectation, prompting has emerged as a popular strategy [3]—adapting the model to downstream perception tasks solely by manually crafting input text. Meanwhile, linear probing has gained traction as an alternative [10, 11], leveraging frozen representations and training only a lightweight linear classifier on top, offering a more structured and efficient approach to downstream adaptation. While prompting and linear probing may appear quite different in practice, they are fundamentally two sides of the same coin. In prompting, natural language input steers the model, and the model’s pretrained linear head for autoregression maps hidden states to output token probabilities. In linear probing, by contrast, the model is frozen and an external linear head is trained to interpret its hidden representations.

However, as we show in [Section 3](https://arxiv.org/html/2505.17169v1#S3), linear probing does not perform equally well across all downstream perception tasks. In some cases, training a model from scratch on the downstream dataset yields significantly better performance. This suggests that representations learned by current LLMs are not universally effective: while some downstream tasks benefit greatly from pretraining and are well-served by simple adaptation methods like linear probing, others are not. Motivated by this observation, we pose the following question:

*How can we quantify the alignment between autoregressive pretraining and downstream perception tasks to explain the varying effectiveness of linear probing across different datasets?*

In this paper, we take a first step towards understanding and assessing the alignment between autoregressive pretraining and downstream perception tasks by:

- •
Systematically evaluating the benefits of autoregressive pretraining—by comparing linear-probe performance on six pretrained models against identical architectures trained from scratch across 12 downstream perception datasets.
- •
Proposing the Next Token Perception Score (NTPS)—a metric that quantifies the alignment between autoregressive pretraining and downstream perception tasks by measuring the overlap between their respective feature subspaces.
- •
Empirically validating the reliability of NTPS—by demonstrating that it not only correlates strongly with linear probe accuracy across 12 diverse datasets and eight pretrained models, but also consistently increases after task-specific LoRA fine-tuning.
- •
Guiding LoRA finetuning with NTPS predictions—by demonstrating that NTPS reliably forecasts the additional accuracy gains from LoRA finetuning, thereby providing a lightweight prescreening tool.

The remainder of the paper is organized as follows. [Section 2](https://arxiv.org/html/2505.17169v1#S2) reviews both empirical and theoretical foundations for applying pretrained LLMs directly to downstream tasks. [Section 3](https://arxiv.org/html/2505.17169v1#S3) presents evidence that pretrained LLM representations are not universally effective when used directly through linear probing and therefore we introduce our proposed NTPS metric to quantify such (mis)alignment. [Section 4](https://arxiv.org/html/2505.17169v1#S4) empirically demonstrates that NTPS correlates with linear probing performance across 12 downstream datasets and eight pretrained models and shows that NTPS consistently increases after task-specific LoRA fine-tuning. [Section 4](https://arxiv.org/html/2505.17169v1#S4) further suggests that NTPS can serve as a good predictor of performance gain from LoRA finetuning. [Section 5](https://arxiv.org/html/2505.17169v1#S5) summarizes our findings. Our code is available at: [https://github.com/Yu-AngCheng/NTPS](https://github.com/Yu-AngCheng/NTPS).

## 2 Background

#### Utilizing Pretrained LLM Hidden Representations for Downstream Tasks

Pretrained large language models can be leveraged for downstream tasks without any gradient-based fine-tuning via two complementary strategies: prompt-based in-context learning and linear probing of fixed hidden representations. In prompting, GPT-3 attains strong zero- and few-shot classification performance across diverse NLP benchmarks using only natural-language templates and without any weight updates [4]. Likewise, when given only in-context examples, GPT-3 can perform both linear and non-linear regression at levels comparable to Random Forests and Gradient Boosting [12]. In linear probing, early work reveals that deep linguistic structures are capturable by training simple classifiers on fixed frozen representations [13, 14]. Recent work show that LLM embeddings preserve Lipschitz continuity and can be utilized in high-dimensional regression settings, outperforming conventional feature-engineering baselines [15].

#### Scaling laws for Predicting Downstream Performance in LLMs

The downstream performance of LLMs has garnered significant attention, with special focus on scaling laws. Gadre et al. [16] have found a power law between validation loss and model FLOPs, and also a power law between language model perplexity and average performance across all datasets. Isik et al. [17] have found a log law in translation tasks between downstream translation performance and the number of tokens in the pretrained task. Chen et al. [18] have proposed a two-stage approach to predict downstream performance: first mapping computational resources to pretraining loss based on a power law, then mapping pretraining loss to downstream task performance based on a linear mapping. Although these formulas achieve reasonable forecasts, they still rely on finetuning smaller models for calibration and offer limited mechanistic insight into why certain tasks benefit more from scale.

#### Theoretical Foundations of Utilizing Pretrained LLM Hidden Representations

Recent studies in understanding pretraining objectives have revealed precise conditions under which different self-supervised losses guarantee—or fail to guarantee—strong downstream performance. Balestriero and LeCun [19] rigorously demonstrates that reconstruction-based training, such as autoencoders or masked reconstruction, can produce features that capture all input variance yet remain uninformative for discriminative perception, underscoring that low reconstruction error alone is insufficient for transfer. Wu et al. [20] identify two necessary conditions for autoregressive next-token models to transfer effectively: the softmax head must break shift invariance, and downstream tasks must not hinge on tokens given vanishingly small pretraining probabilities. Liu et al. [11] show that among models with identical pretraining loss, those converging to flatter minima generalize best—revealing that the implicit bias of optimization plays a crucial role in shaping downstream performance.

## 3 Next Token Perception Score (NTPS): an analytical assessment metric of pretrained LLMs

In this section, we first present empirical evidence in [Section 3.1](https://arxiv.org/html/2505.17169v1#S3.SS1) showing that, while pretrained LLM representations can boost performance on some perception tasks, they may underperform or offer no benefit on others when compared to models trained from scratch. We then develop a linear-regime theory in [Section 3.2](https://arxiv.org/html/2505.17169v1#S3.SS2) that characterizes the optimal feature maps for next-token prediction versus downstream regression, and formally relates their alignment to excess task loss. Finally, in [Section 3.3](https://arxiv.org/html/2505.17169v1#S3.SS3) we illustrate how autoregressive (V) and perception-trained (U) subspaces diverge based on our theory.

### 3.1 On the Need to Monitor LLM Alignment for Perception Task

To demonstrate that LLM representations are not universally effective, we compare the linear probing performance of pretrained models on downstream perception tasks with the performance of the same architectures trained from scratch on the downstream perception datasets.

Here, we evaluate six models: Qwen2-0.5B/1.5B [21] and OpenELM-270M/450M/1.1B/3B [22]. The evaluation spans 12 downstream datasets across a variety of domains, including: Intent Classification [23], Clickbait Title Classification [24], SST-2 [25], Bias Identification [26], Banking[27], Emotion[1], SMS Spam [28], Medical Question Pairs [29], Rotten Tomatoes [30], CommonsenseQA [31], Climate Sentiment [32], and IMDB [33].

For full training, we use the following configuration across all models and datasets: Adafactor optimizer with a learning rate of $10^{-4}$, $\epsilon$ values of $10^{-30}$ and $10^{-3}$, gradient clipping threshold of 1.0, decay rate of 0.8, and weight decay of $10^{-5}$; 10, 000 training steps with a cosine learning rate scheduler with a 5% warm-up phase. For linear probing: we use the following configuration across all models and datasets: AdamW optimizer with a learning rate of $10^{-4}$; 50 epochs. For both cases, we extract the mean token representation from the final transformer block and feed it into a linear classification head.

As shown in [table 1](https://arxiv.org/html/2505.17169v1#S3.T1)(^1^11The full training record for OpenELM 3B is unavailable due to insufficient GPU memory (A100), even when using a batch size of 1.), the effect of autoregressive pretraining with linear probing varies markedly across datasets. On sentiment-analysis tasks—SST-2 [25], Rotten Tomatoes [30], Climate [32] and IMDB [33]—linear probing delivers gains of roughly 5–10%. For intent classification [23], clickbait detection [24], bias identification [26], SMS spam [28] and CommonsenseQA [31], the performance difference between linear probing and training from scratch stays within about 1%. In the most extreme cases—emotion recognition [1] and medical-text classification [29]—linear probing actually underperforms training from scratch by a substantial margin.

**Table 1: Comparison of linear probing performance of pretrained models versus full-training from scratch across downstream datasets. Linear probing can outperform, match, or underperform full-training from scratch, indicating that pretrained LLM representations are not universally effective.**
|  | Qwen2<br>0.5B | Qwen2<br>1.5B | OpenELM<br>270M | OpenELM<br>450M | OpenELM<br>1.1B | OpenELM<br>3B |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Linear | Full | Linear | Full | Linear | Full | Linear | Full | Linear | Full | Linear | Full |
| Intent | 99.7 | 99.6 | 99.9 | 99.5 | 99.3 | 99.6 | 99.5 | 99.6 | 99.8 | 99.8 | 98.4 | 99.0 |
| Clickbait Title | 99.4 | 99.1 | 99.6 | 99.0 | 99.4 | 98.4 | 99.6 | 98.4 | 99.7 | 98.7 | 99.6 | 98.6 |
| SST-2 | 85.4 | 80.4 | 88.2 | 82.1 | 87.6 | 80.3 | 87.7 | 82.5 | 89.3 | 92.0 | 89.9 | 78.7 |
| Banking | 88.1 | 85.4 | 89.4 | 82.4 | 89.8 | 86.3 | 90.5 | 84.8 | 91.3 | 83.3 | 82.0 | 82.3 |
| Bias | 95.5 | 94.9 | 96.4 | 94.6 | 96.5 | 94.7 | 96.4 | 94.4 | 96.8 | 95.5 | 95.4 | 94.8 |
| Emotion | 66.2 | 88.3 | 69.0 | 88.3 | 70.9 | 86.8 | 72.0 | 86.7 | 73.6 | 76.9 | 63.6 | 87.9 |
| SMS Spam | 99.3 | 98.7 | 99.3 | 98.9 | 99.0 | 98.8 | 99.2 | 98.7 | 99.2 | 98.9 | 98.4 | 99.0 |
| Medical | 36.4 | 51.5 | 28.9 | 51.3 | 33.8 | 51.5 | 30.6 | 51.5 | 27.5 | 51.5 | 36.7 | 51.5 |
| Rotten Tomatoes | 81.9 | 75.9 | 85.6 | 73.5 | 82.6 | 74.1 | 84.1 | 76.9 | 86.8 | 75.2 | 84.8 | 74.9 |
| Commonsense | 22.1 | 21.0 | 24.2 | 22.4 | 22.3 | 21.2 | 21.2 | 22.2 | 23.3 | 21.3 | 21.5 | 21.3 |
| Climate | 78.4 | 63.7 | 81.3 | 67.8 | 80.3 | 69.1 | 79.1 | 71.9 | 81.6 | 69.1 | 79.4 | 71.2 |
| IMDB | 92.5 | 86.0 | 94.4 | 84.9 | 92.8 | 84.2 | 99.5 | 84.1 | 94.5 | 83.5 | 94.4 | – |

This variability suggests that the representations learned via autoregressive pretraining do not uniformly align with downstream perception tasks. Therefore, we are going to quantify this alignment (or lack thereof). As a starting point, we first build intuition and establish theoretical results in the *linear regime*. As we will show in [section 4](https://arxiv.org/html/2505.17169v1#S4), this seemingly simplified setting provides surprisingly informative insights into more complex empirical scenarios.

### 3.2 Theoretical foundations: linear approximations

Consider a sentence, whose representation is $X\in\mathbb{R}^{(d,\ell)}$, where $d$ is the hidden dimension size of each token and $\ell$ is the total number of tokens in this sentence. Consider two variants $X_{1},X_{2}\in\mathbb{R}^{d\times(\ell-1)}$ from $X$, where $i$-th column of $X_{1}$ is the representation of the first $i$ tokens of the sentence and $i$-th column of $X_{2}$ is the representation of the $i+1$-th token of the sentence.

Autoregressive training aims to find a model’s parameter $\theta$ to predict $X_{2}$ based on $X_{1}$. Specifically, the training objective is to minimize the following Cross-Entropy(CE) loss:

$$ $\mathcal{L}_{\mathrm{CCE}}=-\mathbb{E}_{X}\bigl{[}\log p_{\theta}(X_{2}\mid X_ {1})\bigr{]},\qquad p_{\theta}(X_{2}\mid X_{1})\propto g_{\theta}(f_{\theta}(X _{1})).$ (1) $$

In the linear setting, $f_{\theta}$ is a linear map $V\in\mathbb{R}^{d\times k}$ and $g_{\theta}$ is another linear map $W\in\mathbb{R}^{k\times d}$. Besides, we assume the $i$-th column of $X_{1}$ represents the sum of the first $i$ tokens in $X$ and the $i$-th column of $X_{2}$ represents the $i+1$-th token in $X$.

Then the loss function $\mathcal{L}$ is defined as:

$$ $\mathcal{L}=\mathbb{E}_{X_{1},X_{2}}\bigl{\lVert}W^{\top}V^{\top}X_{1}-X_{2} \bigr{\rVert}_{F}^{2}=\mathbb{E}_{X}\bigl{\lVert}W^{\top}V^{\top}X\,L_{1}-X\,L _{2}\bigr{\rVert}_{F}^{2}.$ (2) $$

Here $L_{1},L_{2}\in\mathbb{R}^{\ell\times(\ell-1)}$ is for selecting the tokens in $X$, see [Section A.2](https://arxiv.org/html/2505.17169v1#A1.SS2) for full definition.

Similarly, given another pair ($U\in\mathbb{R}^{d\times k}$, $Z\in\mathbb{R}^{k\times c}$). When we use the sum of all tokens in the sentence to predict the label $Y\in\mathbb{R}^{c}$, our loss $\mathcal{L}^{*}$ is defined below:

$$ $\mathcal{L^{*}}=\mathbb{E}_{X,Y}\left\|Z^{\top}U^{\top}X\,1_{\mathrm{\ell \times 1}}-Y\right\|_{F}^{2},$ (3) $$

Note that in both cases, instead of the CE loss, we assume a convex mean square error (MSE) loss. We now state a key guarantee for our choice: as the MSE loss vanishes, so does the probability of decoding error.

###### Lemma 1 (Equivalence between MSE and CE) .

Let $X\in\mathbb{R}^{d\times\ell}$ be the token representations
Denote

$$ $h^{*}=X\,L_{2},\qquad\hat{h}=W^{\top}V^{\top}X\,L_{1}.$ $$

Assume the vocabulary embeddings $\{w_{i}\}_{i=1}^{V}\subset\mathbb{R}^{d}$ satisfy a positive margin

$$ $\Delta\;=\;\min_{j\neq y}\bigl{\langle}w_{y},h^{*}\bigr{\rangle}-\bigl{\langle }w_{j},h^{*}\bigr{\rangle}\;>\;0,\quad M\;=\;\max_{i\neq j}\|w_{i}-w_{j}\|_{2}.$ (4) $$

If $\mathcal{L}\;=\;\mathbb{E}\|\hat{h}-h^{*}\|_{F}^{2}\;\to\;0,$
then

$$ $\Pr\bigl{(}\arg\max_{i}\langle w_{i},\hat{h}\rangle=y\bigr{)}\;\longrightarrow \;1.$ (5) $$

The proof of [Lemma 1](https://arxiv.org/html/2505.17169v1#Thmlemma1) is deferred to [Section A.1](https://arxiv.org/html/2505.17169v1#A1.Ex5) (for empirical validation, see [34]).

Now we can solve the [eq. 2](https://arxiv.org/html/2505.17169v1#S3.E2) and [eq. 3](https://arxiv.org/html/2505.17169v1#S3.E3) under the following theorem.

###### Theorem 1 .

The loss functions $\mathcal{L}$ in [eq. 2](https://arxiv.org/html/2505.17169v1#S3.E2) and $\mathcal{L}^{*}$ in [eq. 3](https://arxiv.org/html/2505.17169v1#S3.E3) are minimized for

$$ $W=(V^{\top}\mathbb{E}[X\,L_{1}L_{1}^{\top}\,X]V)^{-1}V^{\top}\mathbb{E}[X\,L_{ 1}L_{2}^{\top}\,X]$ (6) $$

$$ $Z=(U^{\top}\mathbb{E}[X\,1_{\mathrm{\ell\times 1}}1_{\mathrm{\ell\times 1}}^{ \top}\,X]U)^{-1}U^{\top}\mathbb{E}[X\,1_{\mathrm{\ell\times 1}}Y^{\top}]$ (7) $$

$U,V$ span the top $k$ eigenvectors of the following generalized eigenvalue problems:

$$ $\displaystyle\mathbb{E}[X\,L_{1}L_{2}^{\top}X^{\top}]\,\mathbb{E}[X\,L_{2}L_{1 }^{\top}X^{\top}]\,\tilde{V}$ $\displaystyle=\mathbb{E}[X\,L_{1}L_{1}^{\top}X^{\top}]\,\tilde{V}\,\Lambda_{V},$ (8) $\displaystyle\mathbb{E}[X\,1_{\ell\times 1}\,Y^{\top}]\,\mathbb{E}[Y\,1_{1 \times\ell}\,X^{\top}]\,\tilde{U}$ $\displaystyle=\mathbb{E}[X\,1_{\ell\times\ell}\,X^{\top}]\,\tilde{U}\,\Lambda_ {U}.$ (9) $$

The proof of [Theorem 1](https://arxiv.org/html/2505.17169v1#Thmtheorem1) is deferred to [Section A.2](https://arxiv.org/html/2505.17169v1#A1.SS2).

From Theorem [1](https://arxiv.org/html/2505.17169v1#Thmtheorem1), it shows that $U$ and $V$ capture distinct co-variability structures; hence, the autoregressively derived $V$ may not generalize well to downstream tasks that depend on $U$.

###### Lemma 1 (Equivalence between MSE and CE) .

###### Theorem 1 .

### 3.3 Divergence of autoregressive ( V 𝑉 V italic_V ) vs. perceptual ( U 𝑈 U italic_U ) features

Figure: Algorithm 1 Computation of NTPS

To illustrate that $U$ and $V$ capture fundamentally different axes of covariation, we extract the first embedding layer activations from a pretrained models (OpenELM-450M) on the Emotion[1] dataset. We then solve the two generalized eigenvalue problems in Theorem [1](https://arxiv.org/html/2505.17169v1#Thmtheorem1) to obtain the projection matrices $U$ and $V$, each truncated to its top two eigenvectors.
Figure [1](https://arxiv.org/html/2505.17169v1#S1.F1) visualizes two-dimensional projections of representative words under the perception ($U^{\top}X$) and autoregressive ($V^{\top}X$) mappings. In the left panel, points colored by emotion form two nearly linearly separable clusters—positive versus negative—reflecting the label-conditioned objective. The right panel shows that the same words in $V$-space overlap heavily across emotion classes, indicating that next-token prediction does not prioritize emotional polarity. Instead, the $V$-space shows a clear grouping of adjectives vs. function words, suggesting that autoregressive training emphasizes syntactic category. In short, $U$ specializes in downstream, label-relevant semantics such as sentiment, whereas $V$ encodes the structural, syntactic information essential for language modeling.

To obtain a continuous measure of overlap between the feature subspaces learned by $U$ (perception) and $V$ (autoregressive), we introduce the following alignment score. Let
$P\;=\;V\,V^{\dagger}$
be the orthogonal projector onto the column space of $V$, where $V^{\dagger}$ is the Moore–Penrose pseudoinverse of $V$. We then define

$$ $\mathrm{NTPS}(U,V)\;=\;\frac{\|PU\|_{F}^{2}}{\|U\|_{F}^{2}}.$ $$

Here, $\|P\,U\|_{F}^{2}$ is the total squared projection of $U$ onto $V$’s subspace, and $\|U\|_{F}^{2}$ is the total variance in $U$. By construction,
$0\,\leq\,\mathrm{NTPS}(U,V)\,\leq\,1,$ achieving $1$ if and only if the column spaces of $U$ and $V$ coincide, and approaching 0 0 as they become orthogonal. Higher values thus indicate greater alignment between the two objectives. The pseudocode for computing NTPS is provided in [Algorithm 1](https://arxiv.org/html/2505.17169v1#alg1).

###### Theorem 2 (Excess regression loss vs. subspace alignment) .

Let $U$ be the optimal perceptual encoder obtained from the generalized eigenproblem.
For any other encoder $V$, define the orthogonal projector. Let $\Delta\mathcal{L}:=\mathcal{L}^{\!*}(V)-\mathcal{L}^{\!*}(U)$
be the excess regression loss of $V$.

Then there exist task-dependent positive constants
$C_{\min},C_{\max}$ such that

$$ $\boxed{\;C_{\min}\bigl{(}1-\mathrm{NTPS}(U,V)\bigr{)}\;\leq\;\Delta\mathcal{L} \;\leq\;C_{\max}\bigl{(}1-\mathrm{NTPS}(U,V)\bigr{)}.\;}$ (10) $$

The proof of [Theorem 2](https://arxiv.org/html/2505.17169v1#Thmtheorem2) is deferred to [Section A.3](https://arxiv.org/html/2505.17169v1#A1.SS3).

This shows that the extra regression loss of the autoregressive trained encoder $V$ over the perception trained $U$ is tightly controlled by their subspace alignment: as $NTPS(U,V)$ approaches 1 (perfect alignment), the excess loss vanishes, and as it decreases, the loss grows linearly within the constant bounds.

With these theoretical guarantees in hand, we now turn to empirical validation, demonstrating how our NTPS alignment score—derived in this simplified linear regime—predicts downstream performance across a range of nonlinear, pretrained models.

###### Theorem 2 (Excess regression loss vs. subspace alignment) .

## 4 Empirical validation and practical utility of NTPS

In this section, we show that our NTPS—though derived under a linear model—captures meaningful alignment in nonlinear large-scale LLMs. First, in [Section 4.1](https://arxiv.org/html/2505.17169v1#S4.SS1) demonstrate a monotonic relationship between NTPS and downstream performance by showing Spearman correlations with both MSE loss and classification accuracy across eight pretrained models and 12 downstream perception datasets. Next in [Section 4.2](https://arxiv.org/html/2505.17169v1#S4.SS2), we examine how parameter-efficient LoRA finetuning shifts feature subspaces to boost NTPS, offering an interpretable lens on why LoRA works. Finally and most importantly, in [Section 4.3](https://arxiv.org/html/2505.17169v1#S4.SS3), we demonstrate that NTPS itself can predict the magnitude of accuracy gains from LoRA, making it a practical pre-screening metric for when fine-tuning will be most effective.

### 4.1 Correlation between NTPS and linear probing performance

First, we demonstrate that this score correlates with the downstream performance of eight pretrained models across 12 diverse datasets.

Figure: Figure 2: Correlation between NTPS and downstream MSE loss (rows 1 and 2), and between NTPS and accuracy (rows 3 and 4), with dashed lines indicating linear regression fits. Higher alignment scores correspond to lower MSE loss and higher accuracy in downstream tasks.
Refer to caption: x2.png

Similar to [section 3.1](https://arxiv.org/html/2505.17169v1#S3.SS1), we now include two more pretrained models (Qwen2-7B [21] and LLaMA3-8B [8]), with the other models keep the same. Besides, we reuse the 12 datasets as described in [section 3.1](https://arxiv.org/html/2505.17169v1#S3.SS1).

Downstream performance is measured in two complementary ways. First, we train a linear layer on each downstream dataset using ordinary least-sqaure (OLS) regression with close-form solution since our theoretical derivation in [theorem 1](https://arxiv.org/html/2505.17169v1#Thmtheorem1) is based on MSE loss (as shown in [eq. 2](https://arxiv.org/html/2505.17169v1#S3.E2) and [eq. 3](https://arxiv.org/html/2505.17169v1#S3.E3)). And, we use the final MSE loss as the downstream performance metric. Second, we train a linear layer on each downstream dataset using logistic regression with saga optimizer under a CE loss since it better reflects practical usage. And, we use the final accuracy as the downstream performance metric.

For each model across all datasets, we compute alignment scores over all layers (from the word-embedding layer through the penultimate layer) and every $k$ proportion value from 0.05 to 0.95 in increments of 0.05. We then assess the monotonic relationship between alignment and downstream performance using Spearman’s $r\in[-1,1]$. where $r=1$ denotes perfectly concordant orderings.

To summarize each model succinctly, we report for each model the alignment score corresponding to the configuration that yields the strongest Spearman’s $r$.
Besides, all results are obtained on training set to minimize confounding factors such as distribution shifts between train and test splits, which may obscure the true relationship between alignment score and downstream performance.

The results, shown in [fig. 2](https://arxiv.org/html/2505.17169v1#S4.F2), reveal a clear trend: higher alignment scores are associated with lower MSE loss and higher accuracy in downstream tasks. This strong correspondence indicates that our alignment metric—despite its derivation under simplified linear assumptions—serves as an effective proxy for task alignment even in highly nonlinear models. It provides insight into when autoregressive training is beneficial for downstream tasks and can serve as a practical metric for anticipating the effectiveness of linear probing on pretrained models.

### 4.2 LoRA finetuning enhances NTPS

As a step further, we provide an interpretation of why LoRA is effective for adapting pretrained LLMs to downstream tasks, through the lens of NTPS.

**Table 2: Relative improvement (%) of NTPS after LoRA finetuning across models and downstream datasets. NTPS is universally increased after LoRA finetuning, suggesting that LoRA finetuning enhances the overlap between feature subspaces of autoregressive training and downstream perception tasks. For small models, NTPS slightly decreases probably because these models sacrifice the next-token prediction for higher downstream performance due to their limited capability.**
| Dataset | Qwen2<br>0.5B | Qwen2<br>1.5B | Qwen2<br>7B | OpenELM<br>270M | OpenELM<br>450M | OpenELM<br>1.1B | OpenELM<br>3B | LlaMA-3<br>8B |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Intent | 1.9 | 1.6 | 53.2 | -1.1 | -1.0 | 0.9 | 88.5 | 77.8 |
| Clickbait Title | 1.7 | 1.9 | 70.1 | -0.6 | -0.3 | -0.5 | 74.8 | 74.3 |
| SST-2 | 1.4 | 4.3 | 105.9 | -0.8 | -0.7 | 2.1 | 100.0 | 102.5 |
| Banking | 2.3 | 0.9 | 67.8 | -0.8 | -0.6 | 0.7 | 79.5 | 87.1 |
| Bias | 3.1 | 3.6 | 78.3 | -1.4 | -1.4 | -0.4 | 73.3 | 81.3 |
| Emotion | 2.3 | 2.9 | 109.0 | -1.5 | -2.3 | 0.5 | 83.4 | 80.0 |
| SMS Spam | 0.7 | -0.1 | 124.1 | -1.4 | -0.7 | 0.1 | 78.2 | 79.0 |
| Medical | 2.2 | 2.3 | 120.5 | -0.8 | -0.2 | 0.7 | 228.7 | 92.9 |
| Rotten Tomatoes | 0.8 | 0.2 | 105.6 | -1.6 | -0.4 | 0.1 | 88.6 | 84.3 |
| Commonsense | 1.1 | 0.7 | 108.8 | 1.1 | 0.6 | 1.0 | 91.6 | 95.0 |
| Climate | 1.5 | 1.8 | -14.5 | -0.4 | -1.8 | 1.2 | 76.7 | 99.9 |
| IMDB | 0.4 | -0.5 | 135.1 | 0.9 | 1.5 | 1.0 | 109.1 | 89.4 |

Specifically, we compute NTPS for the same eight models and 12 datasets used in [section 4](https://arxiv.org/html/2505.17169v1#S4), using the exact same configuration for NTPS computation, but after applying LoRA. We adopt a consistent finetuning setup across all experiments: LoRA is applied to all QKV projection layers with rank 32, $\alpha=32$, no bias, and a dropout rate of 0.05. For each input, we extract the mean token representation from the final transformer block and pass it into a linear classification head. We use the Adafactor optimizer with a learning rate of $10^{-4}$, $\epsilon$ values of $10^{-30}$ and $10^{-3}$, gradient clipping threshold of 1.0, decay rate of 0.8, and weight decay of $10^{-5}$. Training is conducted for 5000 steps with a cosine learning rate scheduler and a 5% warm-up phase.

As shown in [table 2](https://arxiv.org/html/2505.17169v1#S4.T2), NTPS increases in 71 out of 96 runs after applying LoRA. This provides empirical support for our interpretation: LoRA may improve downstream task performance by adjusting the representations to better align the feature subspaces used for autoregressive pretraining and those required for downstream tasks, especially in large models. Interestingly, we do see that NTPS slightly decreases in small models like OpenELM-270M and OpenELM-450M, this is probably because these model sacrifice the next-token prediction capability in exchange for higher downstream performance due to its limited capability.

In summary, our empirical validation in [section 4.1](https://arxiv.org/html/2505.17169v1#S4.SS1) and [section 4.2](https://arxiv.org/html/2505.17169v1#S4.SS2) shows that NTPS not only correlates strongly with linear-probe performance across diverse pretrained models and downstream tasks, but also reliably increases after LoRA finetuning—confirming its utility as a practical measure of feature-subspace alignment. Building on these findings, we now turn to leveraging NTPS to predict the additional accuracy gains afforded by task-specific LoRA adaptations.

### 4.3 Predicting LoRA finetuning gain with NTPS

Now, we are going to show that our NTPS can also serve for practical usage, particularly for predicting the LoRA finetuning gain.

Figure: Figure 3: Correlation between NTPS and accuracy gain (LoRA finetuning accuracy-linear probing accuracy), with dashed lines indicating linear regression fits. Higher alignment scores correspond to lower accuracy gain after LoRA finetuning in downstream tasks.
Refer to caption: x3.png

To evaluate whether NTPS can forecast the benefit of parameter-efficient fine-tuning, we measure the “ACC gain” as the difference between accuracy after LoRA adaptation and the baseline linear-probe accuracy (both on the test split). We reuse the same eight pretrained models (Qwen2-0.5B/1.5B/7B, OpenELM-270M/450M/1.1B/3B, and LLaMA3-8B) and the 12 downstream classification tasks described in Section [4](https://arxiv.org/html/2505.17169v1#S4). Again we (1) compute NTPS over all layers and $k$s exactly as before, (2) train a linear probe under CE to get baseline accuracy (AdamW optimizer, learning rate of $10^{-4}$; 50 epochs), and (3) apply LoRA (rank 32, $\alpha=32$, 5000 steps, Adafactor, 5% warm-up) and record the adapted accuracy. Finally, we correlate NTPS with the observed LoRA gains using Spearman’s $r$.

As plotted in Figure [3](https://arxiv.org/html/2505.17169v1#S4.F3), there is a clear monotonic relationship: models with lower NTPS enjoy larger accuracy gains from LoRA, whereas higher NTPS exhibit only modest improvements. Across the eight models, Spearman’s $r$ ranges from 0.40 up to 0.90 (higher absolute $r$ indicates stronger predictivity), confirming that NTPS is a reliable indicator of how much headroom remains for downstream adaptation.

In practical terms, if a pretrained model yields a low NTPS on the target task, one can anticipate a sizable boost from LoRA; conversely, if a model yields a high NTPS, it is unlikely to benefit substantially from further finetuning. This makes NTPS a lightweight pre-screening tool to decide when parameter-efficient fine-tuning is most worthwhile.

## 5 Conclusion

In conclusion, in this paper we have introduced the *NTPS*, a simple yet powerful metric for measuring the alignment between the feature subspaces learned by autoregressive pretraining and those required for downstream perception tasks. In a linear setting, we proved that NTPS both upper- and lower-bounds the excess regression loss of an autoregressive encoder relative to an ideal perceptual encoder. Empirically, we demonstrate that NTPS—computed in closed form from pretrained representations and labeled data—correlates strongly classification accuracy across 12 diverse NLP datasets and eight pretrained models ranging from 270 M to 8 B parameters. By measuring NTPS before and after LoRA finetuning, we show that LoRA systematically increases subspace overlap and thus improves downstream performance as well as NTPS. Finally, we illustrate one important practical usage of our NTPS, i.e., predicting the accuracy gain after LoRA finetuning.

Our work still has several limitations that can be addressed in future research. First, NTPS is derived under a simplified linear setting and does not currently account for the role of attention mechanisms. Incorporating attention—possibly starting with linear attention models [35]—could lead to a more precise formulation. Second, we have not yet explored how to select optimal configurations for computing NTPS in each model beforehand. For example, the choice of $k$ may depend on the model’s compression rate. Developing a more principled configuration strategy could improve efficiency and eliminate the need to exhaustively search over all possible settings.

## 6 Broader impacts

This paper presents work whose goal is to advance the field of
deep learning. There are many potential societal consequences
of our work, none of which we feel must be specifically highlighted here.

## References

- Saravia et al. [2018]
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen.
CARER: Contextualized affect representations for emotion recognition.
In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 3687–3697, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
doi: 10.18653/v1/D18-1404.
URL [https://www.aclweb.org/anthology/D18-1404](https://www.aclweb.org/anthology/D18-1404).
- Radford et al. [2018]
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.
Improving language understanding by generative pre-training.
2018.
- Radford et al. [2019]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.
Language models are unsupervised multitask learners.
*OpenAI blog*, 1(8):9, 2019.
- Mann et al. [2020]
Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al.
Language models are few-shot learners.
*arXiv preprint arXiv:2005.14165*, 1:3, 2020.
- Achiam et al. [2023]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Touvron et al. [2023a]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*, 2023a.
- Touvron et al. [2023b]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
*arXiv preprint arXiv:2307.09288*, 2023b.
- Grattafiori et al. [2024]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Singh [2025]
Ajit Singh.
Meta llama 4: The future of multimodal ai.
*Available at SSRN 5208228*, 2025.
- Kumar et al. [2022]
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang.
Fine-tuning can distort pretrained features and underperform out-of-distribution.
*arXiv preprint arXiv:2202.10054*, 2022.
- Liu et al. [2023]
Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma.
Same pre-training loss, better downstream: Implicit bias matters for language models.
In *International Conference on Machine Learning*, pages 22188–22214. PMLR, 2023.
- Vacareanu et al. [2024]
Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, and Mihai Surdeanu.
From words to numbers: Your large language model is secretly a capable regressor when given in-context examples.
*arXiv preprint arXiv:2404.07544*, 2024.
- Tenney et al. [2019]
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al.
What do you learn from context? probing for sentence structure in contextualized word representations.
*arXiv preprint arXiv:1905.06316*, 2019.
- Jawahar et al. [2019]
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah.
What does bert learn about the structure of language?
In *ACL 2019-57th Annual Meeting of the Association for Computational Linguistics*, 2019.
- Tang et al. [2024]
Ethan Tang, Boyu Yang, and Xinyi Song.
Understanding llm embeddings for regression.
In *ICLR Workshop on Foundation Models*, 2024.
- Gadre et al. [2024]
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al.
Language models scale reliably with over-training and on downstream tasks.
*arXiv preprint arXiv:2403.08540*, 2024.
- Isik et al. [2025]
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo.
Scaling laws for downstream task performance in machine translation.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Chen et al. [2024]
Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji.
Scaling laws for predicting downstream performance in llms.
*arXiv preprint arXiv:2410.08527*, 2024.
- Balestriero and LeCun [2024]
Randall Balestriero and Yann LeCun.
Learning by reconstruction produces uninformative features for perception.
*arXiv preprint arXiv:2402.11337*, 2024.
- Wu et al. [2023]
Chenwei Wu, Holden Lee, and Rong Ge.
Connecting pre-trained language model and downstream task via properties of representation.
*Advances in Neural Information Processing Systems*, 36:47216–47238, 2023.
- Yang et al. [2024]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan.
Qwen2 technical report, 2024.
URL [https://arxiv.org/abs/2407.10671](https://arxiv.org/abs/2407.10671).
- Mehta et al. [2024]
Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari.
OpenELM: An Efficient Language Model Family with Open Training and Inference Framework.
*arXiv.org*, April 2024.
URL [https://arxiv.org/abs/2404.14619v1](https://arxiv.org/abs/2404.14619v1).
- [23]
Bhuvaneshwari.
Intent classification.
[https://huggingface.co/datasets/Bhuvaneshwari/intent_classification](https://huggingface.co/datasets/Bhuvaneshwari/intent_classification).
Accessed: 2025-04-29.
- Chakraborty et al. [2016]
Abhijnan Chakraborty, Bhargavi Paranjape, Sourya Kakarla, and Niloy Ganguly.
Stop clickbait: Detecting and preventing clickbaits in online news media.
In *Advances in Social Networks Analysis and Mining (ASONAM), 2016 IEEE/ACM International Conference on*, pages 9–16. IEEE, 2016.
- Socher et al. [2013]
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts.
Recursive deep models for semantic compositionality over a sentiment treebank.
In *Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing*, pages 1631–1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
URL [https://www.aclweb.org/anthology/D13-1170](https://www.aclweb.org/anthology/D13-1170).
- [26]
Priya Patel.
Bias identification.
[https://huggingface.co/datasets/PriyaPatel/Bias_identification](https://huggingface.co/datasets/PriyaPatel/Bias_identification).
Accessed: 2025-04-29.
- Casanueva et al. [2020]
Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic.
Efficient intent detection with dual sentence encoders.
In *Proceedings of the 2nd Workshop on NLP for ConvAI - ACL 2020*, mar 2020.
URL [https://arxiv.org/abs/2003.04807](https://arxiv.org/abs/2003.04807).
Data available at https://github.com/PolyAI-LDN/task-specific-datasets.
- Almeida et al. [2011]
Tiago A. Almeida, Jose Maria Gomez Hidalgo, and Akebo Yamakami.
Contributions to the study of sms spam filtering: New collection and results.
In *Proceedings of the 2011 ACM Symposium on Document Engineering (DOCENG’11)*, 2011.
- McCreery et al. [2020]
Clara H. McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain.
Effective transfer learning for identifying similar questions: Matching user questions to covid-19 faqs, 2020.
- Pang and Lee [2005]
Bo Pang and Lillian Lee.
Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.
In *Proceedings of the ACL*, 2005.
- Talmor et al. [2019]
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.
CommonsenseQA: A question answering challenge targeting commonsense knowledge.
In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
doi: 10.18653/v1/N19-1421.
URL [https://aclanthology.org/N19-1421](https://aclanthology.org/N19-1421).
- Bingler et al. [2023]
Julia Bingler, Mathias Kraus, Markus Leippold, and Nicolas Webersinke.
How cheap talk in climate disclosures relates to climate initiatives, corporate emissions, and reputation risk.
Working paper, Available at SSRN 3998435, 2023.
- Maas et al. [2011]
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.
Learning word vectors for sentiment analysis.
In *Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies*, pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
URL [http://www.aclweb.org/anthology/P11-1015](http://www.aclweb.org/anthology/P11-1015).
- Hui and Belkin [2020]
Like Hui and Mikhail Belkin.
Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks.
*arXiv preprint arXiv:2006.07322*, 2020.
- Katharopoulos et al. [2020]
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.
Transformers are rnns: Fast autoregressive transformers with linear attention.
In *International conference on machine learning*, pages 5156–5165. PMLR, 2020.
- Ghojogh et al. [2019]
Benyamin Ghojogh, Fakhri Karray, and Mark Crowley.
Eigenvalue and generalized eigenvalue problems: Tutorial.
*arXiv preprint arXiv:1903.11240*, 2019.

## Appendix A Appendix

### A.1 Proof of Lemma 1

See [1](https://arxiv.org/html/2505.17169v1#Thmlemma1)

###### Proof.

For any $j\neq y$, write

$$ $\langle w_{y},\hat{h}\rangle-\langle w_{j},\hat{h}\rangle=\bigl{\langle}w_{y}, h^{*}\bigr{\rangle}-\bigl{\langle}w_{j},h^{*}\bigr{\rangle}+\bigl{\langle}w_{y }-w_{j},\,\hat{h}-h^{*}\bigr{\rangle}.$ $$

By definition the first term is at least $\Delta$, and by Cauchy–Schwarz

$$ $\bigl{|}\langle w_{y}-w_{j},\,\hat{h}-h^{*}\rangle\bigr{|}\;\leq\;\|w_{y}-w_{j }\|_{2}\,\|\hat{h}-h^{*}\|_{2}\;\leq\;M\,\|\hat{h}-h^{*}\|_{2}.$ $$

Hence

$$ $\langle w_{y},\hat{h}\rangle-\langle w_{j},\hat{h}\rangle\;\geq\;\Delta-M\,\| \hat{h}-h^{*}\|_{2}.$ $$

In particular, whenever
$\|\hat{h}-h^{*}\|_{2}<\Delta/M$ we have $\langle w_{y},\hat{h}\rangle>\langle w_{j},\hat{h}\rangle$
for all $j\neq y$, so the arg-max picks the correct token $y$. Since
$\mathbb{E}\|\hat{h}-h^{*}\|_{2}^{2}\to 0$ implies $\|\hat{h}-h^{*}\|_{2}\to 0$ in probability,
the probability of decoding error goes to zero.
∎

###### Proof.

### A.2 Proof of Theorem 1

See [1](https://arxiv.org/html/2505.17169v1#Thmtheorem1)

###### Proof.

Consider a sentence, whose token representation is $X\in\mathbb{R}^{(d,\ell)}$, where $d$ is the hidden dimension size of each token and $\ell$ is the total number of tokens in this sentence. Consider two variants $X_{1},X_{2}\in\mathbb{R}^{d\times(\ell-1)}$ from $X$. For $X_{1}$, the $i$-th column represents the sum of the first $i$ tokens in $X$. For $X_{2}$, the $i$-th column represents the $(i+1)$-th token in $X$ so the whole matrix denotes the sequence of tokens from position 2 through $\ell$ in $X$.

Given a linear model to predict the $n$-th token given the sum of the previous $n-1$ tokens that contains a linear mapping $V\in\mathbb{R}^{d\times k}$ and a linear mapping $W\in\mathbb{R}^{k\times d}$. And set the loss $\mathcal{L}$ as:

$$ $\mathcal{L}=\mathbb{E}_{X_{1},X_{2}}[\|W^{\top}V^{\top}X_{1}-X_{2}\|_{F}^{2}],$ (11) $$

Denote $A=V^{\top}X_{1}\in\mathbb{R}^{k\times(\ell-1)}$, then:

$$ $\displaystyle\mathcal{L}$ $\displaystyle=\mathbb{E}_{X_{1},X_{2}}[\|W^{\top}A-X_{2}\|_{F}^{2}]$ (12) $\displaystyle=\mathbb{E}_{X_{1},X_{2}}[Tr((W^{\top}A-X_{2})^{\top}(W^{\top}A-X _{2}))]$ $\displaystyle=\mathbb{E}_{X_{1},X_{2}}[Tr(A^{\top}WW^{\top}A)-2Tr(X_{2}^{\top} W^{\top}A)+Tr(X_{2}^{\top}X_{2})]$ $$

Taking the derivative of $\mathcal{L}$ w.r.t. $W$.

$$ $\displaystyle\frac{\partial\mathcal{L}}{\partial W}$ $\displaystyle=\frac{\partial\mathbb{E}_{X_{1},X_{2}}[Tr(A^{\top}WW^{\top}A)-2 Tr(X_{2}^{\top}W^{\top}A)+Tr(X_{2}^{\top}X_{2})]}{\partial W}$ (13) $\displaystyle=\frac{\partial\mathbb{E}_{X_{1},X_{2}}[Tr(A^{\top}WW^{\top}A)]}{ \partial W}-2\frac{\partial\mathbb{E}_{X_{1},X_{2}}[Tr(X_{2}^{\top}W^{\top}A)] }{\partial W}$ $\displaystyle=\frac{\partial\mathbb{E}_{X_{1},X_{2}}[Tr(W^{\top}AA^{\top}W)]}{ \partial W}-2\frac{\partial\mathbb{E}_{X_{1},X_{2}}[Tr(W^{\top}AX_{2}^{\top})] }{\partial W}$ $\displaystyle=2E_{X_{1},X_{2}}[AA^{\top}W-AX_{2}^{\top}]$ $\displaystyle=2E_{X_{1},X_{2}}[V^{\top}X_{1}X_{1}^{\top}VW-V^{\top}X_{1}X_{2}^ {\top}]$ $\displaystyle=2E_{X_{1},X_{2}}[V^{\top}X_{1}(X_{1}^{\top}VW-X_{2}^{\top})]$ $$

To minimize $\mathcal{L}$, we set $\lx@cref{creftype~refnum}{eq:l-wrt-w}=0$:

$$ $\displaystyle 2E_{X_{1},X_{2}}[V^{\top}X_{1}(X_{1}^{\top}VW-X_{2}^{\top})]$ $\displaystyle=0$ (14) $\displaystyle E_{X_{1},X_{2}}[V^{\top}X_{1}X_{1}^{\top}VW]$ $\displaystyle=E_{X_{1},X_{2}}[V^{\top}X_{1}X_{2}^{\top}]$ $$

Assume that $V^{T}X_{1}$ has rank $k$, then $V^{T}X_{1}X_{1}^{T}V$ is invertible, and we can express $W$ from [eq. 14](https://arxiv.org/html/2505.17169v1#A1.E14) as below:

$$ $\displaystyle W$ $\displaystyle=E_{X_{1},X_{2}}[(V^{\top}X_{1}X_{1}^{\top}V)^{-1}]E_{X_{1},X_{2} }[V^{\top}X_{1}X_{2}^{\top}]$ (15) $\displaystyle W$ $\displaystyle=(V^{\top}\mathbb{E}[X_{1}X_{1}^{\top}]V)^{-1}V^{\top}\mathbb{E}[ X_{1}X_{2}^{\top}]$ $$

By plugging in $W$ back to $\mathcal{L}$, we have:

$$ $\displaystyle\mathcal{L}$ $\displaystyle=\mathbb{E}[Tr(X_{1}^{T}V(V^{\top}\mathbb{E}[X_{1}X_{1}^{\top}]V) ^{-1}V^{\top}\mathbb{E}[X_{1}X_{2}^{\top}]\mathbb{E}[X_{2}X_{1}^{T}]V(V^{T} \mathbb{E}[X_{1}X_{1}^{T}]V)^{-1}V^{T}X_{1})]$ (16) $\displaystyle\quad-2\mathbb{E}[Tr(X_{2}^{\top}\mathbb{E}[X_{2}X_{1}^{\top}]V(V ^{\top}\mathbb{E}[X_{1}X_{1}^{\top}]V)^{-1}V^{\top}X_{1})]+\mathbb{E}[Tr(X_{2} ^{\top}X_{2})]$ $\displaystyle=Tr(\mathbb{E}[X_{1}^{T}V(V^{\top}\mathbb{E}[X_{1}X_{1}^{\top}]V) ^{-1}V^{\top}\mathbb{E}[X_{1}X_{2}^{\top}]\mathbb{E}[X_{2}X_{1}^{T}]V(V^{T} \mathbb{E}[X_{1}X_{1}^{T}]V)^{-1}V^{T}X_{1}])$ $\displaystyle\quad-2Tr(\mathbb{E}[X_{2}^{\top}\mathbb{E}[X_{2}X_{1}^{\top}]V(V ^{\top}\mathbb{E}[X_{1}X_{1}^{\top}]V)^{-1}V^{\top}X_{1}])+Tr(\mathbb{E}[X_{2} ^{\top}X_{2}])$ $\displaystyle=Tr(\mathbb{E}[V^{T}X_{1}X_{1}^{T}V(V^{\top}\mathbb{E}[X_{1}X_{1} ^{\top}]V)^{-1}V^{\top}\mathbb{E}[X_{1}X_{2}^{\top}]\mathbb{E}[X_{2}X_{1}^{T}] V(V^{T}\mathbb{E}[X_{1}X_{1}^{T}]V)^{-1}])$ $\displaystyle\quad-2Tr(\mathbb{E}[\mathbb{E}[X_{2}X_{1}^{\top}]V(V^{\top} \mathbb{E}[X_{1}X_{1}^{\top}]V)^{-1}V^{\top}X_{1}X_{2}^{\top}])+Tr(\mathbb{E}[ X_{2}^{\top}X_{2}])$ $\displaystyle=Tr((V^{\top}\mathbb{E}[X_{1}X_{1}^{\top}]V)(V^{\top}\mathbb{E}[X _{1}X_{1}^{\top}]V)^{-1}V^{\top}\mathbb{E}[X_{1}X_{2}^{\top}]\mathbb{E}[X_{2}X _{1}^{\top}]V(V^{\top}\mathbb{E}[X_{1}X_{1}^{\top}]V)^{-1})$ $\displaystyle\quad-2Tr(\mathbb{E}[X_{2}X_{1}^{\top}]V(V^{\top}\mathbb{E}[X_{1} X_{1}^{\top}]V)^{-1}V^{\top}\mathbb{E}[X_{1}X_{2}^{\top}])+Tr(\mathbb{E}[X_{2} ^{\top}X_{2}])$ $\displaystyle=Tr(\mathbb{E}[X_{2}X_{1}^{\top}]V(V^{\top}\mathbb{E}[X_{1}X_{1}^ {\top}]V)^{-1}V^{\top}\mathbb{E}[X_{1}X_{2}^{\top}])$ $\displaystyle\quad-2Tr(\mathbb{E}[X_{2}X_{1}^{\top}]V(V^{\top}\mathbb{E}[X_{1} X_{1}^{\top}]V)^{-1}V^{\top}\mathbb{E}[X_{1}X_{2}^{\top}])+Tr(\mathbb{E}[X_{2} ^{\top}X_{2}])$ $\displaystyle=Tr(\mathbb{E}[X_{2}^{\top}X_{2}])-Tr(\mathbb{E}[X_{2}X_{1}^{\top }]V(V^{\top}\mathbb{E}[X_{1}X_{1}^{\top}]V)^{-1}V^{\top}\mathbb{E}[X_{1}X_{2}^ {\top}])$ $$

Denote $R_{11}=\mathbb{E}[X_{1}X_{1}^{\top}],R_{12}=\mathbb{E}[X_{1}X_{2}^{\top}],R_{2
2}=\mathbb{E}[X_{2}X_{2}^{\top}]$, minimizing $\mathcal{L}$ is solving the following maximization problem:

$$ $\max_{V}Tr(V^{\top}R_{12}R_{12}^{\top}V(V^{\top}R_{11}V)^{-1})$ (17) $$

which is equivalent to the following maximization problem:

$$ $\max_{\tilde{V}:\tilde{V}^{\top}R_{11}\tilde{V}=I}Tr(\tilde{V}^{\top}R_{12}R_{ 12}^{\top}\tilde{V})$ (18) $$

And we can observe that the constraint is satisfied when:

$$ $\tilde{V}=V(V^{\top}R_{11}V)^{-\frac{1}{2}}$ (19) $$

Thus, $\tilde{V}$ and $V$ share the same column space. And the subspace can be found via the optimization problem in [eq. 18](https://arxiv.org/html/2505.17169v1#A1.E18), which yields to the generalized eigenvalue problem [36]:

$$ $R_{12}R_{12}^{\top}\tilde{V}=R_{11}\tilde{V}\Lambda$ (20) $$

Since [eq. 18](https://arxiv.org/html/2505.17169v1#A1.E18) is a maximization problem, $\tilde{V}$ contains the eigenvectors of $(R_{12}R_{12}^{\top},R_{11})$ that correspond to the top $k$ largest eigenvalues. And so $V$ spans the same column space as these eigenvectors.

From our definition we have $X_{1}=XL_{1},X_{2}=XL_{2}$ with $L_{1},L_{2}\in\mathbb{R}^{(\ell,\ell-1)}$ defined as:

$$ $L_{1}=\begin{bmatrix}Q_{\ell-1}\\ 0\end{bmatrix},$ (21) $$

$$ $L_{2}=\begin{bmatrix}0\\ I_{\ell-1}\end{bmatrix},$ (22) $$

where $Q_{l-1}\in\mathbb{R}^{l-1\times l-1}$ is a unit upper triangular matrix (i.e. all entries on or above the diagonal are 1 and 0 below the diagonal).

Denote

$$ $D\in\mathbb{R}^{\ell\times\ell}=L_{1}L_{1}^{\top}=\begin{bmatrix}Q_{\ell-1}Q_{ \ell-1}^{\top}&0\\ 0&0\end{bmatrix}$ (23) $$

$$ $S\in\mathbb{R}^{\ell\times\ell}=L_{1}L_{2}^{\top}=\begin{bmatrix}0&Q_{\ell-1} \\ 0&0\end{bmatrix}$ (24) $$

Then we can rewrite the generalized eigenvalue problem in [eq. 20](https://arxiv.org/html/2505.17169v1#A1.E20) as:

$$ $\mathbb{E}[XSX^{\top}]\mathbb{E}[XS^{\top}X^{\top}]\tilde{V}=\mathbb{E}[XDX^{ \top}]\tilde{V}\Lambda$ (25) $$

Now let’s consider a regression task with the label of the sentence $X$ denoted as $Y\in\mathbb{R}^{c}$.

Given a linear model to predict the label based on the sum of all tokens in the sentence that contains a linear mapping $U\in\mathbb{R}^{d\times k}$ and $Z\in\mathbb{R}^{k\times c}$. And set the learning objective as mean squared error (MSE) loss $\mathcal{L}^{*}$ as defined below:

$$ $L^{*}=\mathbb{E}\left\|Z^{\top}U^{\top}X\,1_{\mathrm{\ell\times 1}}-Y\right\|_ {F}^{2},$ (26) $$

where $1_{\mathrm{\ell\times 1}}\in\mathbb{R}^{\ell\times 1}$ is for summing the tokens in $X$.

Similarly, with the optimal $Z$, we will have the optimal $U$ sharing the same column space as $\tilde{U}$ that contains the eigenvectors corresponding to the largest k eigenvalues of the following generalized eigenvalue problem:

$$ $\mathbb{E}[X1_{\mathrm{\ell\times 1}}Y^{\top}]\mathbb{E}[Y(X1_{\mathrm{\ell \times 1}})^{\top}]\tilde{U}=\mathbb{E}[(X1_{\mathrm{\ell\times 1}})(X1_{ \mathrm{\ell\times 1}})^{\top}]\tilde{U}\Lambda,$ (27) $$

which simplifies to:

$$ $\mathbb{E}[X1_{\mathrm{\ell\times 1}}Y^{\top}]\mathbb{E}[Y1_{\mathrm{1\times \ell}}X^{\top}]\tilde{U}=\mathbb{E}[X1_{\mathrm{\ell\times\ell}}X^{\top}] \tilde{U}\Lambda,$ (28) $$

∎

###### Proof.

### A.3 Proof of Theorem 2

See [2](https://arxiv.org/html/2505.17169v1#Thmtheorem2)

###### Proof.

Denote
$N:=\mathbb{E}[X1_{\mathrm{\ell\times\ell}}X^{\top}],\;M:=\mathbb{E}[X1_{
\mathrm{\ell\times 1}}Y^{\top}].$

$$ $\mathcal{L}^{\!*}(V)=\operatorname{Tr}(\mathbb{E}[YY^{\top}])\;-\operatorname{ Tr}\!\bigl{(}V^{\top}MM^{\top}V\,(V^{\top}NV)^{-1}\bigr{)}.$ (29) $$

$$ $\mathcal{L}^{\!*}(U)=\operatorname{Tr}(\mathbb{E}[YY^{\top}])\;-\operatorname{ Tr}\!\bigl{(}U^{\top}MM^{\top}U\,(U^{\top}NU)^{-1}\bigr{)}.$ (30) $$

Recall that the columns of $U$ solve the generalized eigenvalue problem, and $U$ is $N$–orthonormal ($U^{\top}NU=I_{k}$)

$$ $MM^{\top}U\;=\;NU\Lambda,\qquad\Lambda=\operatorname{diag}(\Lambda_{11},\dots, \Lambda_{kk}),\;\Lambda_{11}\geq\dots\geq\Lambda_{kk}>0.$ (31) $$

the minimal regression loss is

$$ $\mathcal{L}^{\!*}(U)=\operatorname{Tr}(\mathbb{E}[YY^{\top}])\;-\sum_{i=1}^{k} \Lambda_{ii}.$ (32) $$

Premultiplying [eq. 31](https://arxiv.org/html/2505.17169v1#A1.E31) by $N^{-1/2}$ and defining the whitened basis $U^{*}:=N^{1/2}U$ gives the *ordinary* symmetric eigenproblem. ($U$ and $U^{*}$ share the same subspace)

$$ $N^{-1/2}MM^{\top}N^{-1/2}\,U^{*}\;=\;U^{*}\,\Lambda.$ (33) $$

Introduce the whitened encoder $V^{*}:=N^{1/2}V\,(V^{\top}NV)^{-1/2}$

$$ $\displaystyle T(V):$ $\displaystyle=\operatorname{Tr}\!\bigl{(}V^{\top}MM^{\top}V\,(V^{\top}NV)^{-1} \bigr{)}$ (34) $\displaystyle=\operatorname{Tr}\!\bigl{(}V^{*\top}\,N^{-1/2}MM^{\top}N^{-1/2} \,V^{*}\bigr{)}$ $\displaystyle=\operatorname{Tr}\bigl{(}V^{*\top}\,U^{*}\,\Lambda\,U^{*\top}\,V ^{*}\bigr{)}$ $\displaystyle=\sum_{i=1}^{k}\Lambda_{ii}\,\|V^{*\top}u^{*}_{i}\|_{2}^{2},$ $\displaystyle=\sum_{i=1}^{k}\Lambda_{ii}\,u^{*\top}_{i}V^{*}V^{*\top}u^{*}_{i},$ $\displaystyle=\sum_{i=1}^{k}\Lambda_{ii}\,u^{\top}_{i}N^{1/2\top}V^{*}V^{*\top }N^{1/2}u_{i},$ $\displaystyle=\sum_{i=1}^{k}\Lambda_{ii}\;u_{i}^{\top}\,N\,V\,(V^{\top}N\,V)^{ -1}\,V^{\top}\,N\;u_{i}$ $$

$$ $\displaystyle\Delta\mathcal{L}$ $\displaystyle:=\sum_{i=1}^{k}\Lambda_{ii}\,\bigl{(}1-\lVert V^{*\top}u_{i}^{*} \rVert_{2}^{2}\bigr{)}$ (35) $\displaystyle=\sum_{i=1}^{k}\Lambda_{ii}\,(1-u_{i}^{*\top}\,V^{*}V^{*\top}\,u_ {i}^{*})$ $\displaystyle=\sum_{i=1}^{k}\Lambda_{ii}\,(1-u_{i}^{\top}\,N\,V\,(V^{\top}N\,V )^{-1}\,V^{\top}\,N\;u_{i}).$ $$

Note that

$$ NTPS $\displaystyle=\frac{\|PU\|_{F}^{2}}{\|U\|_{F}^{2}}$ (36) $\displaystyle=\frac{\operatorname{Tr}\!\bigl{(}U^{\top}P^{\top}PU\bigr{)}}{\|U \|_{F}^{2}}$ $\displaystyle=\frac{\operatorname{Tr}\!\bigl{(}U^{\top}PU\bigr{)}}{\|U\|_{F}^{ 2}}$ $\displaystyle=\frac{\sum_{i=1}^{k}u_{i}^{\top}Pu_{i}}{\|U\|_{F}^{2}}$ $\displaystyle=\frac{\sum_{i=1}^{k}u_{i}^{\top}V\,(V^{\top}V)^{-1}V^{\top}\,u_{ i}}{\|U\|_{F}^{2}}.$ $$

For each $i$, set

$$ $r_{i}=1-u_{i}^{\top}NV(V^{\top}NV)^{-1}V^{\top}Nu_{i}.$ $$

Writing $w=Va$ and minimizing

$$ $(u_{i}-Va)^{\top}N(u_{i}-Va)=u_{i}^{\top}Nu_{i}-2a^{\top}V^{\top}Nu_{i}+a^{ \top}V^{\top}NVa$ $$

over $a$ yields $a^{*}=(V^{\top}NV)^{-1}V^{\top}Nu_{i}$, so

$$ $r_{i}=u_{i}^{\top}Nu_{i}-u_{i}^{\top}NV(V^{\top}NV)^{-1}V^{\top}Nu_{i}=\min_{w \in\mathrm{col}(V)}(u_{i}-w)^{\top}N(u_{i}-w).$ $$

Since $x^{\top}Nx\geq\lambda_{\min}(N)\|x\|^{2}$,

$$ $\displaystyle r_{i}$ $\displaystyle\geq\lambda_{\min}(N)\min_{w\in\mathrm{col}(V)}\|u_{i}-w\|^{2},$ (37) $\displaystyle=\lambda_{\min}(N)[\|u_{i}\|^{2}-u_{i}^{\top}V(V^{\top}V)^{-1}V^{ \top}u_{i}].$ $$

Thus

$$ $\displaystyle\Delta\mathcal{L}$ $\displaystyle\geq\lambda_{\min}(N)\sum_{i}\Lambda_{ii}[\|u_{i}\|^{2}-u_{i}^{ \top}V(V^{\top}V)^{-1}V^{\top}u_{i}],$ (38) $\displaystyle\geq\lambda_{\min}(N)\Lambda_{\min}\|U\|_{F}^{2}(1-\mathrm{NTPS}),$ $$

Similarly, $x^{\top}Nx\leq\lambda_{\max}(N)\|x\|^{2}$ gives

$$ $r_{i}\leq\lambda_{\max}(N)[\|u_{i}\|^{2}-u_{i}^{\top}V(V^{\top}V)^{-1}V^{\top} u_{i}],$ (39) $$

and hence

$$ $\Delta\mathcal{L}\leq\lambda_{\max}(N)\Lambda_{\max}\|U\|_{F}^{2}(1-\mathrm{ NTPS}).$ (40) $$

Combining,

$$ $\boxed{\,\lambda_{\min}(N)\Lambda_{\min}\|U\|_{F}^{2}\,(1-\mathrm{NTPS})\leq \Delta\mathcal{L}\leq\lambda_{\max}(N)\Lambda_{\max}\|U\|_{F}^{2}\,(1-\mathrm{ NTPS})}$ (41) $$

∎

###### Proof.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - Are the results in Figure 2 and Figure 3 reported over all the downstream tasks in Table 1? What does each point represent?
soundness: 3
strengths: - The proposed metric and its derivation seem pretty clear. It is essentially a subspace alignment score between the frobenius norm of the perception encoder U that lies inside the next token subspace spanned by V.
- The metric seems to be well-correlated with downstream performance across different models.
summary: The authors propose a new measure NTPS as a predictive measure for downstream performance on different tasks. It quantifies the overlap between the feature subspace favored by next-token prediction and the subspace optimal for the downstream task. They show that this correlates with the actual downstream performances much better than the linear probe, and is a computationally efficient way of gauging the "fit" of the LLM for the given downstream task.
weaknesses: > Takeaway: Linear probing on pretrained LLM representations can outperform, match, or underperform full-training from scratch.
- Agreed that the linear probing technique is indeed noisy, but Table 1 and this claim seem to be slightly misleading. These linear probes are used as a way to approximate how good the model are at the downstream tasks like Emotion, etc, so a better study seems to be how well the linear probes correlate to the full finetuning performance (when taking different checkpoints for example), not if they are better, worse, or equal.
- For Figures 2 and 3, can we also plot how the linear probe performance correlates with the MSE performance?

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Conceptual justification for "perception": In the introduction of NTPS in Section 3.2, what is the theoretical or empirical motivation for terming U the "perception" subspace? The formulation in Eq. 9 appears to be a standard supervised dimensionality reduction via generalized eigen decomposition. What distinguishes this as specifically capturing "perception" as opposed to other notions?

2. Alternative explanations for misalignment: The paper attributes misalignment primarily to the next-token prediction objective during pretraining. However, could other factors explain the variability in Table 1, such as: training data distribution or optimization dynamics? How do the authors rule out these alternative explanations?

3. Acquisition of U in experiments: The main body of the text lacks clear specification of how U is acquired in each experiment. Specifically which layer's representations are used to compute U? And How is the subspace dimension k chosen (fixed value, cross-validation, proportion of variance)?

4. Motivation for LoRA finetuning analysis (Section 4.2): Why focus specifically on LoRA for demonstrating NTPS's practical utility?
soundness: 2
strengths: 1. The work proposes NTPS as a novel metric for measuring the misalignment between perception and next-token prediction objectives, addressing an important gap in understanding of pretrained LLMs’ limited transferability to downstream tasks.

2. The paper includes comprehensive and extensive experimental results : (1) Table 1 demonstrates that linear probing can outperform, match, or underperform full training from scratch, establishing the motivation for the work (2) Figure 2 shows consistent correlations between NTPS values and both MSE loss and accuracy across 8 models (3) Figure 3 demonstrates NTPS's utility in predicting LoRA finetuning gains (4) Experiments span diverse model families (Qwen2, LLaMA-3, OpenELM) and model scales (270M-8B parameters)
summary: This paper investigates the misalignment between autoregressive pretraining and downstream perception tasks in large language models. The authors demonstrate that linear probe performance varies substantially across tasks, suggesting that features optimized for next-token prediction often do not transfer well to perception objectives. To address this, they introduce the Next Token Perception Score (NTPS), a metric measures the overlap between autoregressive feature subspace (V) and perception feature subspace (U). Theoretically, they prove that NTPS both upper- and lower-bounds the excess regression loss (Theorem 2), establishing it as a valid proxy for downstream performance. Empirically, the authors validate NTPS across 12 diverse NLP datasets and 8 pretrained models (270M to 8B parameters), demonstrating strong correlations with linear probe accuracy and MSE loss. Finally, they show that NTPS reliably predicts accuracy gains from LoRA finetuning.
weaknesses: 1. The paper claims that misalignment between perception and autoregressive spaces arises primarily from the next-token prediction loss during pretraining (lines 54-59, Section 3.1). However, other confounding factors could contribute to this phenomenon, including (1) pretraining data size and distribution mismatches with downstream tasks (2) optimization dynamics and implicit biases.

2. Related to the first point, the paper does not adequately control for or discuss these alternative explanations, weakening the causal claims about the source of misalignment.

3. The paper has many missing experimental details and lack of methodological clarity. In particular, the paper lacks how U is computed in practice: While Algorithm 1 in the appendix describes NTPS computation, the main text (Sections 3.2, 4.1) does not clearly specify which layer representations are used to compute U, how the subspace dimension k is selected, or how these choices affect results. 

4. The narrative and flow are weak and often incoherent. Section 3.2 suggests some empirical evidence for the strength of NTPS, but the justification is not theoretically grounded or conceptually insightful. The experiment demonstrating NTPS as a proxy for LoRA fine-tuning (Section 4.2) is also not well motivated. The paper would benefit from clearer signposting and more explicit connections between theoretical claims and empirical validation.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: My main question is on how to use NTPS in practice -- given an individual downstream task how would I interpret the NTPS score?
Also, it would be helpful to include some discussion or analysis of which k values and which layers tend to perform best across different models and tasks. Are there consistent trends -- for example, are middle layers generally more aligned, or does the optimal k scale with model size?
The correlations are computed after sweeping over both k and layer, and then selecting the configuration that maximizes correlation on the training set. Do results hold if you test the resulting correlation on a held-out validation set?
soundness: 3
strengths: The paper is nicely organized and clearly motivated. The proposed NTPS metric provides an intuitive geometric perspective on alignment and the theoretical section is clear and builds good intuition. The experiments cover a wide range of models and datasets and NTPS shows convincing correlation with MSE loss and additional accuracy gains from LoRA.
summary: The paper introduces the Next Token Perception Score (NTPS) that measures the overlap between autoregressive and perception feature subspaces, motivated by the idea that features from the autoregressive subspace may not be useful for perception tasks. The authors show that NTPS correlates strongly with a linear probe and predicts whether a LoRA finetuning can provide additional accuracy gains.
weaknesses: All reported results are based on rank correlations, which makes it hard to interpret what the metric actually means in practice. If I have a model and a downstream task and compute an NTPS score, how should I interpret its magnitude? The paper doesn’t provide guidance on what constitutes a "high" or "low" score, which limits its usefulness.
The claim on syntactic vs semantic groupings (as in Fig 1) is nice for intuition but does not seem rigorous based on comparing just the top 2 eigenvalues.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: - Q1. It seems that the term "perception task" is a term introduced in this paper, and not very common in NLP. This creates a severe understanding issue.
soundness: 2
strengths: - A new metric, NTPS, to evaluate the misalignment between the pre-trained LM's feature and the features that are required for the downstream task.
- NTSP has theoretical guarantees (upper and lower bounds) on the excess loss for the MSE.
- Empirical results show that NTPS correlates well with the linear probe performance
summary: The paper introduces the Next-Token Perception Score (NTPS) to quantify how well features learned by next-token pretraining align with supervised “perception” tasks. NTPS measures the overlap between representations optimized for next-token prediction and those needed for labels, is computable in closed form from pretrained representations plus labeled data, and is proven to bound the excess loss (in a linear setting).  Empirically, the paper shows that linear-probe accuracy on pretrained models can outperform, match, or underperform full training from scratch across diverse datasets, motivating the need for such an alignment measure. The paper then shows that, across eight pretrained models and twelve datasets, NTPS monotonically predicts linear probing performance and predicts the accuracy gains from LoRA finetuning.
weaknesses: - **Uncommon and narrow fine-tuning methods**: The paper extracts the mean last layer output and uses the mean embedding feed into a linear classifier head. This is significantly different from how we fine-tune LLMs nowadays: using prompting to make the model output natural language. It is unclear how the results may transfer to realistic fine-tuning settings that are widely used. A natural question is to ask whether the more natural fine-tuning setting (or older fine-tuning methods like pattern exploitation training) can make the autoregressive features become more aligned with the perception task's space. 
- **Underspecified theoretical results**: The constants in Theorem 2 are not well discussed. If the constant is very large, the inequality may be less meaningful. While the empirical results in Section 4.1 may try to validate the theoretical results in the previous section, some assumptions (e.g., linear encoder) do not match, so Section 4 is more like empirical results instead of a justification of the theories.
- **Only reports the rank correlation**: The paper only reports the rank correlation between NTPS, accuracy, and MSE. For practical usages, a user may be more interested in how much absolute accuracy gain they can obtain, instead of whether the accuracy gain can be more than another task fine-tuned using the same model.
- **Unclear motivation of using LoRA fine-tuning**: It is unclear why LoRA fine-tuning is introduced in section 4.2. The main theory part of the paper discusses linear mapping, while Section 4.2 moves to a non-linear setting. I assume that the paper is trying to introduce a more natural setting, but the fine-tuning still uses the mean representation (the paper actually did not precisely describe the setting here, so I assume it is the same as the previous). This is still not a natural setting. Another question I have is why we are interested in the performance of LoRA instead of full fine-tuning. I would like to see some justification for why LoRA is selected here.


Decision: Reject