On the Predictive Power of Representation Dispersion in Language Models
Authors: Yanhong Li
Ming Li
Karen Livescu
Jiawei Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Embedding geometry
Unsupervised evaluation
Mechanistic interpretability
Large Language Models
Label-free metrics
Primary area: interpretability and explainable AI

Abstract: We show that a language model’s ability to predict text is tightly linked to the breadth of its embedding space: models that spread their contextual representations more widely tend to achieve lower perplexity. Concretely, we find that representation dispersion—the average pairwise cosine distance among hidden vectors—strongly and negatively correlates with perplexity across diverse model families (LLaMA, Qwen, and others) and domains (Wikipedia, news, scientific abstracts). Beyond illustrating this link, we show how dispersion can be leveraged for a range of practical tasks—without requiring labeled data. 
First, measuring dispersion on unlabeled text allows us to rank examples by difficulty and identify hard slices in new domains, offering a data‐efficient tool for screening and prioritizing models before full evaluation.
Next, we find that identifying layers with higher dispersion pinpoints the best representations for retrieval‐based methods such as $k$NN‐LM, bypassing exhaustive layer‐by‐layer searches. Finally, we integrate a simple “push‐away” objective into training, which increases dispersion in both single‐domain and cross‐domain scenarios and directly improves perplexity in each.

Published as a conference paper at ICLR 2026
ON THE PREDICTIVE POWER OF REPRESENTATION DIS- PERSION IN LANGUAGE MODELS
Yanhong Li'
Ming Li
Karen Livescu
Jiawei Zhou"
1 Allen Institute for Al
University of Maryland
Toyota Technological Institute at Chicago
'Stony Brook University
varhongl@allenat.oog jiawei. zhou cerou..sstoon 1estonybrook.
ABSTRACT
We show that a language model's ability to predict text is tightly linked the breadth of its embedding space: models that spread their contextual represen- tations more widely tend achieve lower perplexity. Concretely, we find that representation dispersion-the average pairwise cosine distance among hidden vectors vectors-Strongly and negatively correlates with perplexity across diverse model families (LLaMA, Qwen, and others) and domains (Wikipedia, news, scientific abstracts). Beyond illustrating this link, we show how dispersion can be leveraged for range of practical focks-withou requiring labeled data. First, measuring dis- persion on unlabeled text allows us to rank examples by difficulty and identify hard slices in new domains, offering a data-efficient tool for screening and prioritizing models before full evaluation. Next, we find that identifying layers with higher dispersion pinpoints the best representations for retrieval-basee methods such as kNN-LM, bypassing exhaustive layer-by-layer searches. Finally, we integrate a simple "push-away" objective into training, which increases dispersion in both single-domain and cross-domain scenarios and directly improves perplexity in each
INTRODUCTION
Large language models can perform remarkably well on tasks ranging from text completion to code generation, Yet their embedding geomelry often exhibits signs of anisotropy or rank collapse, whereby hidden states lie in narrow cone or occupy low-dimensional subspace (Fthayarajh 2019 Gao al. 2019 [i et al. 2020). Although this geometry has been posited to limit expressive power, how preciscly it connects to auto-regressive text generation remains less clear.
In this paper, we present empirical evidence that model's ability to predict text is tightly linked to the breadth of its embedding space. Intuitively, as illustrated in Figure weaker models compress contexts into tight clusters, whereas stronger models separate these contexts even semantically similar ones- more broadly. This broader geometry yields clearer distinctions in the latent space, enabling sharper (lower-entropy) next-token predictions. Concretely, we quantify representation dispersion any chosen layer the average pairwise cosine distance of its hidden vectors. Unless specified otherwise, our empirical sections use the final layer, and we show that higher dispersion consistently predicts lower perplexity (Figure
Beyond revealing this fundamental link, representation dispersion offers practical benefits:
Label-free diagnostics. Dispersion measured on unlabeled text monotonically tracks correct- ness for fixed model and dataset, enabling label-free ranking of examples by difficulty and discovery hard slices ($3.1).
Model selection. Among multiple pretrained or fine-tuned variants within the same model family, larger dispersion provides coarse, inexpensive signal that helps identify clearly under- adapted checkpoints and prioritize promising ones for full evaluation ($3.2).
Layer selection for retrieval augmentation. Although the first two applications exploit the final hidden state, dispersion is equally informative inside the network. In ANN-LM, choosing
CCode available at htccpp/s//gittab...mmm/yaaaaaaap

---

Published as a conference paper at ICLR 2026
Each color (0 .00
indicates cluster of contexts in embedding space.
Representation Dispersion
0.0
WEAK MODEL
BETTER MODEL
0.4
0.2
Perplexity
Figure Representation geometry in weak model vs. better model. In a weak model (top), final-layer embeddings for similar contexts are compressed into tight clusters, limiting discriminative power. In better model (bottom), embeddings are widely dispersed even within semantically related clus- ters Pers-lcading to more confident next-token predictions.
Figure 2: An Illustrative Em- pirical Trend. Across mod- els/datasets, higher dispersion correlates with lower perplexity.
the hidden layer with the highest dispersion yields the best perplexity, providing unsupervised shortcut to sub-layer selection ($3.)).
A training signal. Encouraging higher dispersion through an auxiliary "push-away" loss directly
improves perplexity for both single-domain and cross-domain scenarios ($3.4).
Overall, we show that model's embedding pairwise dixtance-serve serves as robust indicator of predictive quality. By quantifying and encouraging this broader geometry, we gain both conceptual insight and practical benefits, and we hope this perspective fosters new avenues for improving model robustness and interpretability.
by the simple statistic of average
EMPIRICAL ANALYSIS OF REPRESENTATION GEOMETRY
We begin by describing how we measure representation geometry and perplexity ( 2.1 We then present three key global observations that characterize how embedding dispersion evolves with perplexity, across layers, and under fine-tuning ($2.)). Finally, we zoom in on semantic clusters ($2.3) to examine how dispersion behaves among closely related contexts.
2.1 MEASUREMENT SETUP
Following standard autoregressive conventions, let a language model with parameters 0 assign probability to a token sequence (I ...,..., via (1 .2,..., NT = I ( Ien), where Fen = =(xi....,Fx Fn-1). In contemporary Transformer-based models, each partial sequence I Zen is mapped to an internal context vector h, E Rd by a function fo(-). The next-token distribution is then given by Pe Z<n) = softmax( W. h,). where W, e R/Vix xd projects the representation h, into the vocabulary space V. For measuring representation dispersion, we focus on a chosen final-layer vector (e.g., hy for the full sequence), by default, as the representation of each text sample.
Contextual Representations.
Measuring Representation Dispersion. To measure how well the model separates text samples in its embedding space, we first choose particular layer or sub-layer whose representation we wish to examine (e.g., the final hidden state, output after the final attention block, or the second-to-last block). Concretely, we sample N text segments from dataset and pass each segment through the model to extract the corresponding representation E, Rd from the chosen sublayer, for i = I,..,, N. We then compute the average pairwise cosine distance of these representations:
E, E, E
D
Σ ISISIEN
(1)
E
2
2

---

Published as a conference paper at ICLR 2026
WikiText (Liama-3.2-3B)
WikiText (Liama-3.1-88)
News (Llama-3.2-88)
Medical (Llama-3.2-38)
0.80
Phanson 0.0
0.92
Peanco
F--2 0.000
0.80
Phanson
r= 0.000
0.75
Peancn
r 52
Aeerage Cosina Disance
0.78
0.75
0.70
0.76
0.88
0.74
0.36
0.80
0.65
0.84
0.55
0.82
0.50
0.60
0.80
0.45
10 Mean Perplexity
15 Mean Perplexity
Mean Perplexity
Mean Perplexity
Figure 3: Sequence-level perplexity vs. embedding dispersion across different domains and model sizes. Each point represents a bin of text segments with the x-axis showing mean sequence-leve perplexity and the y-axis showing average pairwise cosine distance of final-layer embeddings.
Context Length: tokens
Context Longth: tokens
Context Length: tokans
Context Length: tokens
Fearson
Pearson
Peamson
820
0.000
0.003
80o
Average Coslnee Distance
0.82
0.70
LO050 Mean Perplexity [Last Token)
1.000 1:925 1,075 1.100 Mean Perplexity (ast Taken))
3,625 1.000 Mean Perplexity Ilast Token)
1,003 1.050 1.675 3,300 Heen Perplexity [aatt Taken)
Figure 4: Last-token perplexity vs. embedding dispersion at varying context lengths using LLaMA- 3.2-IB. Each point represents bin of text segments with the x-axis showing mean last-token perplexity and the y-axis showing average pairwise distance.
This quantity reflects how "spread out" the embeddings are, with higher values indicating greater separation among representations.
2.2 GLOBAL OBSERVATIONS ON REPRESENTATION DISPERSION
In this section, we examine how the model's representation space behaves across a broad sample of text segments, highlighting how perplexity, layer depth, and fine-tuning each affect embedding dispersion.
(1) Perplexity vs. Representation Dispersion. Our first finding connects model's sequence-level perplexity to how spread out contextual embeddings are We randomly select 100,000 text segments of 512 tokens, compute their perplexities, and also measure the average pairwise distance of their final-layer embeddings. We sort by perplexity and group segments into bins, and for each bin recording its mean perplexity and mean pairwise distance.
Figure 3 reveals a strong negative correlation: Segments with lower perplexity have more dispersed embeddings, whereas those with higher perplexity show compressed embeddings. This trend appears across multiple model families (Llama, Phi, Mistral, Qwen) and diverse text domains (e.g.
We define the sequence-level perplexity of text segment Ti.t of length L as:
ppl( exp
1 Σ
log log xeee))
1m1
3

---

Published as a conference paper at ICLR 2026
Before Pretraining
After Pretraining
Correlation Coefficient Comparison
1.00
Across Layers
0.90
Correlation Coefficient
0.30
After Pretralning
Before Pretraining
0.2
Average Cosine Distance
3.885
0.10
0.80
+0.32
0.4
0.75
0.6
0.70
-0.75
9 Layer
15
17
0.80
0.60
Network Layers
Layer 1
Layer 9
Layer 17
0.55
Layer 3
Layer 15
Mean Perplexity
Mean Perplexity
Figure 5: Layer-wise embedding separation in LL shown across different perplexity bins (left panel) and the corresponding correlation coefficients (right panel). Note that the negative correlation between perplexity and embedding separation becomes more pronounced as we move to deeper layers (see right panel).
Wikitext- 103, CNN Daily News, PubMed summarization). Full visualizations provided in Section A.1.3 A.1.
We also verify that the relationship holds finer granularity by focusing on last-token perplexity, shown in Figure Across context lengths 16, 64, 128, and 256 tokens, we observe the same negative correlation trend. Intuitively, this negative correlation appears because contexts with more confident next-token predictions (low perplexity) end up pushed into more distinct regions of the embedding space, whereas harder-to-predict contexts remain more compressed.
(2) Layer-Wise Patterns. We next assess how this relationship unfolds across layers. Collecting embeddings from multiple intermediate layers (e.g. Layers 1, 3, 9, 13, 17) and replicating the above procedure, find that the negative correlation strengthens in deeper layers (as illustrated in the right panel of Figure Early layers do not show a clear correlation, likely because they capture lower-level lexical features rather than global predictive cues. Deeper layers exhibit more pronounced embedding distance differences between easier-to fasier-to-predict and harder-to to toaderr-t--pedictt samples. Figure iillustrase this model. Additionally, comparing models
layer-wise progression for representative L.L.M I.LaMA-322-1 before and after pretraining indicates that the negative correlation emerges primarily after the model has been trained predict tokens, pointing to a learned representational structure.
(3) Fine-Tuning Effects. Finally, we examine how fine-tuning reshapes the embedding space. We select the same model ALLaMA-3.2-111 1BB and apply either parameter-efficien (LoRA (Hu et al., 2021) or full-parameter fine-tuning on WikiText- 103. Compared the pre-trained model, both approaches increase the average embedding separation a stronger effect, pushing text samples further apart overall; LoRA, which adapts only low-rank modules, effects smaller but still notable changes. The choice of fine- tuning thus influences how discriminative (i.e. "spread out") the embeddings become a specialized domain.
WikiText- 103 (Figure 66). Full fine-tuning exerts
2.3 DISPERSION WITHIN SEMANTIC CLUSTERS: A FINER-GR GRAINED ANALYSIS
natural question is whether better models merely push semantically dissimilar contexts farther apart, also increase distances among contexts that semantically close. If the latter did not
4

---

Published as a conference paper at ICLR 2026
Within-Cluster Mean
Between-Cluster Mean
Training
Averagle Cosine pistante
0.85
Full FT
0,0500
Value
0.78
2,15
0.80
0.75
0.4400
0.74
0.70
LORA FT
G.po. Checkpoint Stop
0.665
Checkpoint Step
Checkpoint Step
0.60
0.55
Figure 7: Clustering metrics and training loss across training. We form 500 clusters of WikiText- 103 contexts (each cluster con- tains 100 semantically similar contexts that share the same 10-gram continuation). During model training, both the within-cluster dis- tance (left) and berween-cluster distance (middle) consistently grow, while training loss (right) falls.
30 Mean Perplexity
Figure 6: Effect of Oue-Tuning on embedding separation.
happen, then we could imagine a scenario where highly related examples remain tightly clustered, but unrelated examples spread out, thereby inflating overall average distances. We therefore directly measure dispersion among clusters of highly similar contexts to see whether their internal geometry also expands over training.
Setup & Metries. We collect 10-grams from the WikiText-103 training that appear at least 100 times. For each such 10-gram, we retrieve 100 occurrences, each accompanied by 100-token left context, and treat these contexts as a semantically similar cluster. (Manual inspection confirmed that these context sets do indeed relate to the same event, entity, or theme.) We construct a total of 500 such clusters. We then track the contextual embeddings produced by a I.LAMA-32--D BB model at various checkpoints during training. Within each cluster, we compute the within-cluster distance as the average (cosine) distance from each embedding to the centroid of that cluster. Likewise, we define the between-cluster distance the average (cosine) distance among the centroids of all clusters. Thus, the latter measures how far clusters are from each other in the embedding space, while the former measures how tightly each cluster is packed internally.
Results. Figure [ shows that, as training proceeds and the loss decreases, both the average within- cluster distance and between-cluster distance increase. Thus, the trend toward higher overall disper- sion is not driven solely by pushing apart highly dissimilar contexts. Even very similar contexts which all share the same 10-gram continuation become more spread out in the latent space as the model learns, which indicates the model can effectively distinguish them despite the similarities. This in-cluster expansion reinforces our hypothesis that performinf models tend to produce broader embedding geometries in all regions of the latent space, not just pushing away dissimilar examples.
APPLICATIONS
3.1 RANKING EXAMPLE HARDNESS WITHOUT LABELED DATA
In many real-world scenarios, one receives a large unlabeled query set and must decide, before committing annotation compute resources, (1) whether off-the-shelf model will be sufficiently accurate and (2) which specific examples it is likely to get wrong. A reliable label-free indicator would enable rapid model validation, automatic justification of easy versus hard instances, and targeted continued pre-training on precisely those queries that the model currently struggles with. Because higher representation dispersion tracks lower perplexity ($2). we hypothesize that dispersion can serve as such an indicator: if high dispersion coincides with correct predictions, then simply measuring distances allows us to rank examples by expected difficulty and flag likely errors without ever looking at ground-truth labels.
Experimental protocol. To test this idea, we design a controlled experiment that varies the fraction of correct answers while keeping the input distribution fixed: (1) Collect pool of question-anae -nrwer pairs for given dataset- fataset-model model pair. (2) Partition the pool into correct and incorrect subsets by comparing the model's answer with the ground truth. (3) For each desired accuracy level (0%-100%
5

---

Published as a conference paper at ICLR 2026
ARC Challenge
MMLU
0.14
Mean Mean Pairwise Distance
Moan Pairwise Distance
0.14
0.12
0.10
0.10
0.09
0.08
0.08
0.08
0.04
100
Fraction Correct (%)
Fraction Correct
3.2-38
Figure 8: Embedding distance vs. fraction correct. Each point corresponds to a fixed mixture of correctinncoreet model predictions (x-axis). Error bars denote standard error over 10 random seeds. Higher accuracy consistently aligns with larger mean pairwise distance. Similar trends are found in multilingual MMLU, HellaSwag, IFEval, and QuAIL. Details in Section B.I.I and Section B.1.4
in 10% increments), sample 100 queries that contain the requisite mix of correct and incorrect (4) Extract the final-layer embeddings of these queries and compute their mean pairwise cosine distance, averaging over 10 random seeds.
Results. Filote mean pairwise distance against the fraction of correct predictions for several LLaMA variants on ARC-CHALLENGE and MMLU. Across all models and datasets, accuracy rises monotonically with dispersion: slices that the model answers correctly exhibit markedly broader geometry than those it answers incorrectly. Practically, practitioner could therefore dataset by dispersion, inspect only the low-dispersion tail to uncover failure modes, or focus continued training on those "hard" queries. Taken together, these findings establish representation dispersion as a powerful, zero-label indicator of relative hardness and a principled tool for slice discovery and targeted data augmentation, rather than fully calibrated predictor of absolute accuracy.
unlabeled
3.2 REPRESENTATION DISPERSION FOR MODEL SELECTION
In practice, researchers and practitioners often faced with choosing among numerous model variants variants-ranging from different instruction- centtcttio-ttuedd checkpoints to parameter-efficitn or distilled whills-while only having limited labeled data in the target domain. Exhaustively evaluating every checkpoint is typically prohibitively expensive, both in terms of computation and annotation resources. The tight link between representation dispersion and predictive accuracy established in X2 offerrs an attractive alternative: by simply measuring how broadly model separates key tokens in embedding space, one can obtain geometric score that rapidly screens checkpoints within a fixed model family and tokenizer.
adaptations
Setup. To operationalize this idea, we focus on task-relevant tokens that carry significant signal in the domain of interest, such as digits for mathematical reasoning, Python keywords for code generation, legal terms for contract analysis. Let T C V denote small set of such domain-specific tokens, and let 7 denote reference set of common, everyday language tokens. We use the model's original output token embeddings- -tha is, the rows of the output projection matrix-as provided after loading the model weights. This requires no forward passes or input data; all computations performed directly on these pre-trained embeddings. We compute average pairwise distances-eti either cosine or Euclidean-ann among the embeddings of tokens within T. within T. and between T and T.
Motivated by our empirical findings, we propose a single "dispersion gap" metric that succinetly captures both the distinetiveness of the Nomain-relevan tokens and their separation from generic language. Specificallyy, we define
g within(T) between (T,T)

---

Published as a conference paper at ICLR 2026
where within(T) denotes the mean pairwise distance among the domain tokens, and between (T, T) the mean distance between domain and reference tokens. Larger values of & indicate that the model both differentiates between domain-criinn critical tokens and separates them from everyday vocabulary geometric pattern that, as Tables 12 show, strongly correlates with higher task accuracy.
This approach is computationally efficient and entirely label-free: evaluating requires only reading the model's output embedding matrix and performing basic matrix operations on CPU, without any forward passes or GPU computation. In our experiments, ranking models by their dispersion gap within a given family and parameter scale consistently elevates the best-performing models in domains such as math and code, yielding gains of up 40 accuracy points over clearly under-adapted variants. We therefore view the dispersion gap coarse but effective pre-filter: it reliably flags models that have not yet learned the relevant domain geometry, after which more expensive task-specific evaluation can be used to break ties among the remaining strong candidates.
1.5B Models
7B Models
14B Models
2.5
Accuracy
D-NM
90
80
Dittanee
70
1.5
60
50
0.5
40
Qwen2.5 Qwen2.5 MathDitill Qwen
Qwen2.5 Qwen2.5 MatDDicil Owen
Qwen2.5
Distil-Qwen
7B Models
13B Models
C-C
HtumanEivol passa91 (%]
35
NC-NC
Dispersion Sa
30
Mamoon vell mooit
Eartideen
15
Figure 9: Upper: Qwen MATH. Blue bars show D-D (digit-digit) distances, orange bars show D-N M (digit-non-math) distances, and green bars show their sum-the Dispersion Gap. The black dashed line reports task accuracy (%). Lower: Llama/Code bars show C-C (code-keyword) distances, orange bars show NC-NC (non-code) distances, and green bars again give the Dispersion Gap. The black dashed line reports PASS @ (%). Bar heights therefore convey the three dispersion statistics, while the line traces model performance, mirroring the figure legends. full table of the underlying values provided in Section B.2
HUMANEVAL. Blue
Results. FigurePclearty demonstrates how the dispersion gap G aligns with downstream perfor- mance. In the upper panel (Qwen on MATH), accuracy rises monotonically with the green "Gap" bars both within each parameter scale (e.g. 1.5 B - 7 B + 14 B) and across fine-tuned variants (c.g. Spearman correlations between G and accuracy exceed 0.95, and the dispersion gap perfectly ranks all nine checkpoints without single mis-ordering. The lower panel (LLama/CodelLamm on HUMANEVAL) shows the same pattern: CODELLAMA consistently exhibits both a larger gap and a higher PASS rate than its LLAMA2 counterpart at both 7 B and 13 scales, while the increase from B to 13 again boosts both metrics in lock-step. To confirm this pattern is robust beyond static model comparisons, Section B.2.2 extends our analysis to the training trajectory of Olmo-78 7B, showing that dispersion tracks performance improvements across 30 intermediate checkpoints with correlations exceeding 0.90.
QWEN2.5, QWEN2.5-M.rTe
DISTILL-QWEN
3.3 LAYER SELECTION FOR KNN-LM. -M
Many retrieval- retrieval-augmennte augmented language models build datastore by using of the sublayers in the final transformer block as the vector key (Khandelwal et al.) 2020 He et al., [022]] Alon et al., 2022 Li
al. 2024). A central question in these methods is which internal representation of the transformer
7

---

Published as a conference paper at ICLR 2026
N=10
N=50
N=100
Model
Attn
FFN
Attn
FFN
Attn
FFN
DISTILGPT2
0.83+00022
0.33+00.0
0.83:50.01
0.30=0.02
0.82+0.01
0.30+0.001
GPT2-SMALL
0.83+0.02
0.24::.06
0.83++0.00
0.24
0.83+0.00
0.24
GPT2-MEDIUM
0.66+:0.03
0.19:0.02
0.67+0.01
0.21
0.67+0.01
0.21+0001
GPT2-LARGE
0.80.10.04
0.68
0.79:10.01
0.66:10.02
0.8010.011
0.68
Table 1: Representation dispersion (average pairwise cosine distance) of the final block's attention and feed-forward sub-layers for varying numbers of randomly sampled chunks N. Higher values indicate more widely separated contextual embeddings.
to use the "datastore key." In this section, we focus ANN-LM, which augments standard language model with key-value memory of training examples. Specifically, at inference time, retrieves the nearest neighbors of the current hidden state from that memory and interpolates their next-token distribution with the base LM's ss distribution, often achieving lower perplexity on rare or out-of-distribution tokens. Recent evidence (Xu et al. 2023) suggests that taking the attention-layer output (instead the feed-forward layer output) often improves KNN-LM perplexity, but pinpointing the best layer can require expensive end-to-end trials. We show that measuring how widely each layer "disperses" its text embeddings (via average pairwise cosine distance) provides a lightweight, unsupervised way identify a layer likely to yield strong İ:NN-LM performance without running the full interpolation pipeline.
Background. Consider the standard Transformer block used by GPT-2. Let (t-1)) E Raxd denote the hidden states from block - 1. In block I, the hidden states first pass through a multi-headed attention (MHA) sub-layer and residual connection: ( = h(-11)
Dropout MHA MHA (LayerNoooo hh(-))))) We then apply another residual connection around position-
wise feed-forward network (FFN): (i) (0)) + Dropout FFN(LayrNor(((1)) Thus, each
block produces two intermediate sub-layer outputs: h 0 and h h()), Following Xu et al. 2023) we compare using ha versus h from the final block L as the contextual key for KNN-LM.
Experimental Setup. We consider four members of the GPT-2 family-DISTILLGT2 (82 M parameters), GPT2-SMALL (117 M), GPT2-MEDIUM (345 M), and GPT2-LARGE (774 M) trained with the same tokenizer and a context length of 024. For each checkpoint we draw N {10, 50, 100} non-overlapping 512 512-token chunks randomly from the WIKITEXT- 103 validation split. Sampling is repeated 10 times with different random seeds, and the identical chunk indices fed to both sub-layers that any dispersion difference cannot be attributed input variance. For every chunk we extract the last-token hidden state produced by the final Transformer block's ss (i)
attention sub-layer output h and (ii) feed-forward sub-layer output h of N vectors (h, J, distance. We report the mean and standard deviation of dispersion across the 10 random repeats for every (model, N, sub-layer) triple.
(L) Given the resulting set measure their representation dispersion as the average pairwise cosine
Results. Table shows two practitioner-relevant patterns. First, the hidden states taken after the attention sub-layer are always more widely spread than those taken after the feed-forward sub-layer (e.g., 0.8045 vs. 0.6831 for GPT2-LARGE, 0.6593 vs. 0665 for GPT2-MEDIUM), making b (L) the natural choice of key for &NN-LM. Second, dispersion can be estimated with striking efficiency: moving from 10 to 50 or 100 input chunks alters the mean by at most 1.5 % and never changes the layer ranking, so profiling model requires only about 5,000 tokens and a few milliseconds.
3.4 INCORPORATING REPRESENTATION DIVERGENCE INTO TRAINING
While typical cross-entropy training focuses purely next-token prediction, several studies suggest that explicitly encouraging embedding separation can improve generalization and robustness in language modeling (Gunel al. 2021 Jain et al. 2023). Inspired by these findings, augment the
8

---

Published as a conference paper at ICLR 2026
standard language-modeling loss with an auxiliary objective that pushes apart hidden-state vectors, aiming to produce more discriminative representations.
Setup. We consider two scenarios: single-domain setting, where we train GPT-2 small on WikiText, and cross-domain setting, where we train on WikiText plus Python code. In both cases, let { 1==10 be the final-layer hidden-state vectors for token sequences batch (flattened across batch and sequence), where B is the batch size. We normalize each vector to unit length, h, = h: To encourage wider separation, compute an average pairwise cosine distance d and then add an auxiliary loss -d to the standard cross-entropy loss, weighted by hyperparameter X.
In the single-domain setting, dayg is defined over all pairs in the same batch: davg
B[- Σits 1-h: hi In the cross-domain setting, we instead compute only across pairs drawn from different domains (Wiki vs. code) to push embeddings from each domain further apart:
= [사||L E 1 i(AA _ The total loss in both settings Liotal = LCE + À Laux- where LcE is the standard next-token cross-entropy and Laux -davg (single-domain) (cross-domain), and À controls the strength of the auxiliary "spread-out" loss.. We clarify the relation between this auxiliary term and prior contrastive repulsive losses Section B.4.3
-d
Results. Table reports test perplexities for various learning rates and auxiliary loss weights X. In the single-domain (WikiText) setting, introducing the auxiliary spread-out loss yields a slight decrease crplexitt-ttppicllly 1-4 points-reelative the baseline, especially early in training. In the cross-domain (WikiText+Code) setting, the auxiliary loss produces a much more pronounced reduction in perplexity for both domains. For all learning rates, models trained with moderate value of À achieve notably lower perplexity on both WikiText and code, suggesting that explicitly pushing apart representations from distinct domains leads more specialized and di scriminative features. This demonstrates that our approach is particularly effective when bridging heterogeneous data sources.
(a) Single- Domla- (WikiText)
(b) Cross-Domain (WikiText+Codde)
LR
A
Step = 500
Step = 1000
À
Step = 500
550
Step = 1000
Base
+Aux
Base
+Aux
Wiki
Code
Wiki
Code
0.0
226.1
111.3
0.0
295.6
36.9
171.7
23.8
10-3
0.1
217.5
108.2
0.001
270.8
34.5
158.8
22.3
0.0
195.0
96.7
0.0
304.4
35.4
175.7
22.8
7 x 10
0.1
193.8
93.6
0.01
255.2
31.9
150.2
20.8
0.0
166.2
83.0
0.0
268.5
33.7
166.9
22.1
5 x 10-4
165.6
82.0
0.02
253.3
30.5
155.2
20.5
Table 2: Auxiliary spread-out loss improves perplexity in both single- and cross-domain settings. Left: single-domain (WikiText) results; right: cross-domain (WikiText+Code) results. À is the auxiliary loss weight, chosen by validation for each learning rate. We report test-set perplexities at 500 and 1000 steps.
4 RELATED WORK
Geometric Analysis of Embeddings Language Models. growing body of work has examined the geometry of hidden representations in large language models (LLMs)). Early studies identified an anisotropy problem, where embeddings collapse into narrow cone and lose expressiveness at
deeper layers (Mu & Viswanath 2018 Ethayarajh 2019 Gao et al. 2019 Li et al. 2020 Noci et al.) 2022). Recent work uses intrinsic dimension (ID) estimators to trace how representation manifolds evolve across layers (Valeriani et al. 2023), linking geometry to performance through, for example, distinguishing human vs. machine text (Tulchinskii et al., (023), predicting data compressibility
(Cheng al. 2023), and revealing simplex-like structures for categorical concepts (Park et al. 2025

---

Published as a conference paper at ICLR 2026
The study most closely related to ours is Viswanathan al. (022)) which also analyzes token-level embedding distributions and observes cosine similarity rises when tokens in the prompt are shuffled. However, their work remains largely descriptive. In contrast, show how representation dispersion can predict and improve perplexity and downstream accuracy, making geometric insights actionable for model evaluation and selection.
Mechanistic Interpretability. Our work offers geometric perspective on interpretability, comple- menting research into model mechanisms. This field often reverse-engineers models into interpretable circuits (Elhage et al. 2021 [eereska & Gavves 2024), identifying specific components like induction heads for in-context learning (Olsson et al. (022) or analyzing training dynamies through layer-wise gradients (Li al. 2025b a). Rather than dissecting individual components, provide higher- level, component- component-ggnostic metric by quantifying the dispersion representations abstract but behaviorally meaningful property of the model's 's internal geometry.
Our method also differs from post-hoc attribution and probing techniques. Attribution methods like Integrated Gradients assign credit to imputs for given output (Sundararajan al. 2017 Lundberg & Lee 2017). while probing trains auxiliary classifiers on hidden states to identity encoded linguistic features (Belinkov] 2022). These approaches typically require labeled data external probes. In contrast, representational dispersion is label-free, intrinsic measure; we directly relate the geometry of hidden states to the model's own performance metrics, such perplexity and downstream accuracy.
CONCLUSION
In this work, we showed that representation dispersion serves as both a practical diagnostic and training signal for language models. Moving forward, we aim to investigate how representation dispersion interacts with other design choices-sse such as architectural variations tokenization strategies and whether additional regularization signals might further strengthen model robustness and interpretability. We hope these directions will inspire new ways to harness embedding geometry for next-generation language modeling and related tasks.
10

---

Published as a conference paper at ICLR 2026
STATEMENT ON LLM USAGE
We acknowledge the use of Large Language Models (LLMs) to assist in the preparation of this manuscript. Specifically, LLMs were utilized improve grammar and clarity, aid in literature discovery, and generate boilerplate code snippets for our experiments and testing scripts. The authors have carefully reviewed and edited all LLM-generated outputs and take full responsibility for the final content and scientific integrity of this work.
REFERENCES
Uri Alon, Frank Xu, Junxian He, Sudipta Sengupta, Dan Roth, and Graham Neubig. Neuro- symbolic language modeling with automaton-augmentted retrieval. In Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari. Csaba and Niu, Gang and Sabato, Sivan (ed.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 468 485. PMLR, 17-23 Jul 2022. URL https \/proceed wnw.eeeeddii....... nggs..g.... ml press wttps//v122/aalaa
html
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguis- lics, 48(1):20722,, March 2022. doi: (0.1162/coli_a_00/6 htt.ttts:://acllete antho logy
arg/2022.cl-1..
Leonard Bereska and Stratis Gavves. Mechanistic interpretability for Al safety review. Transac- lions Machine Learning Research, 2024. ISSN 2835-8856. |tttttss::/ net/ fest/forum?in www.unniidePPPeeeeeee. Survey Certification, Expert Certification.
Emily Cheng, Corentin Kervadec, and Marco Baroni. Bridging information-thhorett and geometric compression in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12397- 12420, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 1223.emnlp-main.662 URL https: http:://aaaaaahhllaaoooooooooo otg.lllyooooorgooggoo...oooogg.. wwwg//gg/3... emn p ma mamin... mma.i..621 762/
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer
circuits. circuits.pub//00000 hutp///////raaaaoo///tteeeee wttr///wwww///neeeeeeee.o
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and GPT-2 embeddings. In EMNLP-IJCNLP 2019, Hong Kong, China, pp. 25-65, 2019.
Dan Friedman and Adji Bousso Dieng. The vendi score: diversity evaluation metric for machine learning, 2023. Utthtttss:////// http://aaxvvvvo//o org/ abs./210a 02410
Jun Gao, Di Hc, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. Representation degeneration problem in training natural language generation models. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 2019.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. Supervised contrastive learning for pre- trained language model fine-tuning, 2021. URL URL httpt:ttt///t httpa/awww........./oo/aaaooaaaa
014003
Junxian He, Graham Neubig, and Taylor
Efficient nearest neighbor language models. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Pro- cessing, pp. 5703-5714, 144,, Online and Punta Cana, Dominican Republic, November 2021. Asso- ciation for Computational Linguistics. doi: 1/226633/w/222..mmmlppp main.461.... URL https
7/acl www.cannhollco ogy.. org/ 2021 emn
461
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2 10b.22006.0685, 2021. URL
https: /topss//areee wwwp:/iwwwwo//ooooooo//oooooo /006 09685

---

Published as a conference paper at ICLR 2026
Nihal Jain, Dejiao Zhang, Wasi Uddin Ahmad, Zijian Wang, Feng Nan, Xiaopeng Li, Ming Tan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Xiaofei Ma, and Bing Xiang. Contra- CLM: Contrastive learning for causal language model. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6436-6459, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10, 10. URL 3557|
https://aclanttoooooyyyoooo org. 2tt.eaaaw...ooooooorg222222222..
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through Memorization: Nearest Neighbor Language Models. In International Conference Learning Representations (ICLR), 2020.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference EMNLP 2020, Online, November 16-20, 2020, pp.
2020.
Ming Li, Yanhong Li, Ziyue Li, and Tianyi Zhou. How instruction and reasoning data shape post- training: Data quality through the lens layer-wise gradients, 2025a. URL http:: pulttps://amae //arxiv org/aba//e 2504 10766
Ming Li, Yanhong Li, and Tianyi Zhou. What happened in ilms layers when trained for fast vs. slow thinking: A gradient perspective, 2025b. URL http::://///xii.ccoo org/ abaaabss2211 httpp//////.ii..oo////////// 23743
Yanhong Li, Karen Livescu, and Jiawei Zhou. Chunk-distilled language modeling, 2024. URL https: //arxiv wwt./bgg////ooooo..ttt...t..attt abs 2501. 00343
Scott M Lundberg and Su-In Lec. A unified approach to interpreting model predictions. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- neurips ciates, Inc., 2017, httLhttts///roceeeee http:///....rrrrr....pppppappaaa cc/paper_ folffiii files/paper: 2017/file 2017://ieee/22203111///333333/39 /862119996622888 (ow.:3222222624422222222222 http:///...aa22221766666ccccaaa.
Paper.
Jiaqi Mu and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In ICLR 2018, Vancouver, BC, Canada, 2018.
Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198-27211 2022.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodss, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. URL https://aaat /ttp://aaxiv..v... org/
11895
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models, 2025. URL 2406.00 015061
org/abs
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Sergey Nikolenko, Evgeny Burnaev, Serguei Barannikov, and Irina Piontkovskaya. Intrinsic dimension estimation for robust detection of ai-generated texts. Advances in Neural Information Processing Systems, 36: 39257-39276, 2023.
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models. Advances in Neural Information Processing Systems, 36:51234-5122 2023.
12

---

Published as a conference paper at ICLR 2026
Karthik Viswanathan, Yuri Gardinazzi, Giada Panerai, Alberto Cazzaniga, and Matteo Biagetti. The geometry of tokens internal representations of large language models, 2025. URL https !
//openreview net/ ao
Frank F. Xu, Uri Alon, and Graham Neubig. Why do nearest neighbor language models work? In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org, 2023.
13

---

Published as a conference paper at ICLR 2026
Technical Appendices
ORGANIZATION OF CONTENTS
Supplemental Materials for Representation Geometry Analysis
12
A.1
Details regarding Sequence level Perplexity Experiments
12
A.I.I Datasets and Models
12
A.1.2
Procedure for Mean-PPeplyxxty Dispersion Analysis
12
1.3.3
Additional Visualizations
13
A.2
Details regarding Fine- Fine-Tuning Tuning Effect Experiments
18
LoRA Fine-Tuned Model
18
A.2.2
Full-Parameter Tuned Model
18
A.3
Details regarding Dispersion Within Semantic Clusters Training Hyperparameters
19
4
Generalization to Held-Out Data
20
Vendi Score Analysis
Supplemental Materials for Applications of Representation Dispersion
23
B.1
Details Regarding Ranking Example Hardness Without Labeled Data
23
B.I.J
Additional Results
23
B.1.2
Comparison to Perplexity as a Hardness Signal
26
B.1.3
Calibration by Dispersion
28
B.1.4
Testing Distribution Shifi
B.2
Details Regarding Representation Dispersion for Model Selection
31
B.2.1
Full Statistics
B.2.2
Robustness of Dispersion Correlation across Training Trajectories
33
B.3
Details Regarding Layer Selection for KNN-LM
34
B.3.1
Additional Results
34
B.4
Details Regarding Incorporating Representation Dispersion
35
B.4.1
Training Details
35
B.4.2
Impact of Enforcing Clusterization ("Squeeze" Ablation)
................................
36
B.4.3
Relation to Contrastive and Repulsive Objectives
37
Limitations
38

---

Published as a conference paper at ICLR 2026
A SUPPLEMENTAL MATERIALS FOR REPRESENTATION GEOMETRY ANALYSIS
A.1 DETAILS REGARDING SEQUENCE-LEVEL PERPLEXITY EXPERIMENTS
A.1.1 DATASETS AND MODELS
In this section, we provide additional experimental details and visualizations that supplement our main empirical analysis in $ We study range of standard language modeling datasets, including the Salesforce/ikittex segments in diverse domains.
bbisee/cnnddailyman and ccdy/pubneed-uummrzzztionn covering text
For the models, our experiments encompass:
/ LIama families: m aww.mmaaaaaaammmaa Llama 3.2-1B, meta Seta-LlamaLLlame -...maa332222 3.22-B,
meta-11ama/LLamaa3.11 gema
Gemma families: g potp:///oooooooeeeemmaa--2aa
Mistral: hittaaaaalaa/iiisttta----------
Phi: microsoft/phi-2
Qwen families: Owen/Owen2 5-0. 5B, Owen/Qwen2 5-3B, Owen Qwen2. 5-7B
We use the Hugging Face implementation of the above models. All models are standard decoder-only Transformers, for which we collect final-layer embeddings on randomly selected text segments. In line with Equation of the main paper, we measure average pairwise cosine distance to quantify how "spread out" their representations are.
A.1.2 PROCEDURE FOR MEAN-PERPLES PERPLEXITY vs. DISPERSION ANALYSIS
Here, we outline the steps needed to produce mean-perplexity vs. represeetatioon-diserrsioo plot:
Step 1: Randomly sample 100,000 segments (c.g., 512 tokens each) from the data.
Step 2: For each segment:
a) Compute its perplexity over the full sequence.
b) Record the final-layer hidden states for later analysis.
Step 3: Sort all segments by their computed perplexity.
Step 4: Group the sorted segments into bins (e.g., 100 segments per bin) and record each bin's mean perplexity.
Step 5: Perform uniform sampling in perplexity space on these bins to ensure coverage of low-, mid- and nigh-perplexity regions.
Step 6: For each uniformly sampled bin:
a) Retrieve the saved hidden states.
b) Calculate pairwise distances (e.g., average cosine distance) among the segment
embeddings.
Step 7: Produce the final mapping of mean perplexity to average pairwise distance.
Uniform Perplexity Sampling. Since random sampling of text segments often yields distribution heavily concentrated around moderate perplexitics, we use uniform sampling scheme to cover both low- and high-perplexity "tails." The pseudocode below highlights the procedure used in Step (Algorithm
httess//huggnnnreee.coo/asssssss ymai
12

---

Published as a conference paper at ICLR 2026
Algorithm 1 Uniform Perplexity Binning
Require: A sorted list of G perplexity bins {bI..... bo} (with means my S ... S mm))
Require: Desired number of bins K
Ensure: A set of K bins sampled uniformly in perplexity
1: Mmmin f- mii max f mG
2: Define targets
k-1
LE Mmii
(Mmax Momin) for k = 1,.... K
K-1
3: selected t-
4: for k 4- 1 to K do s.t. 6: selected < selected U {j}
5: find s.t. my closest to is
7: end for
8: if [selected] K then
9: add extra bins from the sorted list until you have
10: end if
11: return {b j selected}
This ensures we sample across the entire perplexity spectrum, capturing both rare, low-ppl segments and rare, high-ppl segments. With these selected bins in hand, we can then compute the final-layer embeddings and measure representation dispersion to obtain a mean-ppl vs. dispersion plot.
A.1.3 ADDITIONAL VISUALIZATIONS
Below, present the full set of ceeplexity-versu--iisseee dataset and model, we group 100,000 text segments into perplexity bins and compute their average pairwise representation distances. As described in the main text, we observe a negative correlation between sequence-level perplexity and representation dispersion.
plots referenced in 22. For each
WikIText (Lioma- 3.2-11
WildText (Liama 3.2-38 38)
WikIText (Lioma 3.1 (8)
Peirson
0.50
Fannce
Peirson
2.64
0.78
8.40
5.89
3.50
2.84
0.72
2.52
0.70
Moon Perplexity
Moan Porsloxity
Moan Perplexity
Figure 10: Perplexity vs. Average Pairwise Cosine Distance on Wikitext-103 (Llama family).
(Gemma-2-22
(Gemma-22 2-98)
Desttan
0.66
Fa 6:37
Anornnge Casire Diruunce
E.6
0.62
0.62
0.60
0.58
056
054
102
220
200
220
Moan Perplealty
Meen Perpledity
Figure 11: Perplexity Average Pairwise Cosine Distance Wikitext-100 103 (Gemma family).
13

---

Published as a conference paper at ICLR 2026
WikTTaxt(Qwen2.55e
2,500
F 0.4
0.65
9.090
2.75
Codfee Disanoo
0.30
5.00
0.70
9.255
5:39
3.03
2,05
1.49
2,40
2.5
1.35
3.45
15
25
13
Figure 12: Perplexity vs. Average Pairwise Cosine Distance on Wikitext- 103 (Qwen family).
WikiTaxt (MitTaaxttMtttrallBBBvv..
(Phit)
Paurer p 0.000
Avcrage Cusine Dirance
0.02
R.80 Cco
0.70
0.76
0.01
0.70
Perpleaity
Perpledit
Figure 13: Perplexity vs. Average Pairwise Cosine Distance on Wikitext-103 (Mistral) and Phi.
News (L/mma-1.2-18)
News (Liama-3.2-388
News (UJmma-3.1-0
2.10
2.00
0.75
p Eren
0.50
Avarge Codtre
9.52
0.65
0.75
3.40
0.70
2.30
0.55
055
2.50
Parskoiity
Maan Porplckly
Nuan Perstoiyy
Figure 14: Perplexity vs. Average Pairwise Cosine Distance on CNN DailyMail (Llama family).
(Gemme-2-28)
0.375
2.3000
0.550
Aperage Coline Dulunce
0.425
0.200
0.375
1.35
0.72
50
100
150
30
350
150
Nean Perplently
Perplesity
Figure 15: Perplexity vs. Average Pairwise Cosine Distance on CNN DailyMail (Gemma family).
14

---

Published as a conference paper at ICLR 2026
(Qwen2.5-0.33
News (Qwen2.5:38)
(Qwen2.5-78)
2.50
0.00
Aecigg Codlee DisuINe
2.40
0.455
6000
0450
0.30
0.42
0.54
2.40
0.375
0.150
3.54
0.10
20.2
12.5
Ma-n Perplecity
Hean Perplccty
Nean Perptesity
Figure 16: Perplexity Average Pairwise Cosine Distance CNN DailyMail (Qwen family).
News (Mistral-78-vo.. 1)
News (Phi-2)
Person
0.00
Angrage Coalne Stance
0.8c
0.54
0.00
0.70
Nean Perplexily
Mean Perplesity
Figure 17: Perplexity vs. Average Pairwise Cosine Distance on CNN DailyMail (Mistral. Phi).
Medical (Lisma-3:2-18)
Medical (lamm-332-388
Medical (Llame-3.1-88)
Piatan
Naore
0.78
p-
0.70
Aucogje Costre D:2banc
2.50
0,74
0.72
0.30
3.50
0.58
3.45
0.66
0.50
Perpteiity
Perstoiity
Figure 18: Perplexity vs. Average Pairwise Cosine Distance on PubMed Summarization (Llama family).
Medical (Gemma-2-28)
Medicaa (Gemma-2-98)
0.00
Portce
0.5
Portan
Aeseupee Cuaine Diu-se
8:55
0.60
8.45
0.15
Noon Parplaxll
Mean Parploliyy
Figure 19; Perplexity vs. Average Pairwise Cosine Distance PubMed Summarization (Gemma family).
15

---

Published as a conference paper at ICLR 2026
Medical 20wen2.5-0.8B
Medical (Qwen2.5-38)
Medical (Qwen2.5-7))
0.05
p-
5i60
Masrgge Casno Dic0n00
0.65
0.60
0.55
80
0.50
0.15
0.35
Parpkeiity
Ferplexty
Persteiiy
Figure 20: Perplexity vs. Average Pairwise Cosine Distance on PubMed Summarization (Qwen family).
Medical (Mistral-7B.vo.1)
Medical (Phi-2)
Araroge Oasine DunLiec
12.52
0.60
3.70
0.58
Perpleally
Perplesity
Figure 21: Perplexity vs. Average Pairwise Cosine Distance on PubMed Summarization (Mistral, Phi).
16

---

Published as a conference paper at ICLR 2026
As shown in the figures, the observed negative correlation between sequence-lever perplexity and representation dispersion holds consistently across:
Multiple model sizes and architectures (Llama, Gemma, Mistral, Phi. Qwen).
Multiple data domains (Wikitext-103. CNN DailyMail, PubMed Summarization).
These findings support the main paper's s claim that lower-perplexity contexts tend to occupy more "spread out" regions in the final-layer embedding space, while higher-perplexity (i.e., more challeng- ing) contexts appear more compressed.
17

---

Published as a conference paper at ICLR 2026
A.2 DETAILS REGARDING FINE-TUNIN EFFECT EXPERIMENTS
In 2.2 we examined how fine-tuning influenced representation dispersion. Below are the hyperpa- rameters for the two fine-tuned LLaMA-3.2-1 models used in our experiments. We fine-tuned both checkpoints using the open-source LLaMA-Factory framework 6
A.2.1 LORA FUNE-TUNED MODEL
This model a fine-tuned version of meta- 1aaaaalla//aaama Llama- -.00aa-3.222 3.2-1121 1BBB-BBB on the Wikitext- 103 dataset. It achieved the following the evaluation set:
Loss: 2.1764
Training Hyperparameters.
learning_rate 0.0001
train_batcch__ size:
eval_batch_siiee
seed: 42
gpadlent_ecumllattoo_sssss 8
tstal_traii__aachh_s.ii 64
optimizer: adamw_ with 31 0.9, 82 0.999, e 1 10-8 8 no additional arguments
Ir w_wheehler_ttppeee cosine
lr_scheduler_waaeeee ratto 0.1
aum_epochs: 1.0
A.2.2 FULL-PARAMETER FINE-TUNED MODEL
This model is also fine-tuned version of lmmet-lllameee 21a.ma...maam111 the Wikitext- 103 dataset. It achieved the following on the evaluation set:
Loss: 2.1333
Training Hyperparameters.
learning_rate: le-05
train_batch_siizer
eval_batch_sizze:
seed: 42
distribbted__tpe::ne mat. multi-GPU
num_devices:2
gradieeit_accmmllaaiiiosssssss 16
totai__traih_baccchh_sseddd 64
total_ewal_bbatch_siiess 2
optimizer: Adam with 31 0.9, 82 0.999, 10-6
Ir sys cosine
l 0.1
num_epochs: 5.0
18

---

Published as a conference paper at ICLR 2026
A.3 DETAILS REGARDING DISPERSION WITHIN SEMANTIC CLUSTERS TRAINING HYPERPARAMETERS
In 2.3 examined how dispersion evolves within carefully constructed semantic clusters of text segments that share the same 10-gram continuation. Below are the training hyperparameters for the model used in this experiment:
learning_rate: le-05
train_ train_batch_siee 10
@ral_batch_st1
seed: 42
distributed_tyye: multi-GPP m
num_devices: 8
gradient__ccmmllatiio_ssssss 8
total__tralh_haatchh_sssse..e. 640
totel_eval__aacch_szeee 8
optimizer: ADAMN_TORCH with 31 = 0.9, 82 = 0.999, e = 1 x 10-8, 88 no additional
arguments
1 __scheuuler__tppe cosine
1r_ scheduler_warmup
0.1
num_epochs: 5.0
We used these hyperparameters to train the model from checkpoint of meta-llama, /Llamal 3. 2- on WikiText-103, then tracked within-cluster and between-cluster distances of the resulting contextual embeddings at several checkpoints during training. The model is also fine-tuned using the open-source source LLLMA-Factor framework.
19

---

Published as a conference paper at ICLR 2026
A.4 GENERALIZATION HELD-OUT DATA
To verify that the correlation between dispersion and perplexity holds on unseen data, we compare the standard distribution (Validation) against strict Held-Out (Test) split using LLAMA-3.2- 1- BB
As shown in Figure 22 the negative correlation consistent across both settings. While the test split (Right) shows higher perplexity overall, the geometric relationship remains: lower perplexity samples are associated with higher embedding dispersion.
Scatter Pot 2.b0
Scotter Pist lamz 2,2
0.4
0.6
0.22
0.6
2en
0.00
0.12
0.4
(a) Validation Set (Standard)
(b) Validation vs. Test Split
Figure 22: Generalization Check (LLAMA-3.2-1BB). Left: The standard correlation on the validation
set. Right: The correlation on the held-out test set (orange) vs validation (bluc). The negative slope remains robust on unseen data.
20

---

Published as a conference paper at ICLR 2026
A.5 VENDI SCORE ANALYSIS
To ensure our results are not specific to the Cosine Distance metric, we replicate our analysis using the Vendi Score (Friedman & Dieng. 2023). Figure 23 presents side-by-side comparison of Average Cosine Distance (Left Column) and Vendi Score (Right Column) across three model scales.
In all cases (LLAMA-3.2-1D) 3.2-3B, and 3.1-8B), the Vendi Score follows the exact same trend as Cosine Distance: segments with lower perplexity exhibit higher diversity scores. This confirms that our main metric is reliable proxy for the intrinsic dimensionality and spread of the representation space.
21

---

Published as a conference paper at ICLR 2026
Scatter Flst for 12-1
Wend Searn vs PP brilame
042
0.6
0.4
0.6
0.5
13
(a) Cosine Distance (LLAMA-3.2-1B)
(b) Vendi Score (LL.AMA-3.2-IB
Scatter Fist 8am1-122-bb
Wendi Score PRI lama 2-36
Miui
Moun
(c) Cosine Distance (LLAMA-3.2) -3.2-BB
(d) Vendi Score (LLAMA-3322
0.0
Miuee
Moueh
(c) Cosine Distance (LLAMA-3.1) 3.1-BBB)
(f) Vendi Score (LLAMA-3.1-88)
Figure 23: Metric Robustness across Model Scales. We compare our primary metric (Average Cosine Distance, left) with the Vendi Score (right) across IB, 3B, and 8B parameter models. The structural relationship with perplexity is identical across both metrics, validating the robustness of our geometric analysis.
22

---

Published as a conference paper at ICLR 2026
B SUPPLEMENTA MATERIALS FOR APPLICATIONS OF REPRESENTATION DISPERSION
B.1 DETAILS REGARDING RANKING EXAMPLE HARDNESS WITHOUT LABELED DATA
B.1.1 ADDITIONAL RESULTS
Below we provide extended experimental results following the methodology of $3.1 Each figure contains results for three models: Ilama-3.2-IB-IIstruue JJamm--3223B-Insttuue and Llama-3.1- 8B-Instruct.
Liama 3.2 3.2-18 Instruct
Llama 3.2 38 Instruct
Llama 3.1 8B Instruct
0.320
Mean Pareise Detance
0.310
0.055
0.13
0.305
0.12
0.300
0.050
0.045
0.10
0.085
0.040
0.000
Figure 24: Downstream performance estimation ARC Challenge (containing results for Llama-
B.2-1B-Instre 1aama-3.2-3B-Instruue IIIntrucct,,, and Llama-3.1-8 1-BBBBBBBBBBBBBB- 1.----...---II--uccc)
Uama-332-188 2-tt-ttctt
(lama-3.2-38-Ins-e
Llama-3 1-88-Instrcc
0,0600
0.15
Woun Nerehe Diflancee
0.13
0,0550
0.14
0.12
0.0525
0.11
0,0475
0.0450
0.10
3.09
0.6425
0.09
2.08
20
4.
60
80
100
20
40
60
50
100
20
60
80
100
Fraction Correct 1761
Fraction Correct 1961
Fraction Correct 1961
Figure 25: Downstream performance estimation on MMLU (English) (containing results for Llama- 3.2-1 I-Intrrcc,, Llama-32-33--Inssee and 1-aamaa-.---B-Innneeee
Liama 3.2 18 Instruct
Llama 3.2. 3B Instruct
Llama 3.1 8B Instruct
Modi Moon Palnelie Dizancs
8.C48
0,090
0.046
0.080
0.075
0.042
0.065
20
80
100
20
80
100
20
80
100
Frectior Cornect (86)
Prection Correct (86)
Frectior Correct 196)
Figure 26: Downstream performance estimation on Multilingual MMLU (German) (containing results for 1.2ma-3.2-11 Instrruut,, 3.2---3...-BB--tttte and Llama-3.1 -nn--truuuc,
23

---

Published as a conference paper at ICLR 2026
Llama-3.2 B-Instruct
Llama-3.3 w.w.-..-1188-eeeeee 88-Instruct
0.14
0.300
Mear Painstie Distarre
0.050
0.12
0.645
0.085
0.046
0.042
0.10
0.075
0.070
0.240
20
80
100
20
80
300
20
80
100
Practior Correct 196
Praction Correct 1961
Fractior Correct 1961
Figure 27: Downstream performance estimation on Multilingual MMLU (Spanish) (containing results for Llama-3.2-1BB In-ttucctt Llama-3.2-32 22-3B-Instrruc, and Llama-3.1-8B
Liama 3.2 1B Instruct
Llama 3.2 3B Instruct
Llama 3.1 8B Instruct
0.100
0.046
0.005
Moan PariisssDDDa Dirance
0.044
0.12
0.005
0.11
0.080
0.040
0.10
0.070
0.238
0.065
20
80
100
20
800
100
20
80
100
Proction Correct (96)
Proction Correct 1961
Froctioo Correct 1961
Figure 28: Downstream performance estimation on Multilingual MMLU (French) (containing results for L.amm-3.2-1IB- Ins-trucc,,,, L.ama-3.2-3B-Insttee and Llama-3.1-88 --8-111tttttt
Llama-3.2 3-mma-3--1Bt B-Instruct
1rmma-3.2-38-Instrne
Llama-3.1 1.11-88--nstturct
0.050
Mean Painstie Distares
0.049
0.C48
0.647
0.048
0.645
0.070
0.046
0.005
0.065
20
20
100
20
80
100
20
80
100
Practior Correct (16)
Praction Correct 1961
Fractior Correct 1961
Figure 29: Downstream performance estimation on Multilingual MMLU (Hindi) (containing results for Llama-3.2 B--trruuct Llama-3.22 2--2--B-Instuuue and Llama-3.1-88 1.B--B--IIBB---- Instrrutu....
Liama 3.2 2-1B- I--nnsstuctt
Llama 3.2 3B Instruct
Llama 3.1 BB Instruct
0.10
0.048
0.13
Moan Paivitze PaiiiiiaDDDa Disancee
0.046
0.12
0.044
0.642
0.040
20
80
100
20
80
100
20
80
100
Praction
Correct (96)
Proctioon Correct (76)
Fractior Correct 1961
Figure 30: Downstream performance estimation on Multilingual MMLU (Italian) (containing results for Llama-3.2 1.mma-3.-1B--nnstteee Instrucc, Llama-3.2 I.ntruct and Llama-3. 1--1--8--nntuuue
24

---

Published as a conference paper at ICLR 2026
Liama--32 3.2-1B-Instrucc
Llamaa332 -.m-n--..--B-lii- 3--3t--nstttct
Liama 3.1- 1--8--nnttuctt
0.14
0.300
Moan Parvize Dicencce
0.048
0.13
0.045
0.046
0.085
0.11
0.044
0.075
3.69
0.070
0.040
20
40
60
80
100
20
40
80
300
20
40
80
100
Fractiar Correct 186
Fraction Currect 196)
Fraction Correct 166
Figure 31: Downstream performance estimation on Multilingual MMLU (Portuguese) (containing results for Llama-3.2- 1--1IB-nnttrut, -lama-3.2-3B-IIntee and Baama--.--BB-Intttueee
1amma-3.2-1B-Instruce
Haama-3.233--In-..
0.0450
0.0750
Meer Naterke Daterre
0.0445
0.0725
0,0440
0.9435
0.0675
0.070
0.0550
0.0430
0.0425
0.065
0.0600
0.060
20
4.0
60
80
300
20
60
60
80
300
20
40
60
80
300
Fraction Correct (66)
Fraction Correct 196)
Fraction Correct 196)
Figure 32: Downstream performance estimation on Multilingual MMLU (Thai) (containing results for Llama-3.2-1D IB-stuuttsst 1a-tma--.2-3B-Instttee and Llama-3.1 1-1-BBBBBBB----- Instructtt...
25

---

Published as a conference paper at ICLR 2026
B.1.2 COMPARISON TO PERPLEXITY AS A HARDNESS SIGNAL
In $3.1 we used representation dispersion rank slices of examples by their hardness for fixed model and dataset. Here we replicate the same controlled slicing protocol but compare dispersion against the model's 's own perplexity on the query text as two competing label-free hardness signals. For each dataset (MMLU and ARC-CHALLENGE) and each model size (Llama-3.2 IB/3B-Instrucc and Llama-3. 1aama-..-8B--Intruuuee --8--nnstruc), we construct synthetic slices with target fractions of correct answers from 0% to 100% in steps of 10%. For every slice, we compute (i) the mean dispersion of the slice and (ii) the mean perplexity of the slice, averaging over 10 random seeds.
Model
Frac, correct (%)
Perplexity
Dispersion
1B
100
5.273 = 0.063
0.0584
0.0015
90
5.232 = 0.056
0.0578
0.0016
80
5.196 = 0.068
0.0559 + 0.0016
70
5.201 = 0.064
0.0553
0.0015
60
5.178 = 0.050
0.0528
0.0013
50
5.166 = 0.033
0.0513
0.0013
40
5.147 = 0.044
0.0504 = 0.0013
30
5.092 0.054
0.0488 + 0.0013
20
5.038 = 0.042
0.0470 0.0010
10
5.070 = 0.032
0.0450
0.0008
0
5.058 = 0.033
0.0425 0.0008
3B
100
4.740 = 0.055
0.1525 0.0018
90
4.753 = 0.043
0.1507 + 0.0023
80
4.761 = 0.044
0.1468 + 0.0023
70
4.738 = 0.040
0.1414 0.0026
4.730 = 0.041
0.1355
0.0022
50
4.760 = 0.050
0.1298
0.0019
40
4.803 = 0.049
0.1221 = 0.0018
30
4.780 0.042
0.1158 0.0018
4.742 = 0.033
0.1077 0.0016
10
1.762 = 0.040
0.0987 + 0.0013
0
1.751 = 0.025
0.0923 + 0.0012
8B
100
4.528 = 0.044
0.1395 0.0013
90
4.500 = 0.043
0.1374 0.0014
80
4.477 = 0.047
0.1342
0.0015
70
4.472 = 0.047
0.1310 + 0.0020
60
4.488 = 0.048
0.1244 0.0014
50
4.498 = 0.051
0.1186 + 0.0014
40
4.488 = 0.045
0.1127 1 0.0014
30
4.458 = 0.049
0.1048 0.0016
4.438 = 0.045
0.0979
0.0020
10
4.419 = 0.044
0.0894 0.0015
4.418 = 0.043
0.0801 + 0.0012
Table 3: MMLU controlled slices: mean perplexity (ppl) and dispersion function of the fraction of correct answers. Dispersion is strongly monotone with correctness for all model sizes, whereas perplexity is nearly flat.
Across both datasets and all three model sizes, dispersion is strongly monotone with the fraction of correct answers, while perplexity is nearly constant and sometimes even slightly better (lower) on the hardest slices than the easiest ones. This supports the claim that dispersion captures geometric information about contextual separation that is not reducible to next-token confidence alone.
26

---

Published as a conference paper at ICLR 2026
Model
Frac. correct (%)
Perplexity
Dispersion
IB
100
21.846 + 0.083
0.0607 + 0.0013
90
21.670 + 0.105
0.0589 + 0.0013
80
21.466 + 0.148
0.0573 0.0012
70
21.289 0.130
0.0553 0.0011
60
21.389 = 0.183
0.0538 = 0.0010
50
21.094 + 0.226
0.0516 = 0.0009
40
20.898 + 0.236
0.0491 = 0.0008
30
20.842 0.225
0.0456 0.0007
20
20.779 = 0.239
0.0436 0.0007
10
20.561 + 0.237
0.0426 1 0.0006
0
20.589 + 0.227
0.0404 1 0.0007
3B
100
13.943 + 0.090
0.1437 = 0.0020
90
13.917 = 0.099
0.1418 = 0.0019
80
13.875 0.101
0.1381 0.0022
70
13.957 + 0.111
0.1328 0.0021
60
13.933 + 0.107
0.1277 + 0.0024
50
14.052 + 0.081
0.1232 + 0.0025
40
13.966 = 0.091
0.1183 = 0.0018
30
13.939 0.120
0,1111 0.0019
20
13.915 + 0.117
0,1051 = 0.0020
10
13.862 + 0.081
0.0986 = 0.0014 4
0
13.846 + 0.090
0.0900 = 0.0010
8B
100
16.313 +:: 0.157
0.1177 1: 0.0014
90
16.272 1 0.160
0.1157 +: 0.0013
80
16.247 0.156
0.1132 0.0013
70
16.102 = 0.189
0.1105 = 0.0015
60
15.939 + 0.176
0.1077 = 0.0013
50
16.052 + 0.158
0.1045 + 0.0014
40
15.931 + 0.166
0.0997 0.0011
30
15.869 + 0.109
0.0951
0.0014
20
15.748 + 0.113
0.0905 = 0.0012
10
15.755 + 0.147
0.0844 = 0.0008
0
15.835 + 0.094
0.0794 = 0.0007
Table 4: ERC-CHALEENGE controlled slices: mean perplexity (ppl) and dispersion as a function of the fraction of correct answers. Again, dispersion is strongly monotone with correctness, whereas perplexity fails to separate easy from hard slices.
27

---

Published as a conference paper at ICLR 2026
B.1.3 CALIBRATION BY DISPERSION
To test whether dispersion can be used as calibrated proxy for accuracy on truly unseen data, perform a "Calibration by Dispersion" experiment on MMLU. We partition the dataset into three disjoint subsets: (1) a Calibration Ser (20% of the data), (2) Validation Ser (10%), and (3) Test Ser (70%). On the calibration set we use a dense sampling strategy to create synthetic batches with target accuracies ranging from 0% to 100% of the Dispersion
increments, which allows us to densely sample the shape
curve.
We then train a suite of regression models- linear, polynomial, isotonic, and random forest-a to predict accuracy from dispersion, and select the model that minimizes prediction error the validation set. Finally, we apply the selected regressor to the unseen test set (using dispersion only) and compare the predicted accuracy to the true accuracy. We repeat this procedure over 10 random seeds and report the mean absolute error (MAE) between predicted and true accuracy. Table summarizes the results.
Model
MAE (%)
Llama-3.2-1B 1.--a-...----Bmmm- I-Istuunt
1.75
Llama-3.1-88 1--8B-Instruct
2.15
Table Mean Absolute Error (MAE) between predicted accuracy (derived solely from dispersion) and true accuracy on held-out 70% MMLU test set. Results are averaged over 10 random seeds. Dispersion supports highly accurate calibration, with absolute error of roughly 1.4-2.2 percentage points.
These results show that, for fixed model and dataset, dispersion only monotonically related to correctness but can also be mapped to well-calibrated accuracy estimate with small error. A practical workflow emerging from this experiment
1. Label small random subsample of the data (e.g., 10-20%)
2. Fit standard regression curve (c.g.. isotonic or low-degree polynomial) from Dispersion Accuracy on this subsample.
3. Apply the learned mapping to the remaining unlabeled data to estimate performance with typical margin of error of about 2 percentage points.
While this calibration is, by construction, model- and domain-specific, it demonstrates that dispersion can serve a practical, calibrated proxy for accuracy once small labeled calibration is available.
28

---

Published as a conference paper at ICLR 2026
Frac. correct (%)
mean s.c.
mean s.c.
8B mean s.c.
100
0.0350 + 0.0004
0.0709 + 0.0010
0.0750 + 0.0012
0.03445 1 0.0005
0.0694 1 0.0009
0.0738 0.0013
80
0.03413 1: 0.0004
0.0674 1: 0.0010
0.0714 +: 0.0012
70
0.03.11 1: 0.0005
0.0665 1: 0.0009
0.0692 1: 0.0011
60
0.0335 = 0.0006
0.0642 + 0.0011
0.0669 + 0.0009
0.0325 + 0.0006
0.0620 0.0011
0.0646 0.0007
40
0.0319 = 0.0006
0.0599 + 0.0009
0.0627 = 0.0006
30
0.0311 + 0.0006
0.0583 + 0.0010
0.0605 + 0.0006
20
0.0306 + 0.0006
0.0560 + 0.0007
0.0578 + 0.0009
0.0306 + (10006
0.0540 + 0.0006
0.0546 + 0.0010
0.0302 + 0.0005
0.0511 + 0.0008
0.0513 + 0.0008
Table 6: HELLASWAG HELLASWAG_CHATT SHTT
mean dispersion (with standard error) function of the fraction of correct answers for Llama-3.2-1BB3B-- -.nttutt and
Frac. correct ( (%)
1B mean + s.e.
3B mean + s.e.
8B mean + s.e.
100
0.7289 + 0.0069
0.7266 + 0.0064
0.7113 + 0.0039
90
0.7277 1 0.0068
0.7277 1 0.0070
0.7141 +: 0.00413
80
0.7238 + 0.0061
0.7250 +: 0.0048
0.7091 + 0.0031
0.7203 +: 0.0054
0.7270 + 0.0050
0.7074 + 0.0031
60
0.7180 + 0.0052
0.7246 + 0.0035
0.7039 + 0.0037
50
0.7145 + 0.0053
0.7211 + 0.0038
0.7043 + 0.0055
40
0.7133 + 0.0042
0.7238 + 0.0033
0.7031 + 0.0048
30
0.7113 + 0l0045
0.7191 + 0.0042
0.7031 + 0.0036
0.7039 + 0.0037
0.7156 + 0.0056
0.6996 + 0.0041
10
0.7070 + 0.0030
0.7172 + 0.0044
0.6961 + 0.0021
0
0.7078 1 0.0018
0.7164 1 0.0039
0.6926 1 0.0015
Table 7: [F_EVAL mean dispersion (with standard error) as a function of the fraction of correct answers for 3.B.-.//B1//B-nntrrr and 10mma-3.1-8B-Inset
B.1.4 TESTING DISTRIBUTION SHIFT
To probe robustness under distribution shift, we further apply the controlled slicing experiment from to three datasets with distributions very different from MMLU and ARC-CHALLENGE: HELLASWAG. CHAT, IF EVAL, and QUAIL. For each dataset and for each of the three Llama-Instruce Instruct models, we construct slices with target fractions correct from 0% to 100% in steps of 10%, and compute the mean dispersion (with standard errors) for each slice. In all cases we observe the same qualitative trend: higher fractions correct correspond to larger mean dispersion, confirming that dispersion remains reliable hardness signal even under substantial distribution shift. The full numbers are reported in Tables 6
29

---

Published as a conference paper at ICLR 2026
Frac. correct (%)
1B mean s.e.
3B mean 1 S.c.
8B mean 1 s.c.
100
0.1938 + 0.0020
0.1259 + 0.0024
0.0893 + 0.0014
90
0.1936 + 0.0020
0.1234 + 0.0024
0.0880 + 0.0015
80
0.1941 = 0.0021
0.1220 + 0. 0.0023
0.0865 + 0.0014
70
0.1930 + 0.0021
0.1207 + 0.0023
0.0841 + 0.0014
0.1931 +: 0.0023
0.1186 + 0.0028
0.0826 + 0.0014
50
0.1915 + 0.0021
0.1165 + 0.0024
0.0806 + 0.0012
40
0.1913 + 0.0016
0.1148 + 0.0018
0.0788 + 0.0010
30
0.1901 + 0.0013
0.1122 + 0.0024
0.0762 + 0.0007
0.1898 + 0.0012
0.1087 0.0022
0.0735 0.0009
0.1876 + 0.0011
0.1046 + 0.0019
0.0686 + 0.0009
0.1884 + 0.0012 0.1014 + 0.0020
0.0656 + 0.0009
Table 8: QUAIL: mean dispersion (with standard error) as a function of the fraction of correct answers for
and Llama-3. 1ettaa...--B-Inaa I-8B-Instruct.
30

---

Published as a conference paper at ICLR 2026
B.2 DETAILS REGARDING REPRESENTATION DISPERSION FOR MODEL SELECTION
B.2.1 FULL NUMERIC STATISTICS
This appendix compiles the full numeric statistics that underpin the analyses in $3.2 We report complete Euclidean and cosine distance figures for every model variant, together with their task- specific accuracies, so that readers can perform fine-grained checks, reproduce our correlation calculations, and explore alternative dispersion metrics. Table 2 Table TO Table and Table T2 complement the visual summaries in Figure by exposing each component of the Dispersion Gap in detail.
Table 9: Embedding Dispersion vs. MATH Performance (Qwen Variants, Euclidean). We show the mean + standard error of Euclidean distances among digit embeddings (D-D), among non-math tokens (NM-NM), and between digits and non-math tokens (D-NM). We also list each model's accuracy on MATH (%).
Model
Euclidean Distances
MATH (%)
D-D
NM-NM
D-NM
Qwen2.5-1.5B
0.7006 + 0.0087
1.4072 + 0.0268
1.4331 t: 0.0222
35.0
Qwen2.5-Math-LL55
0.8916 + 0.0111
1.6423 = 0.0286
1.6991 + 0.0203
49.8
Distil-QQenn-L5B
0.9406 1 0.0103
1.6104 = 0.0247
1.7014 + 0.0188
83,9
Qwen2.5-7B
0.4505 + 0.0052
0.8712 0.0189
0.9840 t 0.01
49.8
Qwen2.5-Math-7B
0.6896 + 0.0076
1.3479
0.0287
1.4044 = 0.0205
55.4
wieni
0.7216 1 0.0076
0.0292 1.3406
1.4244 1 0.0215
92.8
Qwen2.5-14B
0.6993 0.0074
1.5284
0.0314
1.4550 + 0.0168
55.6
Distill-Qeen-4BB
0.7415 0.0073
1.5223
0.0402
1.4659 + 0.0229
93.9
Table 10: Embedding Dispersion MATH Performance (Qwen Variants, Cosine). We show the mean standard error of Cosine distances among digit embeddings (D-D), among non-math tokens (NM-MM), and between digits and non-math tokens (D-NM) We also list each model's accuracy MATH (%).
Model
Cosine Distances
MATH (%)
D-D
NM-NM
D-NM
Qwen2.5-1.5B
0.2489 1 0.0060
0.9334 1 0.0089
1.0068 1 0.0217
35.0
Qwen2.5-Math-1.55
0.3347 + 0.0074
0.8984 = 0.0110
1.0755 + 0.0172
49.8
Distill-Qeen-15B
0.3562 1 0.0073
0.8973 = 0.0109
1.0781 + 0.0165
83,9
Qwen2.5-7B
0.1786 +: 0.0042
0.9360 0.0072
1.0000 + 0.0159
49.8
Qwen2.5-Math-7B
0.2672 + 0.0061
0.9257 = 0.0089
1.0554 = 0.0166
55.4
DietillQQeen7BB
0.2775 1: 0.0061
0.9260 + 0.0084
1.0640 1 0.0177
92.8
Qwen2.5-14B
0.2573 0.0053
0.9333
0.0084
0.9622 + 0.0100
55.6
Dietill-Qeen-44B
0.2811 0.0054
0.9350 0.0111
0.9701 # 0.0135
93.9
Table 11: Embedding Dispersion vs. HumanEval Performance (Llama2 vs. CodeLlama, Eu- clidean). We report the mean + standard error of Euclidean distances among code tokens (C-C), among non-code tokens (NC-NC). and between code and non-code tokens (C-NC). We also list each model's HumanEval pass@) (%).
Model
Euclidean Distances
HvmanEval (%)
C-C
NC-NC
C-NC
Llama2-7B
1.4961 + 0.0060
1.3741 + 0.0313
1.4465
0.0148
12.2
CodeLlama-7B
2.3417 1 0.0110
2.3437 1 0.0475
2.3623
0.0249
33.5
Llama2-13B
2.1274 + 0. 0.0098
1.9002 + 0.0524
2.0324 0.0247
20.1
CodeLlama-133
2.5499 +: 0.0134
2,5305 + 0.0539
2.5397 + 0.0275
36.0
31

---

Published as a conference paper at ICLR 2026
Table 12: Embedding Dispersion HumanEval Performance (Llama2 vs. CodeLlama, Cosine). We report the mean + standard error of Cosine distances among code tokens (C-C), among non- code tokens (NC-NC), and between code and non-code tokens (C-NC). We also list each model's HumanEval pass (%).
Model
Cosine Distances
HumanEval
C-C
NC-NC
C-NC
Llama2-7B
0.9603 i 0.0021
0.9402 0.0075
0.9652
0.0028
12.2
CodeLIamm-7BB
0.9451 + 0.0027
0.9486 0.0074
0.9636
0.0037
33.5
1lama2-138
0.9340 + 0.0025
0.9283 + 0.0101
0.9460 + 0.0061
20.1
lama-Ia3
0.9433 + 0.0027
0.9435 + 0.0078
0.9589 + 0.0037
36.0
32

---

Published as a conference paper at ICLR 2026
B.2.2 ROBUSTNESS OF DISPERSION CORRELATION ACROSS TRAINING TRAJECTORIES
In $3. we demonstrated strong correlation between the Dispersion Gap and model performance across different model families. To verify that this relationship is not an artifact of small sample sizes or specific model architectures, we conducted high- high-resolution analysis of single model's ' training trajectory.
Setup. We utilized the training checkpoints the Olmo-7B model, sampling every 20,000 steps from step 120,000 step 700,000. We exclude the initial 120,000 steps to bypass the 'burn-in' phase and focus on the stable training regime where representational geometry has consolidated. This yielded set of N 30 distinct checkpoints. For each checkpoint, calculated the Dispersion Gap (9) using our standard protocol and compared it against the model's zero-shot performance on GSMSK (Math) and HUMANEVAL. (Code).
we observe remarkably strong monotonic relationship between dispersion and downstream accuracy throughout the training process.
Results. As shown in Figure 33
For Mathematical Reasoning, the correlation between the Dispersion Gap and GSM8K accuracy is p = 0.90 (p < 10-11).
For Code Generation, the correlation between the Dispersion Gap and HUMANEVAL pass@) is p 0.95 (p 10 15).
These results reinforce our claim that representation dispersion serves robust, label-free proxy for model capability. Notably, the dispersion metric tracks the underlying improvement of the model with high fidelity, maintaining strong correlation even amidst the natural fluctuations inherent in generation-bant based evaluation benchmarks.
Math: Dispersion GSV8K Speaman=00.00
Code: Dispersion vs HumanEval Spearman=0.95
accurtty
HumanEval Poo:91
0.07
0.00
0.05
8..
0.2
01.4
8.6
11.0
9.0
9.2
33.0
13.5
11.3
14.5
15.0
15.3
16.0
Lispersion bap (Math)
Dispersion cap (Code)
Figure 33: Dispersion Gap tracks performance across the Olmo-7B training trajectory (N 30). observe extremely high Spearman rank correlations (p Re 0.90 for Math and z 0.95 for Code) with high statistical significance (p 10-11). Shaded regions indicate the 95% confidence interval.
33

---

Published as a conference paper at ICLR 2026
B.3 DETAILS REGARDING LAYER SELECTION FOR KNN-LM
B.3.1 ADDITIONAL RESULTS
We present extended findings on sub-layer selection for kNN-LM, Figure 34 displays results for four GPT-2 variants (disti 11ttt,, gpt2, gpt2-medi um, 9pt2-large). As the main text, each point represents 512-token chunk of text, with its mean perplexity plotted against the sub-layer's average pairwise cosine distance (blue for attention output, for the feed-forward output). Interestingly, the negative correlation is weaker for the attention output than for the feed-forward output.
DistilGPT-2
GPT-2 Small
0812
0.822
0.8
Distance
Avg. Paireite Costine Distance
0.7
0.6
0.5
0.340
0.265
0.1
0.1
30
120
15
45
Mean Perplexity
Mean Perplexity
GPT-2 Medium
GPT-2 Large
0.9
Pairwise Distance
Avg. Pairwizu Cosine Distance
0.6
0.8
0.5
0.3
0.6
0.230
0.2
0.5
0.1
16
24
32
12
18
24
30
Mean Perplexity
Mean Perplexity
ATT (Attention Layer)
FFN (Food-Forward Layer)
Trend Line
Figure 34: Mean perplexity vs. sub-layer average pairwise cosine distance for four GPT-2 variants (distilgpt2, 2pt gpt2-medium gpt2-largen Each point a 512-token chunk text.
34

---

Published as a conference paper at ICLR 2026
B.4 DETAILS REGARDING INCORPORATING REPRESENTATION DISPERSION
B.4.1 TRAINING DETAILS
All experiments in 3.4 were conducted on NVIDIA A100 80GB GPUs.
Single-Domaie Setting. We train GPT2-SMALL from scratch on WikiText, using a batch size of 64 and block size (sequence length) of 512. The auxiliary loss weight À is tuned over the set:
{0.5, 0.2, 0.1, 0.07, 0.05, 0.02, 0.01, 0.007, 0.005, 0.002, 0.001}
We experiment with learning rates {1 x 10-3, 7 x 10-1, 5 x 10-1).
Cross-Domales Setting. For joint WikiText Python code training, we similarly use GPT2-SMALL (from scratch), a batch size of 128, and block size of 256. The auxiliary loss weight À and learning rates are swept over the same sets as above:
A values: {0.3, 0.2, 0.1, 0.07, 0.05, 0.02, 0.01, 0.007, 0.005, 0.002, 0.001}
Learning rates: (1 10-3, 7 1
5 x 10-4]
For both settings, we select À by validation for each learning rate. All experiments use standard AdamW optimizer settings unless otherwise specified.
35

---

Published as a conference paper at ICLR 2026
B.4.2 IMPACT OF ENFORCING CLUSTERIZATION ("SQUEEZE" ABLATION)
To further investigate the causal relationship between representation dispersion and model perfor- mance, we conducted an ablation study where we explicitly discouraged dispersion. In contrast to the "push-away" objective described in $3.4, here added an auxiliary loss that minimizes the average pairwise cosine distance, effectively squeezing representations into tighter clusters.
We trained GPT2-SMALL on WIKITEXT- 103 using the same hyperparameter sweep our baseline experiments. Table 13 compares the test set perplexity of the Baseline model against the "Squeeze" model (where the auxiliary loss weight 0.1).
Perplexity (Lower better)
Learning Rate
Step
A PPL
Baseline
Squeeze (Cluster)
500
226.1
232,9
+6.8
1 x 10-3
1000
111.3
137.0
+25.7
500
195.0
206.1
+11.1
x 10
1000
96.7
124.2
+27.5
500
166.2
169.2
+3.0
x 10
1000
83.0
101.7
+18.7
Table 13: Efffec of artificially reducing dispersion ("Squeeze" ablation). Comparing the Baseline
model to a model trained with an auxiliary loss that forces emheddings to cluster. Across all learning rates, restricting the geometry leads to significantly higher (worse) perplexity, reinforcing the causal link that adequate embedding breadth is necessary for strong predictive performance.
The results show a consistent and significant degradation in performance when dispersion is penalized. For the optimal learning rate (5 x 10 4)_ the perplexity worsens by over 18 points at step 1000. This substantial performance drop supports the hypothesis that representation dispersion is functional driver of model performance; when the model is geometrically constrained from spreading its representations, its ability to minimize entropy and predict accurately directly impaired.
36

---

Published as a conference paper at ICLR 2026
B.4.3 RELATION TO CONTRASTIVE AND REPULSIVE OBJECTIVES
Our auxiliary "spread-out" loss from $3.4 is conceptually related to prior contrastive and repulsive objectíves that encourage more discriminative or isotropic representations, rather than being an entirely new idea. For example, Gunel al. (202T) introduce supervised contrastive loss for fine- tuning language models, in which representations of same-labeled examples are pulled together while representations of different classes are pushed apart. Likewise, Jain et al. (2023) propose ContraCIMM contrastive framework for causal language models that explicitly improves the discrimination of token and sequence representations. Traditional contrastive losses (e-g.. InfoNCE-style objectives) follow the same spirit: they add term during training that repels embeddings from one another (except for designated positive pairs), mitigating representation collapse and improving downstream generalization.
Our formulation can be viewed as a deliberately simplified variant of these contrastive objectives. In contrast InfoNCE-based losses, we do not define positive pairs sample specific set of negatives for each anchor. Instead, our loss considers all pairs of hidden states in the batch and directly maximizes their average cosine distance. This simplicity has two practical benefits: (i) it avoids
sampling bias and the need for careful negative mining-eveen every pair contributes equally to the repulsive force, so we are less sensitive mini-batch composition heuristic data augmentations; and (ii) is easy to implement alongside the standard maguaage-oddelee dusine-distance computation without large softmax denominators or momentum sareenuuuuuuuuueeeee mechanisms. In this sense, our "push-away" loss can be viewed as a minimal, unsupervised contrastive regularizer that encourages a more uniform spread of representations on the unit sphere.
loss, involving only batch-wise
37

---

Published as a conference paper at ICLR 2026
C LIMITATIONS
While our findings underscore a strong empirical link between representation dispersion and model performance, there are several limitations. First, our analyses focus on average pairwise cosine distances of final-layer representations, which may not capture all nuanced aspects of embedding geometry or model behavior. Second, although we observe consistent negative correlations between dispersion and perplexity across several model families and domains, causality cannot be definitively concluded; certain architectures or objectives may modulate this relationship in unforeseen ways. Third, our experiments center primarily on English text from standard benchmarks and limited set of specialized domains (e.g. code, scientific abstracts). It remains unclear how well our observations extend to other languages, modalities, or highly domain-speciffe corpora. Further research is needed to fully understand these trade-offs and develop robust methods for controlling embedding geometry.
38


Reviews:
[{"review_id": "PP3yEMb4MQ", "reviewer": "Reviewer_3MuU", "summary": "The paper studies a simple geometric measure on representation dispersion (the average pairwise cosine distance among hidden vectors and shows it negatively correlates with language-model perplexity across model families and text domains. The authors report strong negative correlations between dispersion and perplexity at the sequence level and also for last-token perplexity across context lengths. They further observe the correlation strengthens in deeper layers and is amplified by (full) fine-tuning and that within-cluster as well as between-cluster distances grow during training. Finally, the paper shows how dispersion can be used for many practical tasks, such as predicting downstream performance, identifying best layers for retrieval, and introducing dispersion objective during pretraining.", "questions": "1. The average cosine distance does not indicate the spread-out or diversity of the embedding. Can you also try out Vendi Score [1] to see if the observations in the paper still holds? \n\nReference:\n1. Friedman & Dieng. The Vendi Score: A Diversity Evaluation Metric for Machine Learning. TMLR 23'.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Dispersion is very simple to compute and model-agnostic. Consistent trends are observed across many families and datasets. \n2. The paper demonstrates that dispersion increases even among highly similar contexts.\n3. More importantly, the observations can be used for many downstream tasks. Adding domain dispersion loss for pretraining seems a very simple technique to improve model performance.", "weaknesses": "1. My main concerns is that whether the correlation between dispersion and lower perplexity is due to memorization or generalization. Since LLMs are pre-trained on so many text in the web, the query data used in the experiments are seen by LLMs. Therefore, the resulting of dispersion is not that surprising as model tend to produce better representation on the data it has been seen. Could you provide a clear train / val split set up for GPT-2 to see why the dispersion can also happen for hold-out samples from the same domain. This would be the results more solid.\n2. Also, how would one come up with domain tag for dispersion loss proposed for pretraining LLM? Can we apply a domain classifier on web-scale pretraining data to label a text into a domain? Then apply dispersion loss on that. This would make the method useful actual LLM pretraining rather than a contrived case.\n3. Another question is whether this idea generally is applicable to vision encoder or multimodal applications? In fact, vision encoders are often trained with positive-negative samples regularization and therefore naturally encourages dispersion across pre-training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YB2ggiZudl", "reviewer": "Reviewer_CstH", "summary": "This paper argues that one can tie the predictive quality (i.e. through next token prediction) of a language model by a geometric quantity of the language model. The define the representation dispersion as the cosine similarity between vector embeddings produced by the language model between different inputs in the dataset. They argue that batches with high dispersion tend to be ones the model is more accurate on, as measured by the perplexity on the next token prediction. They propose several use cases, such using this for layer-selection, model-selection, or as a training objective.", "questions": "1. It might be of interest to compare the layer-wise patters you observe here to those that have been observed in prior work, such as \"Layer by Layer ...\" by Skean et al. They also propose a metric called \"dataset entropy\" which seems in some sense similar to what is measured here.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality: This paper presents a novel metric which states several interesting correlations. They then showcase some natural and interesting use cases for this.\n2. Quality: The empirical study in this paper is extensive, and convincing. \n3. Clarity: Everything is presented clearly, except for in section 3.2 (as mentioned below)\n4. Significance: There are several actionable applications presented that showcase how this framework could be helpful.", "weaknesses": "1. I think it is not obvious what to take from Figure 9. It seems like the relationship between the gap and the performance is not as obvious as the results section made it out to be. I am also not convinced that the Spearmans correlation is useful here on such a low number of data points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QM0FMymwqa", "reviewer": "Reviewer_uSU9", "summary": "This paper presents a comprehensive empirical study establishing a strong negative correlation between representation dispersion and perplexity in language models. The authors demonstrate this relationship across multiple model families and domains . Beyond this core correlation, the paper explores several practical applications of this insight: (1) predicting downstream accuracy on unlabeled data, (2) a \"dispersion gap\" metric for label-free model selection, (3) unsupervised layer selection for kNN-LM, and (4) an auxiliary training loss that increases dispersion and improves perplexity. The work is well-supported by extensive experiments and offers both conceptual insights and practical tools for the community.\n\nThis is an excellent paper that makes a substantial contribution to our understanding of representation learning in language models. It identifies a robust and generalizable phenomenon and, most importantly, derives a set of practical, effective, and efficient applications from this insight. The work is empirically sound, clearly presented, and has immediate utility for both researchers and practitioners. The minor weaknesses do not detract from the overall significance and quality of the work. I recommend acceptance.", "questions": "(1) Have you considered or attempted any ablation studies that directly reduce dispersion (e.g., by adding a loss that encourages representations to cluster) to see if perplexity increases? This would provide even stronger evidence for a causal link.\n\n(2) In Section 2.2, you note the negative correlation strengthens in deeper layers. Do you have a hypothesis for why this is the case? Is it simply that deeper layers are more task-specific and directly linked to the final prediction?\n\n(3) The auxiliary loss requires computing pairwise distances within a batch, which is O(B²). Did you find this to be a significant computational bottleneck during training, and were any approximations (e.g., sampling pairs) considered for larger batch sizes?\n\n(4) A Minor Issue. On Page 8, Table 1: The standard deviations for GPT2-Large (FFN, N=10) are `0.68±0.06`, which is very large compared with other values in the same column. Is this expected? A brief comment might be useful.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The core finding—higher representation dispersion correlates with lower perplexity—is simple, intuitive, and empirically robust. The paper does an excellent job of moving beyond mere observation to demonstrate a range of practical applications, making the finding directly useful for practitioners.\n\nThe empirical analysis is thorough. The authors validate their claims across a wide range of models and datasets, which significantly strengthens the generalizability of their conclusions.\n\nThe proposed applications are compelling and well-executed, making strong and practical contribution. The \"dispersion gap\" for model selection (Section 3.2) is a particularly elegant and efficient method that requires no forward passes, only the model's output embedding matrix. The layer selection for kNN-LM (Section 3.3) provides a simple, unsupervised heuristic that could save significant computational resources. The auxiliary training objective (Section 3.4) successfully translates the observational finding into an actionable method for improving model performance, especially in the cross-domain setting.\n\nThe paper goes beyond global averages to perform a fine-grained analysis of dispersion within semantic clusters (Section 2.3). This is a crucial experiment that convincingly shows the effect is not merely about pushing dissimilar contexts apart but also about separating semantically similar ones, reinforcing the core hypothesis.\n\nThe paper is well-written, clearly structured, and the figures effectively illustrate the key trends. The appendices provide substantial additional detail and results.\n\nThe paper is highly reproducible. The methodology for computing dispersion is clearly defined. The appendices provide extensive details on datasets, models, hyperparameters (for fine-tuning and training with the auxiliary loss), and the uniform binning algorithm. The use of standard, publicly available models and datasets further aids reproducibility.", "weaknesses": "The conclusion slightly lacks for causation. The core evidence is primarily correlational. The auxiliary loss experiment in Section 3.4 is the strongest argument for causality, but it is limited to the GPT-2 architecture. A more nuanced discussion of this distinction would strengthen the paper. Is dispersion a fundamental driver of performance, or is it a side-effect of a model learning better, more discriminative features?\n\nThe experimented domain/language coverage is slightly limited. As acknowledged in the limitations (Appendix C), the experiments are primarily on English text from standard benchmarks. While the inclusion of code is a positive step, the claim of \"diverse domains\" could be more strongly supported by including data from truly different distributions (e.g., low-resource languages, highly technical manuals, social media). The multilingual MMLU results in the appendix are a good start but are still based on translated versions of a known benchmark.\n\nThe discussion of the most related work (Viswanathan et al., 2025) is not sufficient. The authors state their work is more \"actionable,\" but a more direct comparison of the findings (e.g., do they observe similar dispersion patterns?) would better situate this contribution within the existing literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KV0YtF8XNt", "reviewer": "Reviewer_4BDa", "summary": "The paper introduces a simple geometric statistic of language model representations, called \"dispersion,\" defined as the mean pairwise cosine distance between hidden states (or between token embeddings). Higher dispersion corresponds to representations that are more spread out in embedding space. The authors argue that dispersion tracks model quality: it is strongly and consistently negatively correlated with perplexity, and can therefore be used for practical purposes such as predicting downstream accuracy in new domains and ranking or selecting models. They further propose a \"push-away\" auxiliary loss that explicitly increases dispersion during training, and report that this improves perplexity.", "questions": "Q1: Can perplexity on unlabeled in-domain text replace dispersion in Sections 3.1 and 3.2? If perplexity already produces the same ranking or predictions, then dispersion does not add unique value beyond possible computational savings. If dispersion performs better, please quantify that improvement.\n\nQ2: In Section 3.2, can the proposed model-ranking method generalize across different model families (e.g., Qwen vs Llama), or is it only intended for ranking closely related checkpoints within one family?\n\nQ3: Can you provide standard errors for Tables 3 and 4 to clarify whether the reported dispersion gaps (e.g., D–D = 0.27 vs 0.28) meaningfully differ?\n\nQ4: For Section 3.1, can you provide evidence that dispersion can yield a calibrated accuracy estimate on truly unseen data? In particular, how should a user interpret a computed dispersion value to make an actionable inference about expected accuracy?\n\nQ5: For Section 3.1, have you tested whether the dispersion–accuracy relationship you observe on ARC/MMLU for one model persists on a different dataset drawn from a different domain (i.e., under distribution shift)? If so, please report it. If not, please clarify that the current result is model- and dataset-specific, and should not yet be treated as a universal performance predictor.\n\nQ6: In Section 3.2 (training with an auxiliary dispersion objective), can you clearly define all notation? In particular, $d$ appears with different meanings, and $\\mathcal{L}_{\\mathrm{CE}}$ and $\\mathcal{L}_{\\mathrm{aux}}$ are referenced without a full specification. Please also explain how this auxiliary objective relates to prior work on contrastive/repulsive representation learning, so that readers do not interpret the loss as entirely new.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper takes long-standing intuitions about representation geometry and turns them into actionable tools. The proposed dispersion metric is inexpensive to compute and is presented as useful for several practical tasks (checkpoint selection, layer selection, etc.). The authors also provide fairly broad empirical evidence across multiple model families and domains to support these claims.", "weaknesses": "## Weakness 1. \"Predicting downstream performance without labeled data\" is oversold\nSection 3.1 claims to predict downstream performance without labels. What the method actually provides is a relative hardness ranking: low-dispersion examples are more likely to be wrong than high-dispersion examples. Given a new dataset with zero labels, computing dispersion yields only that ordering, not an absolute expected accuracy. The paper does not discuss how dispersion on a new task would be mapped to accuracy, and it does not test whether the dispersion–accuracy monotonicity continues to hold under distribution shift. Moreover, the monotone relationship is clearly tied to a specific model; for example, in Figure 8, if the computed mean pairwise distance is 0.08, it suggests a high correct rate for the 1B model but a lower correct rate for the 3B model.\n\n## Weakness 2. The usefulness for model selection is overstated. \n\nSection 3.2 only demonstrates results within very specific model families. The paper never shows that the same metric can compare across unrelated families (e.g. Qwen vs Llama), nor does it explicitly warn against that use. Even within one family, monotonicity with skill is not clean across scales: Distill-Qwen-14B scores higher on MATH than Distill-Qwen-1.5B, yet it has lower digit–digit and digit–non-math dispersion. Therefore, the claimed usefulness for model selection breaks when you move across parameter sizes. This suggests dispersion is mainly a heuristic for ranking sibling checkpoints of similar size, not a universal capability score. Without clearly stating these limitations, the paper may lead readers to overgeneralize.\n\nEven within one family and one parameter size, the usefulness remains in doubt. For example, Qwen2.5-Math-7B has D–D equal to 0.27 with a MATH score of 55.4%, while Distill-Qwen-7B has D–D 0.28 with a MATH score of 92.8%. It is unclear whether 0.28 is meaningfully larger than 0.27, because the paper does not provide standard errors for these summaries; and if 0.27 and 0.28 are effectively similar, then the large change in performance actually suggests that the D–D gap is not useful for predicting performance, which contradicts the paper's claim. \n\n## Weakness 3. Unclear benefits over existing alternatives\n\nThe paper claims that dispersion can rank models for a domain and identify hard inputs without labels, but perplexity on unlabeled domain text already gives a label-free measure of adaptation. The paper does not compare dispersion against that baseline, so we cannot tell whether dispersion provides genuinely new information or is just a rephrasing of model confidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)