Less is More: Local Intrinsic Dimensions of Contextual Language Models
Authors: Benjamin Matthias Ruppik
Julius von Rohrscheidt
Carel van Niekerk
Michael Heck
Renato Vukovic
Shutong Feng
Hsien-chin Lin
Nurul Lubis
Bastian Rieck
Marcus Zibrowius
Milica Gasic
Venue: NeurIPS 2025 poster ()
Keywords: latent space geometry
contextual embeddings
LLMs
intrinsic dimension
training dynamics
generalization
overfitting detection
grokking
Primary area: general_machine_learning
TL;DR: Lower local intrinsic embedding dimension signals better performance, detecting when LLMs improve, overfit, or grok.

Abstract: Understanding the internal mechanisms of large language models (LLMs) remains a challenging and complex endeavor. 
Even fundamental questions, such as how fine-tuning affects model behavior, often require extensive empirical evaluation. 
In this paper, we introduce a novel perspective based on the geometric properties of contextual latent embeddings to study the effects of training and fine-tuning. 
To that end, we measure the local dimensions of a contextual language model's latent space and analyze their shifts during training and fine-tuning.
We show that the local dimensions provide insights into the model's training dynamics and generalization ability.
Specifically, the mean of the local dimensions predicts when the model’s training capabilities are exhausted, as exemplified in a dialogue state tracking task, overfitting, as demonstrated in an emotion recognition task, and grokking, as illustrated with an arithmetic task.
Furthermore, our experiments suggest a practical heuristic: reductions in the mean local dimension tend to accompany and predict subsequent performance gains.
Through this exploration, we aim to provide practitioners with a deeper understanding of the implications of fine-tuning on embedding spaces, facilitating informed decisions when configuring models for specific applications. 
The results of this work contribute to the ongoing discourse on the interpretability, adaptability, and generalizability of LLMs by bridging the gap between intrinsic model mechanisms and geometric properties in the respective embeddings.

Less is More: Local Intrinsic Dimensions of Contextual Language Models
Benjamin Matthias Ruppik' Julius von Rohrscheidt Carel van Niekerk' Michael Heck'
Renato Vukovic* Shutong Feng' Hsien-chin Lin'
Nurul Lubis*
Bastian Rieck2.4 Marcus Zibrowius" Milica Gašić
Faculty of Mathematics and Natural Sciences, Heinrich Heine University Düsseldorf, Germany
2 Institute of Al for Health, Helmholtz Munich, Germany
3 Technical University of Munich, Germany
University of Fribourg, Switzerland fruppik, miekerk, bilekerr,,eckmm,rrereeeke linh, Lubis,,arcus www.cus.zibrowiiii..... wwbroriis. marcus
julius. rohrrschhiitthhhelmmollt---mmmmd de
bastian, www.ssenbaccerrrriifrrccc
Abstract
Understanding the internal mechanisms of large language models (LLMs) remains a challenging and complex endeavor. Even fundamental questions, such as how fine-tuning affects model behavior, often require extensive empirical evaluation. In this paper, we introduce a novel perspective based on the geometric properties contextual latent embeddings study the effects of training and fine-tuning. To that end, we measure the local dimensions a contextual language model's latent space and analyze their shifts during training and fine-tuning. We show that the local dimensions provide insights into the model's training dynamics and generalization ability. Specifically, the mean of the local dimensions predicts when the model's s training capabilities are exhausted, as exemplified in a dialogue state tracking task, overfitting, as demonstrated in an emotion recognition task, and grokking, as illustrated with arithmetic task. Furthermore, our experiments suggest a practical heuristie: reductions in the mean local dimension tend accompany and predict subsequent performance gains. Through this exploration, we aim to provide practitioners with a deeper understanding the implications of fine-tuning embedding spaces, facilitating informed decisions when configuring models for specific applications. The results of this work contribute the ongoing discourse on the interpretability, adaptability, and generalizability of LLMs by bridging the gap between intrinsic model mechanisms and geometric properties in embeddings.
Introduction
Large language models (LLMs) have transformed natural language processing in recent years, achieving impressive performance across a variety of tasks (Brown et al., 2020 Devlin et al. 2019 Tiang al., 2023 Radford et al. 2018 Touvron et al.) 2023). These models learn contextual token embeddings in high-dimensional latent spaces, whose structure governs how information is represented and processed. However, most performance diagnosties in LLMs rely on supervised validation or task-specific probes, with few attempting to understand the geometry of the LLM embedding spaces. Our paper motivated by central question: Can structural changes in the embedding space yield unsupervised insights into model behavior across language modeling tasks? We address this question by applying localized version of the TwoNN estimator (Facco al.) 2017) to quantify the local intrinsic dimension of contextual token embeddings. While the ambient
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

dimension of these embeddings is typically large (ranging from hundreds to thousands of dimensions), our observations show that the local intrinsic dimension reflects a owerr-dimensional manifold structure that varies across data space regions. This heterogeneity allows us to derive model- and data- specific geometric signatures without relying on labeled validation data or task-specific supervision.
Contributions We present a unified framework for analyzing LLM training dynamics through dimensionality. Based on this framework, we (i) demonstrate that fine-tuning reshapes the local intrinsic dimension in a dataset-specific manner, allowing us to infer overlap between training data used for fine-tuning with validation (or test) data from geometric shifts alone, without requiring labels; (ii) show that dimensionality anticipates the onset of grokking in synthetic arithmetic tasks, identifying generalization beyond training data from the dimensionality of the model's 's embeddings of the training data alone; (iii) demonstrate that local intrinsic dimensions can indicate training convergence, as evidenced in a waageeeneetaginn---eee -ased dialogue state tracking task where stabilizing dimension correlates with model stabilization; and (iv) show that local intrinsic dimensions detect overfitting in sequence classification setting, where an initial drop followed by a rise in dimension reflects the model's 's tradeoff between generalization and memorization. In total, our results suggest that local intrinsic dimensions serve as a valuable unsupervised signal for practitioners seeking to interpret and monitor LLM behavior. Beyond offering empirical insights, our work thus highlights the potential of geometrie descriptors to complement traditional evaluation methods and potentially serve to inform future model design
2 Related Work
Our work lies the intersection of geometry-aware language model analysis and dataset-level representation diagnostics. While prior efforts focus on the intrinsic dimensions of single sequences or model parameters, our contribution is quantifying local intrinsic dimensions across datasets and training stages, enabling unsupervised inspection of generalization dynamics and fine-tuning effects.
Notions of intrinsic dimensions of language models In first qualitative analysis of the internal representations of transformer models, [Ethyaarjh] (2019) and Cai et al. (2021) identify clusters and low-dimensional manifold structures contextual embedding spaces. Recent work has started studying relationships between geometric properties of LLMs and corresponding semantics. In Tulchinskii et al. (2023), the authors discover that the average intrinsic global dimension of artificially generated texts is lower than that of human-written texts. In that work, the points in the latent space are taken from a single text paragraph, so their work applies to single data input sequences, not to entire datasets like our approach. Valeriani et al. (2023) investigate how the global intrinsic dimension estimated is altered as data is passed through an LLM. Aghajanyan et al. (202T) define a notion of intrinsic dimension based restricting the model's parameter space and its effect on the objective function, and show that larger L.MSs tend to have smaller intrinsic dimensionality. That work studies the dimensionality of the model's parameter space, and not the latent space created from specific datasets. In Viswanathan et al. (2025), the authors analyze token-level intrinsic dimensions, linking token geometry to model next-token- preeiiciiiooo loss. The difference between their method and ours is that they always consider the contextual token embeddings of single, sufficiently long prompt as making up the embedding space. In contrast, we sub-sample from an entire dataset split. Lee et al. (2025a) identify structural similarities between the token embedding spaces of different language models. particularly reporting correlations in local intrinsic dimension measures. However, their analysis is limited to the non-contextual token embedding and unembedding layers, without considering contextual representations.
Topology-based analysis of models The topological descriptors in Kushnareva al. (2021) }; Lee et al. (2025b); Perez and Reinauer (2022); Tulchinskii et al. (2023) correspond to entire sequences defined the token level. In Durrani et al. (202)), of input data, while our proposed estimates pre-trained and fine-tuned models are compared using hicrarchical clustering and alignment functions. The results indicate that the latent space in the higher layers adapts to task-specific concepts, while the lower layers preserve the general concepts learned in the pre-trained model. [Ed-dib et al.) (2024) introduces a method for fine-tuning large language models by dynamically adjusting LoRA
and 'Our code is available at https:t://// //ggtth////tttt com/ aad/s lab/Topo_LLMy httpppbblc...naatss///g/ig.... https://github
hon/aidos-ae 1ab/ wgrgggokkiin-io via-vidi 1id

---

ranks based on the intrinsic dimensionality of hidden state representations. Here, the intrinsic dimension is measured as the rank of an information matrix, and not from the data space in our work. In Chang et al. (022), the representational geometry of multilingual language models is explored, focusing on how they balance encoding language-spectt
and language-agnostic features within shared multilingual space. [Ruppik et al. (2024) utilize topological features derived from neighborhoods in a contextual embedding space to improve performance task. The definition of their features requires fixed ambient data corpus, whereas our methods give an intrinsic measure of the dataset under consideration; thus, our measures are comparable between different model checkpoints of the feature generation model. Both works are situated in the nascent field of topological deep learning (Papamarkou et al. 2024), employing concepts from geometry and topology in observational manner to study machine learning models. In contrast to mechanistic interpretability (see, for example, the survey by Ferrando et al. 2024), in which one tries explain how language model's internal representations fit together into higher-level abstractions, our approach more atomic. We study the smallest non-divisible representation in the model (the contextual embedding of single tokens) and the space created by collections of these.
a sequence tagging
3 Methods
3.1 Large Language Models
Modern contextual language models typically either masked language models (MLMs) (Devlin et al. 2019) et al. (019) or autoregressive language models (ALMs) (Radford et al. 2019). MLMs predict masked tokens based on bidirectional context, while ALMs generate tokens sequentially, conditioned on preceding tokens. In both cases, the representation of token in context is given by vector each model layer, enabling geometric analysis of data corpus on layer-wise basis. We now give an overview of this construction; the whole procedure is summarized in Algorithm
3.2 Latent Space Modeling
Let D (sp, .1,...; sD) be text corpus and M language model (MLM ALM) of depth with tokenizer T. Each sequence 8m yields tokenized input:
T(sm) (t5t pp
(1)
with corresponding contextualized token embeddings at layer i:
M;(s..) (t.,("""........""
(2)
We distinguish between regular embeddings (tokens embedded as-is during a forward-pass) and masked embeddings (specific tokens replaced by [MASK], requiring the model to infer their representa- tions from the surrounding context). Despite the simplified notation, embeddings are context-sensitive The token embedding point cloud at layer is:
T = (M. O=......D; jEIm
with Im denoting token indices in the m-th sequence. Distances between points measured in the ambient Euclidean space. In practice, T, can contain millions of vectors, making subsequent neighborhood computation on the full dataset infeasible.
To obtain a representative subset of tokens T, we take a two-step sampling approach: We first sample M sequences from D, and after de-duplication, we sample N vectors from the resulting token embedding space. Finally, we compute neighborhoods (ty; TTy for each token using a locality parameter L. We assume the dataset is drawn from underlying text-generating distribution, and our sampling approximates its geometric structure. We confirm this assumption in our sensitivity analysis in Appendix A
3.3 Comparing Latent Spaces Across Models
We compare base model M with its fine-tuned variant MA trained on a corpus A. Since both models share the same architecture and tokenizer, there exists a canonical bijection between M,(D) and M (D), mapping each token's representation in the base model to counterpart in the fine- tuned model. More generally, such bijection arises whenever the underlying model architectures
3

---

Algorithm 1:
COMPUTE LOCAL DIMENSION ESTIMATES
Input:
Text corpus D;
Embedding model M (with corresponding tokenizer 7);
Dimension estimator dim (default dim = TwoNN);
Three parameters:
Size of text corpus sequence sub-sample M E N;
Size of token sub-sample N E N;
Local neighborhood size L E N.
Output: Token-level local estimates given as vector in IR i.e., one non-negative number for each of the sub-sampled tokens.
Take a random sub-sample D of size M of the dataset sequences;
2 Compute embeddings for each token in each sequence of D..
De-duplicate the embedding vectors;
Take token vector sub-sample of size N, call the resulting set token vectors T, for all E T do
Compute the L nearest neighbors of t, in T, resulting in the neighborhood MZ (t); Compute the local estimate dim(N2 (t;; Rz0 of this neighborhood;
return The vector of token-level dimension estimates.
and tokenizers are identical. After identical sub-sampling, this allows for point-wise comparison of the resulting latent spaces to evaluate geometric changes. Next, we define the local estimates used for this comparison.
3.4 Local Dimension Estimates
The TwoNN estimator (Fcco et al. 2017) approximates the intrinsic dimension of point cloud as a positive real number using only the distances 71 and r2 of each point to its nearest and second- nearest neighbors. It turns out that under weak assumptions, the ratio y1 Pareto-distributed (see Denti et al. (2021) for details). Then, the intrinsic dimension from parameters describing this distribution, as long as the density around the points is locally approximately constant on the scale defined by the distance of the second nearest neighbor.
of these distances is
be estimated
We turn this into local measure as follows: Let X be a point cloud and v E X. For L E N we define the L-local TwoNN estimate of X at as TwoNN(Ny, X)), i.e., as the TwoNN estimate of the local neighborhood (u; X) of v of size in X. In this way, the parameter L controls the locality scale of the dimension estimation. In the context of our model M and corpus D, we take X T and compute a vector of positive real-valued numbers TwoNN(NV. X))=eT E R. one for each token 0 E T. We subsequently aggregate this to mean estimate per corpus/model setup. While sensitive to hyperparameters such as J. and sample sizes, we show in Appendix A that estimates remain stable under reasonable settings, enabling consistent comparisons across datasets and models.
Experiments
We now apply our method to analyze how embedding spaces evolve during L.LM-related learning tasks. Unless stated otherwise, all results pertain to the model's final hidden layer, i.e., T- as defined in [Section 3 We refer the reader to Appendix C.2 for results on layers other than the last. Subsequently, we will focus on four central questions:
(QI) How does fine-tuning on different datasets alter latent space geometry?
(Q2) How can local dimension estimates detect grokking?
(Q3) How can local dimension estimates detect the limit of training capabilities?
(Q4) How can local dimension estimates detect overfitting?

---

4.1 Fine-Tuning Induces Dataset-Specific Shifts in Heterogeneous Local Dimensions
To understand how fine-tuning alters model representations, we investigate the distribution of local intrinsic dimensions across token embeddings. These dimensions, estimated using the method introduced in Section reveal nuanced changes the geometry of the embedding space.
Setup We evaluate models on datasets with varying domains and familiarity the base and fine-tuned models: MultiW0Z2.1 (Eric et al. 2020): Human-human multi-domain dialogues; Schema-Guided Dialogue SGD (Rastogi et al. 2020): Human-virtual assistant dialogues; Reddit:
Nearm 22, Nedinn Si5-
Hearwe 57; Medines 36, 2
TwoNN
O 5 15
RoBERTa RoBERIa fine-tuned MultwOO
Reddit comments from the year 2022 mentioning Tesla, Inc.; ICLR 2024 Submissions: Titles and ab- stracts of ICLR 2024 papers collected by and Wikipedia: The Hugging Face wiki text-103-vv corpus.
(a) TwoNN estimates on MultiWOZ validation
40
Mauae 1.55 Mediurel1 Sidm4877
Mean 18 09: Median 17 97405:954---1 S:1-5112
TwoNN
30
20
As can be seen from the selection of pre-training datasets of the respective models (Liu et al. [019
10-
RoBERTa ReBERTa
MultiWOZ
Radford et al. 2019). this choice allows to com- pare: (i) distributions not seen during pre-training or fine-tuning (e.g., Reddit, ICLR 2024, which were released after the pre-training of the model con- cluded), (ii) distributions seen during pre-training (Wikipedia), and (iii) distributions used for fine- tuning (e.g., MultiWOZ, SGD). This contrast allows us probe how local intrinsic dimensions behave in both seen and unseen data regimes. We split Reddit and Wikipedia into training (80%), valida- tion (10%), and test (10%) subsets. Chronologically- ordered datasets (e.g.. Reddit) t) are shuffled before splitting to avoid temporal bias. Wikipedia was pre- precprocesed by removing headings, empty lines, and stripping leading and trailing whitespace. Fur- ther dataset statistics are available in Table 1
(b) TwoNN estimates on Wikipedia validation
Maan-13.2 Medis-13 3..8 Sitw
30
Meac 12 80 Medien 12.ainn112..00 12.58 s.m
TooNN
20
1o
RoBERTa RoBERTa fine-tured Multwozz
(c) TwoNN estimates on Reddit validation
Figure I: Comparison of local intrinsic dimen- sions (LIDs) across three data modalities. The distribution of the local estimates is shown in the violin plot, together with their means and quartiles. The LID) of embeddings originating from the fine-tuning distribution (Mul (WOZ) differs markedly between models, whereas the LIDs for the out-of-distribution corpora (Wikipedia, Reddit) are almost in- distinguishable.
tokens
Fine-tuning of the RoBERTa- models (Liu et al.) 2019) is performed using masked language model- ing with masking probability of 0.15. Each model trained for epochs 10 000 training examples using a batch size of 8, a learning rate peaking at 5- 101 with 500 warmup steps, and linear decay thereafter. Weight decay of 0.01 is applied through- out. For evaluation, we embed a subset of the validation split of each dataset using both the base and fine-tuned model, ensuring the embeddings are out-of-sample relative to the fine-tuning set.
Our discussion this section focuses on the masked language model RoBERTa, whose architectures remain widely used in practice where encoder-based models are needed (Warner et al. 2025) and autoregressive alternatives are less suitable, such as in information retrieval. Results for additional fine-tuning of the autoregressive models GPT-2-medium can be found in Appendix C.T where similar observations hold.
and Llama-3.1-8
Results We find that local intrinsic dimensions vary significantly across tokens, as illustrated by the wide spread the TwoNN distributions in In/Fiuure I Here, we choose the parameters M 7000, N = 60 000, L = 128. This heterogeneity is consistent across different models and datasets. It aligns with prior observations that numerical data often not confined to a single manifold of uniform dimensionality. but is instead composed of multiple regions with varying local geometry (Brown) ett al. 2023). These findings reinforce our decision to favor local over global estimates of embedding dimensionality.
5

---

37.5
Thiihng Pacton
0.30
0:35
0.
0.15
0.40
0:30
201
06
1002
20100
22000
40030
50302
02000
10000
20002
35000
43000
50002
50322
30030
20002
30000
40020
50002
60002
Stop
Soas
Soip
(a) Training accuracy
(b) Validation accuracy
(c) Training mean local estimates
Figure 2: Training a model on addition mod p 197 with different training data fraction selected from 10.1; 0.15; 0.2; 0.25; 0.3; 0.4; 0.5]. The plots show the development for 60 000 batches, with mean and 95% confidence interval over 5 training seeds per configuration (plots per seed are in Appendix C.3 The mean local estimates are computed the training split for the parameters 3000; L 64. Dashed lines highlight the runs where grokking did not occur.
Our central observation is that fine-tuning systematically lowers the local intrinsic dimension only on the dataset used during the fine-tuning process. As shown in Figure Th, the TwoNN estimates for the Woltii00-vvaidaation embeddings are markedly lower for the fine-tuned RoBERTa model than for the base model. By contrast, embeddings of unrelated datasets-such as Wikipedia (Figure Ip) or Reddit comments (Figure -h)-show no only minimal changes in their local dimension distributions. Quantitatively, this observation can be validated by the standardized mean difference between the respective cohorts, with value of 1.19 for MultiWOZ, and 0.08 (resp. 0.1) for Reddit (resp. Wikipedia).
We find this shift to be stable across training, validation, and test splits of the dataset, suggesting that it reflects robust structural changes in the latent space rather than artifacts (see Apppendix A for thorough sensitivity analysis showing the method's robustness the choice of sampling parameters). This behavior highlights strong dataset-speciffcity in how local geometry adapts under fine-tuning. The effect is absent when evaluating on out-of-distribution data, supporting the view that local dimension reductions correspond to improved fit and specialization on the fine-tuned task, while leaving unrelated regions of the embedding space unchanged.
4.2 Local Dimensions Detect Grokking
Grokking is the phenomenon that a machine learning model acquires specific skills only after extended training far beyond overfitting on the training set, which was first discovered by Power al. (2022). In such cases, from the model's is performance on the training set alone, it is a complex problem to predict under which choices of hyperparameters grokking will occur (Junior al. 2024).
Setup We here consider the task of learning arithmetic operation on small group (addition modulo p). For example, for p = 197 such an input sequence would be [155, 'o', 88, '='], where the model should predict the result 46. The tokenization is constructed that every operand in the group is encoded as a single token, with the addition of the operation token 'o" and the equality token ? A certain fraction of all valid expressions in the group, ranging from 10 percent to 50 percent, is taken as the training set, and the remainder will be taken as the validation set.
We use tiny decoder-only transformer model with two layers, 128 hidden dimensions, and four attention heads trained from scratch. Optimization is performed by AdamW (Loshchilov and Hutter 2017) with learning rate 0.001 (with linear schedule over 400k steps, warmup of 10 steps), batch size 512, weight decay 0.01. Models are trained for many steps beyond the point where training performance saturates, keeping the same setup in the original paper (Power al. 2022).
For the local estimates, we the number of sampled tokens N as the minimum of 3000 and the maximum number of tokens available, and a neighborhood size L of 64. Since the number of input sequences comparatively small, we choose M to encompass everything, and only subsample in the token selection step. Still, we observe similar qualitative behavior for other reasonable choices of these hyperparameters. The hidden states at the last layer are sampled from all available tokens, which could come from the operands and the operation equality symbol.

---

Results Performance measures versus local estimates are shown in Figure 2 for the first 60 000 training steps averaged over seeds (we present plots for individual seeds in Figure Tôlin the appendix). The training accuracy quickly reaches an almost perfect score in all cases, with the validation accuracy lagging behind. In the time frame under consideration, the model can generalize and reach almost perfect validation accuracy only in those runs for which the training data portion exceeds 20%. Based on the loss and accuracy of the training data alone, one would not be able to predict which configurations can break out and generalize to the validation data.
typical pattern for the mean local dimension computed on the training data in all these runs is that it increases in the first few thousand global steps (see Figure 2c). But subsequently, the training mean local dimension starts dropping significantly for those runs that exhibit grokking. Observe that the timing of this drop coincides with the start of the increasing validation accuracy in FFigure 26) We can conclude that this setting, a drop in the mean local dimension on the training set strongly indicates a successful generalization to unseen validation data.
For those runs in which the validation accuracy does not increase beyond the fraction of training data during the selected training time (with 10% and 15% of training data, highlighted via dashed lines), the local dimension increases and then stays mostly flat. Quantitatively, the Spearman rank correlation between the training/validation /al/ddatiinnnnnoo accuracy and mean local dimension measured over the training split positive for those training fractions where the model does not grok (0.880/0 38000.076 and 0.922/0 00/2/0.220 for training fractions 10% and 15%). This behavior suggests that the model only tries to learn the training examples by heart and fails to generalize to unseen data. For all runs grouped by training fraction 2 20%, which all exhibit grokking, the Spearman rank correlation between both training/validation accuracy and training mean local dimension negative.
4.3 Local Dimensions Detect Exhaustion of Training Capabilities
Finding relevant text segments in response to query, also called span prediction or sequence lagging, is a highly relevant problem in various natural language processing settings (Jurafsky and Martin 2025). One such application is retrieval augmented generation (RAG), where, during the information retricval step, relevant passages from a large corpus need to be selected (Fan et al.] 2024) before generating answer. Here we study it in the context of dialogue state tracking a critical task of dialogue modeling.
Setup To demonstrate the application of our local dimension estimates in this setting, we compute them for the soguence-tagging-basee dialogue state tracking model TripPy-R (Heck et al. 2022) trained on the MultiwOz2. 1 dataset (Eric et al. 2020). In dialogue state tracking. the task is to predict the user's intent from the natural language input utterances, and keep track of the user's s goal described throughout the conversation by updating the dialogue state Young al. 2010). The contextualized hidden states for computing our measure are taken from the last layer of the unified encoder component, which has the RoBERTa architecture. The actual state tracking is performed based on classifier outputs derived from this encoder's 's output. The encoder is fine-tuned during the TripPy-R training via the loss signal derived from predicting the current dialogue turn's 's state update. We train the models for 20 epochs with Adam, with a linear learning rate schedule up to 5 10 that starts with one warm-up epoch.
The input data for the local dimension estimates is formatted in the same way as it is used during training and inference. single input sequence consists of dialogue turn, followed by the dialogue history, where the different sequence components are separated via special tokens. For the local dimension estimates, we sample M = 7000 sequences from the training, validation, and test split, with token subsample size N 60 000, and neighborhood size L = 128.
Results Figure shows the development of two model performance measures: One is the differen- tiable model loss, which is used for backpropagation. The other is the non-differentiable joint- goal accuracy (JGA), which is used to judge the downstream performance of the state tracking model.
Note that the mean local dimension estimates behave similarly when comparing the training, valida- tion, and test splits. This is in contrast to the loss and JGA: On the training set, the loss (green) is monotonically decreasing, while the JGA (orange) is increasing. On the other hand, the loss reaches a minimum on the validation set after 7500 batches, long before the JGA converges. In this case, the differentiable model validation loss would give the wrong impression that the model has finished
7

---

Rean oealdrr jost Coul Acomncy
Dustetton mHAKIO
Mainbcal
Stoga
Stopa
Stopa
MultiWOZ Training split
(b) MultiwoZ Validation split
(c) MultiwOZ Test split
Figure 3: Development of TripPy-R R performance measures (model loss in green; joint goal accuracy orange) compared with mean local dimension estimates (blue) evaluated on the training, validation, and test split of the MultiWOZ dataset. We show the mean and standard deviation of the measures evaluated the end of each epoch over six different model training seeds.
learning generalizable representations. However, from the dimension estimate, we conclude that the model has not converged at that point. The mean local dimension is still decreasing, and one should continue training. Notably, we can make this observation on the training data alone and would not need a separate labeled validation set when relying upon our estimates.
Quantitatively, taken over the whole time series progression and step-wise averaged over six seeds, the mean local dimension exhibits a strong negative Spearman rank correlation with the JGA of 2.9822/0.958/-000 for the training/validationntes split, respectively. This confirms that pro- gressive compression of the embedding space directly reflects improvements in generalizable task performance. Correlation between the mean local dimension and the loss is positive (0.982) on the training set, and negative 100.612/00.733 on the validation and test set, mirroring the fact that the loss does not align consistently with JGA performance over the different splits.
In another direction, we can see that later in the training, after roughly 25 000 batches, the stabilization of the local dimension estimates coincides with the convergence of the model performance regarding IGA on the validation and test sets. This effect of stabilization of the local estimates, indicating that the training capabilities have been exhausted, be seen more clearly for longer training Appendix C.4
(50 epochs) in Figure 17
4.4 Local Dimensions Detect Overfitting
We now examine the mean local dimension as an indicator of overfitting in a challenging sequence classification setting.
Setup use the EmoWDZ dataset (Feng et al. 2022), with the task of classifying dialogue utterances into one of seven emotion classes. Our experiments employ the ERToD model (Feng et 2023) in its baseline configuration, linear classifier built upon a part-baas-uncaaee 2019) model feature extractor. We intentionally decide the setup without dialogue history, data augmentation, or auxiliary predictions, which are modifications proposed by the authors to solve the problem of classifying rare emotions. The model is trained for 8 epochs using linear learning rate schedule with warmup for Adam W and peak learning rate of 2 10-5, At the end of each training epoch, local dimension estimates layer. Thus, this is the latent space that the model has tuned to aid its subsequent classification. We select representative subsets of M 7000 sequences from the training, validation, and test split, with a token subsample size = 60 000, and neighborhood size L = 128. Model performance is evaluated using cross-entropy loss and weighted F1 and macro F1 classification scores. To better capture effects on minority classes, the majority "neutral" class is excluded from the F1-scores.
(Devlin al.)
computed from hidden states extracted at the last BERT-encoder
Figure 4/depicts the model's performance measures plotted against the mean local estimates. After the first training epoch, the model's 's mean local dimension drops notably to ~ 7.25 from the base model's initial dimension As 9.94. Over the subsequent eight training epochs, this dimension increases to value of about 8. Note that this behavior of the mean local estimates can be observed on all three data splits. As expected, the training loss monotonically decreases, while both weighted and macro Fl Fl-scores increase on the train set.
Results

---

Roun socooonn OnN SM
37
0.2
2.5
88
Calaatin
0.2
Spoch
Esoch
(a) EmoWOZ Training split
(b) EmoWOZ Validation split
(c) EmoWDZ Test split
Figure 4: Development of emotion recognition model performance measures (loss in green; weighted F1 in orange; macro F1 in red) compared with mean local dimension estimates (blue) evaluated on the training, validation, and test split of the EmowOZ dataset. We show the mean and standard deviation of the measures evaluated at the end of each epoch over four different training seeds.
However, the minimum of the local dimension, followed by an increase, suggests that the model has already found an efficient representation after a single training epoch. We previously observed a connection between rising dimension and memorization in the grokking experiments in Section 4.2 This suspicion is confirmed when considering the performance measures on the validation set in Figure 4b After the first epoch, the validation loss increases strongly over the following epochs; thus, the model is clearly overfilting on the training examples.
In terms of how this influences the classification accuracy, the effect is more nuanced. While the macro Fl-score increases and plateaus after about three epochs, the weighted Fi-score declines from the end of epoch two onward. This combined behavior implies that the model improves minority classes at the expense of performance on the majority classes, potentially memorizing rare instances and diminishing its generalization capability. This interplay between macro and weighted F1 suggests that the model is learning a decision boundary that trades off the performance of majority and minority classes, phenomenon that may be overlooked when looking the loss curve alone. Practitioners monitoring the shifts of local dimensionality on the training set could, therefore, identify subtler model behavior without requiring labeled validation or test data.
Note that in general, the evaluation criteria are task-specific: In contrast to the observations in Section 4.3 where the validation loss is not reliable stopping criterion when aiming for the target metric JGA, here, the validation loss and weighted Fl are tightly aligned. Hence, the loss serves valid stopping signal here. The local dimension measures support this point, since they negatively correlated with the training loss (Spearman rank -0.952), but positively correlated with the validation (0.952) and test loss (0.976).
4.5
Computational Resources
Fine-tuning of the RoBERTa-bais and GPT-2-medium models (in Appendix C.I on one of our selected datasets can be performed efficiently on single NVIDIA V100 or GTXI080TI-12GH GPU within few hours. Additionally, computing the embeddings for a single layer of these models requires only forward passes, which takes approximately 10 minutes on the same hardware.
For the TwoNN computations, which CPU-intensive, the computational requirements depend on the size of the dataset and the dimensionality of the embeddings. The computation is feasible in 20 minutes on an E5-2640v4 (Broadwell) 2.40GHz dual-core machine with 32GB of RAM using package (Bac et al.) 2021) for a typical dataset with tens of thousands of points in high-dimensional space (ambient dimension in the hundreds). For the grokking experiments, where we evaluate the local dimension frequently, this amounts to total of 48 CPU core hours per run in the depicted range. However, precise timing may vary depending on implementation optimizations and the chosen neighborhood sizes. For constant and small neighborhood size L and ambient embedding dimension d. the computation the neighborhoods for a subsample N using exact search (Johnson et al. 2019) takes (ddN2). Then, since we already have the ratios of closest distances available for each neighborhood, the TwoNN local dimension estimation is constant overhead of performing a linear fit values of transformed cumulative distribution.
the

---

4.6 Limitations
While versatile, our framework has several limitations. One drawback is its comparatively high computational complexity; to get reliable dimension estimate, we need to build sufficiently large sub-sample of the latent space by performing forward passes on the input data and subsequently computing the neighborhoods and estimates. Being independent of the model training procedure, these computations can be performed parallel to any potential fine-tuning, thus not slowing down the training process. Further work could focus on making the local estimate computation more efficient and developing local, stable, and computationally efficient descriptors.
An intrinsic limitation arises from the assumptions of our dimension estimates. The TwoNN estimate, which we apply locally, is known to accurately reflect the true local dimension only under strong assumptions on the local point sampling distribution: The point sample should come from a Poisson point process on a subspace of uniform dimension. Note that these assumptions and locally constant density for the hidden states of transformer models have been empirically demonstrated (Valeriani et al. 2023 Viswanathan et al. 2025). While the TwoNN method is non-parametric, the precise absolute value of the mean TwoNN estimate depends on the hyperparameter choices in selecting the latent space sub-sample and neighborhood size. This implies that our mean of local dimensions is of a "relative" nature, because we cannot directly compare the dimensions' values between different model architectures. However, this is only of secondary importance for our method, as we are more interested in changes of dimension than in absolute values. Moreover, our framework is general, permitting the use of other estimation methods. Our working hypothesis is that high-dimensionall neighborhoods reflect over- over- parameterizeed and poorly localized behavior, whereas very low LID indicates compressio or memorization. Between these extremes lies a "sweet spot" in LID, where representations retain the essential degrees of freedom required for effective generalization. Empirically, this intermediate range correlates with peak generalization performance. While this correlation supports the interpretability of our analysis, establishing causal link between LID) reduction and improved generalization remains an open question for future work.
5
Conclusion
We introduce a novel geometric perspective on LLM training dynamics by measuring the local intrinsic dimension of contextual token embeddings. While the latent spaces of contextual language models exhibit regional variation in dimensionality, the mean local dimension provides a stable, interpretable summary across dataset splits. Across diverse tasks, a sustained drop in mean local dimension reliably suggests improved generalization, offering a practical, unsupervised diagnostic signal that, among other things, enables the detection of grokking. Such a marked reduction in intrinsic dimension in supervised downstream tasks highlights their strong compressive effect on the latent space. On the other hand, rising dimensions can point to a tradeoff between generalization and specialization, typically observed during overfitting. Our approach enables monitoring without supervision by labels, which is particularly valuable in low-resource settings where validation labels are limited or unavailable. Thus, our findings highlight the utility of gcometric descriptors for monitoring and interpreting LLM behavior beyond traditional label-based metries.
To our knowledge, we are the first to investigate local dimensional shifts and their implications in training and fine-tuming contextual language models for different downstream tasks. The findings of this work enhance the ongoing discussion regarding the interpretability, adaptability, and generalizability of LLMs. Our work provides foundation for designing better model architectures and developing interventions that utilize the insight that lower intrinsic dimensions benefit machine learning problems. Our measures could inform parameter choices such LoRA- ranks in the spirit of (Ed-dib et al. 2024), and are broadly applicable for other (post)-training phases, for example, those which involve RL-tuning. While our current dimension estimation method is not differentiable (due to nearest-neighhor graph construction and the subsequent TwoNN estimator), we designing differentiable surrogate proxy loss to encourage local compression during training as an exciting opportunity for future work. The locality of the dimension estimates promises further applications beyond the scope of the current work. One immediate avenue for investigation is to see to what extent the dimension can be used to detect which data the model has been trained on. Another important direction would be to find connections between the underlying meaning that the tokens carry and their corresponding dimension estimates.
Impact and Future Work
10

---

Acknowledgments and Disclosure of Funding
This work was made possible through the support of the European Research Council (ERC) under the Horizon 2020 research and innovation program (Grant No. STG2018 804636), the Ministry of Culture and Science of North Rhine- WWesthhlla within the Lamarr Fellow Network, and the Swiss State Secretariat for Education, Research, and Innovation (SERT). Computational resources were provided by the Centre for Information and Media Technology at Heinrich Heine University Düsseldorf, and Google Cloud.
References
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A.,
Bao, J., Behl, H., Benhaim, A., Bilenko, M., Bjorck. J., Bubeck, S., Cai, M., Cai, Q., Chaudhary,
V., Chen, D., Chen, D., Chen, W., Chen, Y.-C., Chen, Y.-I.. Cheng, H., Chopra, P., Dai, X., Dixon, M., Fldan, R., Fragoso, V., Gao, J., Gao, M., Gao, M., Garg, A., Giorno,
D., Goswami, A., Gunasekar, S., Haider, E., Hao, J., Hewett, J., Hu, W., Huynh, J., Iter, D., Jacobs, S. A., Javaheripi, M., Jin, X., Karampatziakis, N., Kauffmann, P., Khademi, M., Kim, D., Kim, J., Kurilenko, L., Lee, J. R., Lee, Y. T., Li, Y., Li, Y., Liang, C.,, Liden, L., Lin, X., Lin, Z., Liu, C., Liu, L., Liu, M., Liu, W., Liu, X., Luo, C., Madan, P., Mahmoudzadeh, A., Majercak, D., Mazzola, M., Mendes, C. C. T., Mitra, A., Modi, H., Nguyen, A., Norick, B., Patra, B., Perez-Becken D., Portet, T., Pryzant, R., Qin, H., Radmilac. M., Ren, L., de Rosa, G., Rosset, C., Roy, S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang, N., Sharma, H., Shen, Y., Shukla, S., Song, X., Tamaka, M., Tupini, A., Vaddamanu, P., Wang, C., Wang, G., Wang, L., Wang, S., Wang, X., Wang, Y., Ward, R., Wen, W., Witte, P., Wu, H., Wu, X., Wyatt, M., Xiao, B., Xu, C., Xu,J. Xu, W., Xue, J., Yadav, S., Yang, F., Yang, J.. Yang. Y., Yang, Z., Yu, D., Yuan, L.. Zhang, C., Zhang, C., Zhang, J., Zhang, L. L.., Zhang, Y., Zhang, Y., Zhang, Y., and Zhou, X. (2024). Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint. 1422
Aghajanyan, A., Gupta, S., and Zettlemoyer. L. (2021). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuuing, In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). pages 7319-7328, Online. Association for Computational Linguistics. doi: 160::/ 1/2022 L.cl long. 568 arXiv:2012.13255
AI Meta (2024). Llama 3 Model Card. http:://ggtuu..ccommmaaaaa/////s main/MODEL CARD. md.
Bac, J. Mirkes, E. M., Gorban, A. N., Tyukin, I., and Zinovyev, A. (2021). Scikit- Python Package for Intrinsic Dimension Estimation. Entropy, 23(10).
Dimension:
doi: 10.339//223 101 368 1rXiv:2109.0255
Brown, B. C., Caterini, A. L., Ross, B. L., Cresswell, J. C., and
G. (2023). Verifying the Union of Manifolds Hypothesis for Image Data. The Eleventh International Conference on Learning Representations (ICLR). RhenReeview:ReeeeCC... a arXiv:2207.0286)
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu. J., Winter, C., Hesse, C., Chen, M.. Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language Models are Few-Shot Learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc. 100S
Cai. X.
Huang, J., Bian, Y., and Church, K. (2021). Isotropy in the Contextual Embedding Space: Clusters and Manifolds. In International Conference on Learning Representations (ICLR). Open Review:x YGNOS6OWDH
Chang, T., Tu, Z., and Bergen, B. (2022). The Geometry of Multilingual Language Model Rep- resentations. In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Proceedings of the 2022

---

Conference on Empirical Methods in Natural Language Processing, pages 119-136, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. doi: 160r110.16653//222222.mm main.9 ar arXiv:2205.1064
Denti, F., Doimo, D., Laio, A., and Mira, A. (2021). Distributional Results for Model- Intrinsic Dimension Estimators. arXiv preprint. UrXiv:2104.1383
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidi- rectional Transformers for Language Understanding. In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Pa- pers), pages 1171-4466, Minneapolis, Minnesota. Association for Computational Linguistics. 100.018853//1/N119111
1rXiv:1810.04550
Durrani, N., Sajjad, H., Dalvi, F., and Alam, F. (2022). On the Transformation of Latent Space in Fine- Tuned NLP Models. In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1495-1516 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. 10::100.166333/1222222.m
main.97 arXiv:2210.12699
Ed-dib, A., Datbayev, Z., and Aboussalah, A. M. (2024). GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning. arXiv preprint. arXiv:2412.092500
Bric, M., Goel, R., Paul, S.. Sethi, A., Agarwal, S., Gao. S., Kumar, A., Goyal, A., Ku, P., and Hakkani- Tur, D. (2020). MuliWOZ 2.1: A Consolidated Multi- DoatiiDDmminn Dialogue Dataset with State Corrections and State Tracking Baselines. In Calzolari, N., Béchet, F., Blache, P., Choukri. K. Cieri, C., Declerck. T. Goggi. S., Isahara, H., Maegaard. B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., and Piperidis, S., editors, Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 422-428, Marseille, France. European Language Resources Association. ACL arXiv:1907.01666
Anthology:0020.rree-
Ethayarajh, K. (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 55-55, Hong Kong, China. Association for Computational Linguistics. doi: 10.11..3// I/D 11--D199900000 1006 10Xx:11199.0012
Facco, E., d'Errico, M., Rodriguez, A., and Laio, A. (2017). Estimating the intrinsic di- mension of datasets by minimal neighborhood information. Scientific reports, 7(1):12140. doi: 000000..00///11199001-11117
Fan, W., Ding, Y., Ning, L., Wang. S., Li, H., Yin, D., Chua, T.-S., and Li, Q. (2024). A Survey on RAG Meeting LLMs: Towards Retrieval- Augmented Large Language Models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing, KDD '24, page 6491-6501, New York, NY, USA. Association for Computing Machinery. doi: 10::10..145// 1et.1..11//333722.6666 1470 MiXiv22405.06211
Feng, S., Lubis, N., Geishauser, C., Lin, H.-c., Heck, M., van Niekerk, C., and Gasic, M. (2022). EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems. In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk, J., and Piperidis, S., editors, Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4096- 4113, Marseille, France. European Language Resources Association. ACL Anthology::0222.ree 1.436 doi: ww..221/eeoooo..ofllo 6506504 21X1211994
Feng, S., Lubis, N., Ruppik, B., Geishauser, C., Heck, M., Lin, H.-c., van Niekerk, C., Vukovic, R., and Gasic, M. (2023). From Chatter to Matter: Addressing Critical Steps of Emotion Recognition Stoyanchev, S., Joty, S., Schlangen, D., Dusek, O.,
Learning in Task-oriented Dialogue. Kennington, C., and Alikhani, M., editors, Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 85-103, Prague, Czechia. Association for Computational Linguistics. doi: 18.13. http:////222..ss wiw.2222..sidiiall.... arXiv:230.. 12648
12

---

Ferrando, J., Sarti, G., Bisazza, A., and Costa-jussà, M. R. (2024). A Primer on the Inner Workings of Transformer-based Language Models. arXiv preprint. arXiv:2405.00208
Heck, M., Lubis, N., van Nickerk, C., Feng, S., Geishauser, C., Lin, H.-C., and Gašié, M. (2022). Ro- bust Dialogue State Tracking with Weak Supervision and Sparse Data. Transactions of the Associ- ation,for Computational Linguistics, 10:1 111-11999. 1oi:10.1162/tac a 00513 wwXiv:2202.0333.2
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. 1., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L., et al. (2023). Mistral 7B. arXiv preprint.
Johnson, J., Douze, M., and Jégou, H. (2019). Billion-scale similarity search with GPUs IEEE Transactions on Big Data, 7(3):535-547. doi: 10. T100//BDAAA.22112222255555 aiv: 702.08874
Junior, T. N. P., Zhou, H., Pezeshki, M., Rish, I., and Dumas, G. (2024). Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models. OpenRe
AUJSSShj arXivv:306.1325,
Jurafsky, D. and Martin, J. H. (2025). Speech and Language Processing: An Introduction Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. Online manuscript, 3rd edition. Online rellasseel August 24, 2025.
Kushnareva, L., Cherniavkkii D., Mikhailov, V., Artemova, E., Barannikov, S., Bernstein, A., Piontkovskaya, I., Piontkovski, D., and Burnaev, F. (2021). Artificial Text Detection via Exam- ining the Topology of Attention Maps. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 635-649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. 161:10.16633///2222..mm m0 arXiv:2109.04825
Lee, A., Weber, M., Viégas, F., and Wattenberg, M. (2025a). Shared Global and Local Geometry of Language Model Embeddings. In Second Conference on Language Modeling. OpenRe view:alDykpJAYF arXii225222
Lee, J. H., Jiralerspong. T., Yu, L.., Bengio, Y., and Cheng, E. (2025b). Geometric Signatures of Compositionality Across a Language Model's Lifetime. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T., editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5292-5320, Vienna, Austria. Association for Computational Linguistics.
265 4r.14:2410.01444
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint. Xiv: 1907.11699)
Loshchilov, I. and Hutter, F. (2017). Decoupled Weight Decay Regularization. arXiv preprint. 171.0:1711.05101
Papamarkou, T., Birdal, T., Bronstein, M., Carlsson, G., Curry, J.. Gao, Y., Hajij, M., Kwitt, R., Liò, P., Lorenzo, P. D., Maroulas, V., Miolane, N., Nasrin, F., Ramamurthy, K. N., Rieck, B., Scardapane, S., Schaub, M. T., Veličković, P., Wang, B., Wang, Y.. Wei, G.-W., and Zamzmi, G. (2024). Position: Topological deep learning is the new frontier for relational learning. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F., editors, Proceedings of the 4/st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 99529-39555. PMLR. Openkeview.NI3RG55XW. arXiv:2402.00871
Perez, I. and Reinauer, R. (2022). The Topological BERT: Transforming Attention into Topology for Natural Language Processing. arXiv preprint. ar Xivv:22006.51155
Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. arXiv preprint. MrXXiv2201.021177
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, (2018). Improving Language Under- standing by Generative Pre- Pre-Training OpenAl blog. Link
13

---

Radford, A., Wu, J., Child, R., Luan, D., Amodei. D., Sutskever, I,, et al. (2019). Language models are unsupervised multitask learners. OpenAl blog. Link
Rastogi, A., Zang. X., Sunkara, S., Gupta, R., and Khaitan, (2020). To-
wards Scalable Multi-Domain Conversational Agents: The Schema-Guides Dialogue
Dataset. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8689 8696.
doi: Not:10.1009/aaii.. v34i05, 6394 ar 1IXiv:1909.05855
Ruppik, B. M., Heck, M., Niekerk, C., Vukovic, R., Lin, H.-c., Feng, S., Zibrowius, M., and Gasic, M. (2024). Local Topology Measures of Contextual Language Model Latent Spaces with Applications Dialogue Term Extraction. In Kawahara, T., Demberg. V., Ultes, S., Inoue, K., Mehri, S., Howcroft, D., and Komatani, K., editors, Proceedings of the 25th Anmual Meeting of the Special Interest Group on Discourse and Dialogue, pages 344-356, Kyoto, Japan. Association for Computational Linguistics, doi: 10. 18653/v 1/223////002.siidddaaa
ar arXiv:248.0030 03706
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv preprint. arXiv:2302.13971
Tulchinskii, E., Kuznetsov, K., Kushnareva, L., Cherniavskii, D., Nikolenko, S., Burnaev, E., Baran- nikov, S., and Piontkovskaya, (2023). Intrinsic Dimension Estimation for Robust Detection of Al-Generated Texts. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems, volume 36, pages 39257-39276. Curran Associates, Inc. arXiv:2306.04723
Valeriani, L., Doimo, D., Cuturello, F., Laio, A., Ansuini, A., and Cazzaniga, A. (2023). The geometry of hidden representations of large transformer models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems, volume 36, pages 31234-51252 Curran Associates, Inc. arXiw:2302.00294
Viswanathan, K., Gardinazzi, Y., Panerai, G., Cazzaniga, A., and Biagetti, M. (2025). The Geometry of Tokens Internal Representations of Large Language Models. arXiv preprint. arXiv:2501. 105739
Warner, B., Chaffin, A., Clavié, B., Weller, O., Hallström, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., Adams, G. T., Howard, J., and Poli, I. (2025). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T., editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers). pages 2526-2547, Vienna, Austria. Association for Computational Linguistics. 10. 18653vv1/2225.cclle
127 arXiv:2412.1366
Young, S., Gašié, M., Keizer, S., Mairesse, F., Schatzmann, J., Thomson, B., and Yu, K. (2010). The hidden information state model: A practical framework for POMDP based spoken dialogue management. Computer Speech & Language, 24(2):150-174, 1o0.100.00166
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper has four main claims, which are listed the last paragraph of the Introduction Section Each claim corresponds to subsection of the Experiments
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the methods section, we mention limitations whenever they lead restric- tions in our setup, for instance, in Section 3.4, We additionally dedicate Section 4.6 to separate discussion of limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do not provide new theoretical results in this work. For those local dimension estimation methods that we apply, in Section 3.4 we clearly state references to the original publications that contain a detailed discussion of the assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our method is described in full detail in Section 3) and in Algorithm Sections 4.1 to 4.4 contain all the details necessary to reproduce the model training runs, which form the basis of our main findings. Moreover, our code is released part of the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Our code is submitted as part of the supplemental material and will be made public upon publication. All datasets are freely available under permissive licenses; more details can be found in the experiments in Section 4 and Appendix B
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
public//giidee/CodeSuuitti sow.soooobssssoooioooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips https///cccpllicgggeeeee/e/eeeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and test details, including hyperparameters and details about splits are described in the experiment setup in [Sections 4.1 to 4.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
17

---

Justification: To confirm the findings of our main results, we run every experimental setup for several seeds, and report confidence intervals or standard deviation in Figures 2 to All standard deviations in this paper are sample standard deviations, computed with the numpy (ddof=1) (ddoff())) and pd. Our sensitivity analysis in Appendix A] studies the effect of sampling size and locality parameters by reporting the results for multiple sub-sampling seeds per hyperparameter choice. The violin plots of the local estimates in Figure and Appendix C.I show the entire distribution and are reported together with mean and standard deviations.
and pandas library functions np.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss the runtime and computational resources in Section 4.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: We reviewed the NeurlPS Code of Ethics and confirm that we have followed every aspect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
18

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work foundational and provides general insights into the training dy- namics of language models. While we acknowledge the general risks involved in developing and deploying machine learning models, our insights not lead new negative societal implications. Since we aim to geometrically understand the internal mechanism in lan- guage model under different fine-tuning and training regimes, we hope that our work can help in safety and interpretability research in the future.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We believe foundational work does not pose such safety risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
19

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original research papers introducing the models used in Sections 3.1 and to 4.4 and acknowledge code that we used as the basis for our work in the submitted supplemental material. All datasets discussed freely available under permissive licenses, and we provide links to the dataset cards in Appendix B
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code, which is submitted as part of the supplemental material and will be made public upon publication, documented in the repository. This includes example hydra configuration files for the datasets and model selection. The Python package management system uv is used to reproduce the virtual environment with all dependencies, and run commands for the most important entry points into the codebase provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We employ LLMs as standard components of the machine learning models under consideration, and describe their usage in detail in Sections 3 and 4 Our development process did not involve LLMs ntn-standard way.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https://neurips.. for what should or should not be described.
21

---

Appendix (Supplementary Materials)
Sensitivity Analysis
22
A.1 Dependence on Hyperparameter Choices
22
A.2 Noise Analysis
26
A.3 Masked Token Embeddings versus Regular Token Embeddings
27
Additional Details on Datasets Used
28
Additional Experimental Results
28
Additional Fine- Tuning Results
28
Layer- Wise Computation of Local Estimates
29
Additional Grokking Results
29
Additional TripPy-R Dialogue State Tracking Results
A Sensitivity Analysis
The following amalysis gives insights into the stability of the sampling process in our pipeline of Algorithm I In particular, we discuss the influence of the following variations on our final mean local estimate:
In Appendix A.I
show that with increasing sample and neighborhood sizes, the TwoNN estimates tend to converge our setting, and already a small portion of samples suffices for reasonable reliability.
In Appendix A.2 we show that the mean local estimates are stable under applying noise to the embedding vectors.
In Appendix A.3 we show that for masked language models, both masked and regular token
embeddings lead to similar mean estimates.
All embeddings in the following analyses are computed using the regular token embeddings from the last layer of the RoBERTa-base model.
A.1 Dependence on Hyperparameter Choices
Dataset sequence sub-sampling (dependence M) Text corpora data splits usually contain different numbers of text sequences, since, for example, the training dataset is larger than the validation or test dataset. Here, we present experiments to show that our method can be made stable in the size of text corpus sequence sub-sample M E N, and how it depends on the seed of the sequence sub-sample.
Note that in our sub-sampling, we shuffle the dataset split using a given seed, and then truncate beginning segment of the specified size. Thus, when comparing different data sequence sub- sampling sizes for fixed seed, the smaller sequence sub-samples are subsets of the larger ones. In all the experiments in this section, the size of token sub-sample neighborhood size parameter L = 128. Note that for the smallest sequence sub-sample size M = 2000, enough non-padding tokens are necessarily available to reach token sample size of 60 000, but with more sequences, we get this common token space size.
N 60 000, and the local
We take sequence sub-samples of each given size with 5 different seeds and select random token sub-sample for each of these, on which the local TwoNN estimates are based. For the results, see Figure 5 5|and 6
For given sequence sample size, the average of the mean local estimates, calculated across different sampling seeds, remains consistent between the various dataset splits of the same dataset. This
22

---

observation indicates that the sampling process is robust to differences in how the data is divided. Furthermore, the estimates stabilize as the sequence sample size increases, suggesting convergence of the local mean estimates with larger samples. Based on these observations, we set M to the values 7000 or 10 000 for the natural language-hase -aaee tasks Sections 4.1 [43 a.4 because it represents practical compromise between computation effort and stability of the value at that scale. This value provides sufficient sampling density to capture the underlying data distribution comparably across datasets while maintaining computational efficiency.
10.00
9.75
Mean TWoMN Estinate
9:25
3.00
8.75
8:25
8:00
14000
Sequence Sample Size
(a) MultiWOZ Training split
10.00
Mean ToooNN Estimate
9.25
9.00
8.25
8:00
2000
8000
10000
12000
14000
Sequence Sample Size
(b) MultiWOZ Validation split
Moan TwoNN Estimate
9.50
9.25
9.00
8.50
8:25
8.00
2000
4000
6000
5000
10000
12000 Le0c0 16000
Sequence Sample Size
(c) MultiwOz Test split
Figure Boxplots of the mean local TwoNN estimates for different sequence sample sizes ranging from 2000 to 16 000. We compute the mean estimates for 5 different sequence sub-sampling seeds for the three splits of the MultiWOZ dataset. Here and all subsequent boxplots, the average of the mean estimates is depicted green triangles, their median is the orange line; outliers shown as circles.
Token sub-sampling (dependence on N) Here, we want to investigate the influence of the token sub-sample size on the local TwoNN estimate. We conduct our analysis on the MultiWOZ dataset. Since here are interested in the effect of the second sub-sampling process, want fix the first
23

---

15
off
Meann TwoMN Estimate
14.
10
4000
6000
воnо 10000
18000 20006 22006
Sequence Sample
(a) Reddit Training split
15
#
Ho
B
Mean TwoNN Estimate
14
10
8000 10000 2000 1s0ze 22000
Sequence e Sample Size
(b) Reddit Validation split
Mean TwoNN Estimate
34
19
10000 2000 18000 inize 20ine 220ne Sequence e Sample Size
(c) Reddit Test split
Figure Mean local estimates for different sequence sample sizes M for splits of the Reddit dataset, shown for 5 sampling seeds.
sampling step the sequences: We take the sub-sample the first M 10000 sequences from the MultiWOZ training, validation, and test split. From the resulting token embedding spaces, we then sample 100 000 points (ignoring padding and end-of-scquence (EOS) tokens). Finally, we compute the local estimates based the first N 10 10000, 20 000. ... . 90 000, 100 000 points in the given sub-sample. Note that we are left with slightly fewer points the local estimation step for the largest sub-sample size because of our de-duplication step in Algorithm This means that in a given token sub-sample, the beginning segment of the first 10 000 points is the same even when increasing the sample size, and compare the mean of the local estimates of this beginning segment.
The analysis of the datasets reveals a clear trend associated standard deviations as the sample size increases, see 7 Here, the mean stabilizes quickly, and the standard deviation comparatively small. This justifies our choice of N 000 in the natural language dataset settings.
behavior of the truncated mean values and their

---

9.56
9.45
Mean TwoNN Estimate
9.40
9.25
9.20
9.15
30000 600co sc000 cocon 70000 50000100000
Sample
(a) MultiWOZ Training split
9.50
9.45
Mean TwoNN Estinate
9,40
9.35
9.20
9:15
sc000 8ccop
Sample Size
(b) MultiWOZ Validation split
9.50
9.45
Mean TwioNN Estimate
9.40
9.35
9.30
9.20
9.15
3nice sodca
sccaa ondco100oon
Sample Size
(c) MultiwOZ Test split
Figure 7: Distribution of the mean local TwoNN estimates for different cardinalities of token sub- samples. We compute the mean estimates for 5 different token sub-sampling seeds and show their distribution for each fixed subsampling cardinality.
Neighborhood size for local estimates (dependence on T.) Having fixed sample sizes M and N, Figure 8 shows the distribution of TwoNN dimension estimates across varying locality seale parameters, represente by the number of neighbors L. A key observation is the relative stability of the estimates as the number of neighbors increases, with smaller locality scales (fewer neighbors) yiclding more diverse estimate distributions. This indicates that local properties of the data significantly influence dimension estimates.
The increasing stability with larger neighborhoods suggests that the embedding space exhibits more homogeneity at broader locality scales. However, the variability at smaller scales aligns with the observation in Brown et al. 2023) that in many cases, numerical data is not confined to single manifold of uniform dimensionality but rather consists of multiple manifolds or regions with varying local dimensions.
25

---

Our findings provide evidence that token embedding spaces of this nature do not conform to a single, unique dimension. Instead, they exhibit a complex structure, where local estimates diverge, reflecting the diverse geometric and topological properties within the space. This emphasizes the importance of considering locality when evaluating dimension estimates in such embedding spaces.
TwoNi Estimates
30
TwoNN Estlinates
60
40
10
20
16
32
64
128
256
512
1024
16
32
64
128
256
512
1024
Noi ghigs
Nei Neighbors
(a) Multiwoz Validation split
(b) Reddit Validation split
Figure Distribution of the local foken-wise TwoNN estimates over different locality scales (number of neighbors). The mean and quartiles of the resulting measure varying over the different tokens fixed subsample are shown here. The more global estimates, the smaller the standard deviation in the resulting distribution.
A.2
Noise Analysis
0.010
10.6
Global Bellmate
0.000
Nose Dictortion
Mean Dointwcise
10.4
0.008
Neas Distortion
9.5
000
10.2
0.006
9.4
10.0
9.3
0.004
9.8
0.004
0.002
0.082
Hausdorff Distance
Hausdorff Distance
(a) Global TwoNN estimate
(b) Mean of local TwoNN estimates
0.010
0.008
Sid Polttise
4.0
3.5
0.004
2.5
0.002
Hlusdort Distance
Standard deviation of local TwoNN esti-
mates
Figure 9: Effeet of artificial Gaussian noise masked token embeddings. On the æ-axis, mark the Hausdorff distance of the noisy embeddings to the clean embeddings, with color denoting the distortion in the artificial Gaussian noise.
The following experiments investigate the stability of the TwoNN dimension estimates under small perturbations of point clouds in a realistic setting of high-dimensional language model embedding spaces. The original work in Facco et al. (201)) studies the influence of artificial noise the
26

---

global TwoNN estimator for specific toy datasets (points sampled from 2- 2-dimensional plane in 20-dimensional space, and points sampled from 2-dimensional Gaussian distribution wrapped around a Swiss roll). Here, we confirm the stability under noise for our mean local estimates and in the realistic setting of high-dimensional contextual language model embeddings.
We analyze the effect of Gaussian noise on the embeddings from the MultiWOr test set using fixed set of parameters and varying noise distortions. Specifically, we use M 10 000 random sequences from the MultiWOZ test set and extract the masked token embeddings from the last layer of the RoBERTa base model, which produces 768- (68-dimensional vectors. To compute local estimates, we sub-sample N = 60 000 points from the embeddings and set the neighborhood size to L = 128.
Gaussian noise is applied to all dimensions of the embeddings with varying distortion parameters, E ; 0.002; 0.003; 0.004; 0.01), and multiple random seeds to introduce variability. For each noise level, we compute the approximate Hausdorff distance between the point cloud of the 60 000 clean embedding vectors and the 60 000 noisy embedding vectors, using the Fuclidean distance as the distance metric.
The results are visualized in Figure 9 where we plot the Hausdorff distance against three measures: the global TwoNN-estimace of the noisy point cloud, the mean of the local TwoNN estimates, and the standard deviation of the local TwoNN estimates.
From the plots, we observe the following trends. The Hausdorff distances between the clean and noisy embeddings and the global and the mean local TwoNN estimates increase with larger noise levels, and there is clear correlation between the two. Notably, for distortion parameter of = 0.01, the difference in the dimension estimates exceeds one. As expected, the standard deviation of the local TwoNN estimates also increases with higher noise levels, reflecting the increased variability introduced by the noise. Nevertheless, under small noise levels, the mean local estimate remains relatively stable and has lower variability than the global TwoNN estimate.
A.3 Masked Token Embeddings versus Regular Token Embeddings
The plots in Figure 10 compare the TwoNN estimates for regular tokens (left) and masked tokens (right) in the MultiwOZ dataset at the last layer. Each box plot represents the distribution of token-wise TwoNN estimates.
The estimates for regular tokens have an interquartile range from approximately 5.5 8, with median near 7. This distribution closely resembles that of masked tokens, which exhibit similar range and median. However, the broader spread for regular tokens suggests distinct token-level geometric behaviors influenced by their role in the learned representation space. In any case, this justifies the computation of the mean local estimates for the regular token embeddings, which are cheaper to obtain: Every sequence token must be masked individually retrieve masked embeddings with corresponding model forward pass. In contrast, for the regular embeddings, a single forward pass of the entire sequence automatically produces embeddings for every input token.
15.0
TwinNN estimates
12.5
10.0
2.5
60
masked
Figure 10: TwoNN estimates for regular and masked tokens in the MultiWOZ dataset at the last layer. Both token types show similar distributions.
27

---

B
Additional Details on Datasets Used
See lable for details on the dataset cards and corresponding split sizes.
Table 1: Dataset sizes for various datasets used in the experiments. Some datasets use pre-determined splits, while others involve random splits.
Dataset
Training Size
Validation Size
Test Size
Nultiw072.1
113 556
14 748
14 744
InoWUZ
66 474
8509
8634
https:
httpp::/paaarssattccadde http://prssatttaaaaacaaaaaaaaaaa
844994
https: http://giggaccccccc/aaaaatst//ta reseaacccheeeeecccccococo. datasets/
achema vchena_guided guided_ datc8
ICLA 2024
5764
721
720
submissions abstracis)
Reddit
174 996
21 874
21 874
frondom
80% 10%
split)
https:/ htspee//huggingffre
one one-year- year-errrr--- of of- tela- 1ecllll-o---- reddit
kikipedia
1 444 080
180 135
180 135
frandom
split)
attpe//huggingaace http:///....ttttteeeeeeeeetttaa.
vieser/sikitett. 103-v1
C Additional Experimental Results
C.1 Additional Fine-Tuning Results
Fine-tuning on unseen data Continuing the discussion in Section 4.1 Figure 11 we provide additional distribution plots of the local estimates for fine-tuned RoBERTa-base models. Here and in the following. the plots labeled with FT correspond embedding spaces created from the respective base model fine-tuned on the given dataset (MultiWOZ, Reddit, Wikipedia).
Fine-tuning autoregressive language model Moreover, in Figure 12 we show the distribution of local estimates for the auto-regressive GPT-2-medium model, and their shift under fine-tuning on causal language modeling task. The GPT-2-medium fine-tuning here is performed with standard parameters on the first 000 sequences of the given dataset's 's train portion, and we show the local estimates at the checkpoint after 1200 batches.
Interesting cases to note are the RoBERTa and GPT-2-medium models fine-tuned on the Reddit data: Since the text from the dataset was created in the year 2022, long after the training of RoBERTa and GPT-2-medium finished, this combination illustrates the behavior of a model fine-tuned on data that it has not seen during pre-training. The general observations from Section 4.1 hold for the autoregressive models as well: The LIDs embeddings originating from the fine- tuning distribution decrease markedly between models, whereas the LIDs for the largely unaffected or have increased.
corpora remain
Local estimates for larger models The geometric perspective and methodology we propose are broadly applicable across model architectures. Local estimates computations for the Phi-3.5-ming instruct model in Figure 13 and for Llama-3.1-8B in Figure 14) demmonstraee that our sub-sampling procedure yields meaningful estimates even for latent spaces whose ambient dimension is in the thousands, typical setting for modern LLM architectures.
Phi-3.5-mini min---mini-i is decoder model with 3.82B parameters and hidden dimension of 3072 (Ab din et al. 2024), while Llama-3.1-88 1-8B is even larger with 8.03B parameters and a hidden dimension of 4096 (Al@ @ Meta 2024). We fine-tune via LoRA with rank constraint r 16 on a random subsample of 10 000 sequences from the MultiW0Z and Reddit training splits, with other hyperparameters as described in /eecion 4.T The violin plots in Figures 13 and 14 show the distribution of the LIDs of the validation split for the model checkpoint after 800 batches.
28

---

Medwen
5.09 19.37: (RoBERT--osee)
20
6.67 14,90 26. [FT MuRtINZ)
8.52 18,84 72.61 IFT Reedil)
TwoNN
2.81/5.05//.98 IFT Wiitert)
10
RoBERTa-base
FT MultiWOz
FT Reddit
FT Wikitext
(a) TwoNN estimates on MultiW0Z Validation embeddings
20
(2.41 [NoSEKTe-bass) 5.88 55.96 12.60 [FT MultNNZ)
15
6.91 16,74 6.74 2.118 (FT Readit)
5.99/7..0 2244 (FT (ikisee))
TwoNN
10
RoBERTa base
FT MultiWoz
FT Reddit
FT Wikitext
(b) TwoNN estimates on SGD Validation embeddings
5.00 IFT
30
11.69 11.37 (467 IFT Raddit)
TwoNN
12:88/ 12.78 IFT WiRtort
20
RoBERTa-base
FT MultiWoz
FT Reddit
FT Wikitext
(c) TwoNN estimates Reddit Validation embeddings
40
18.09 17.85 15.12 (FT NuR/NOZ)
30
15.50 16.29 14.78 IFT Reedil)
TwoNN
15.39/ 15.27 15.31 (FT Wiitort
10
RoBERTa-bast
FT Multiwoz
FT Reddit
FT Wikitext
(d) TwoNN estimates on Wikipedia Validation embeddings
Figure 11: Distribution of LIDs of the masked language model RoBERTa-base and fine-tunes (FT),
C.2 Layer-Wise Wise Computation Local Estimates
Our setup from Section 3 and Algorithm naturally applies to embeddings derived from arbitrary layers the language model. In particular, the mean local estimates can be compared layer-wise between different checkpoints of a model, that the drop mean local dimension on the dataset used for fine-tuning is visible over all the model layers. It appears be most pronounced in the intermediate and last layers of the model.
show in Figure 15 The main observations here
C.3 Additional Grokking Results
This section supplements the grokking experiments Section 4.2] See See Figure for a plot comparing model performance and mean local estimates on the training shown separately for the training
29

---

run seeds. This representation is useful, as the timing of the onset of grokking can be very dependent on the model and dataset shuffling seed. remains true that for individual seeds, the drop in mean local dimension on the training set coincides with the increase in validation accuracy.
C.4 Additional TripPy-R Dialogue State Tracking Results
We here show additional results for [Section 4.3] where we discuss exhausting model training capa- bilities. Segure f7cffor longer TripPy-R dialog state tracking runs over 50 epochs. After about 20 training epochs, the mean local estimates stabilize and suggest the convergence of the model performance on the downstream task.
30

---

Huar Mizian
L992
6.22// 5.22/5.99// /19911.50 1.54 (FT MuRtWOZ)
15
71.8/7.5/1000 91 RedoR) 8.30/17
TwoNN
10
GPT-2-medium
FT MultiWOZ
FT Reddit
FT Wikitext
(a) TwoNN estimates on MultiW0Z Validation embeddings
Hean Median (GFT2-mmdum))
27-29/ ReddR! 8.13/7 1/.5/7..92/ 7 (FT Wkkkost
TwoNN
10
GPT-2-medium
FT MultiWOz
FT Reddit
FT Wikitext
TwoNN estimates on SGD Validation embeddings
30
(15.81/4.30 MJJOWOZ) 15 115.79 75/4.11 4.11 RedeRt
15.48 155, 79..79 4.25
(FT Wkihodd)
TwoNN
20
10
GPT-2-medium
FT MultiWOz
FT Reddit
FT Wikitext
(c) TwoNN estimates ICLR 2024 Validation embeddings
30
13.991/5.88 medium)
/ 13.36/13.///5 MuttWO2
13.12/ /13.2///////33 5.33 (FT RedoR!
13.99 /5.82 WKRkOs!
TwoNN
GPT-2-medium
FT MultiWoz
FT Reddit
FT Wikitext
(d) TwoNN estimates Reddit Validation embeddings
Vean /Metten
/1/.9/.2//3.... /3../ (GPT:2-medium)
18.8 1R.51 11.4///55/.. 3.95 (FT Wull
30
/ / 88 AwddR1
TwoNN
17.33/17.91/3.80
20
GPT-2-medium
FT MultiWoz
FT Reddit
FT Wikitext
(e) TwoNN estimates on Wikipedia Validation embeddings
Figure 12: Distribution of LIDs of the autoregressive GPT-2-medium model and fine-tunes (FT),
31

---

7.48 7:48/7.51/3.77 3.5 mini InstrUct)
6.91/6.82/344 IFT NUtIVOZ))
7.28 1.28/7.24/3.544 Reeidh))
TwoNN
30
10
MultiWOZ
FT Reddit
(a) TwoNN estimates on MultiWOZ Validation embeddings
50
Medie
(44//681/4400 (PHi-3 5-mirienstnuutt
40
5.74/5.73/3.97 IFT Mutiwo22)
6.18 2.18/6.34/4288 IFT Roc
TwoNN
30
20
10
Phi-3.5-minl-nnsrruct
MultiWOz
FT Reddit
(b) TwoNN estimates on SGD Validation embeddings
mini
30
13:89 13.59 13..9//5.22 IFT NuttiOz)
13:55 13/./33..400 23.42/4.47
IFT Red
TwoNN
20
10
Phi-3.5-min miniiitriit
FT MultiWoz
FT Reddit
(c) TwoNN estimates on ICLR 2024 Validation embeddings
Mean/ Median
14.83/ 9:47 (PN-3.55-m t-mrinnstutt 16-42 16, 66.60/66.0 26 IPT MullWWZ
13:81 13.60 /5 IFT Red
TwoNN
30
10
Phl-3.5-min--nstrucct
FT MultiWOz
FT Reddit
(d) TwoNN estimates on Reddit Validation embeddings
50
Moun/ Motiun Std
15.18 14.05/5555 JPh.-355-mm www.aa3.miii-nsttnuu
40
18.84 18.14/55.33 22 IFT MuttWOZ)
/3.773/5..0 Red
TwoNN
30
Pni.3.5-minninstrucct MultiWOz
Reddit
(c) TwoNN estimates on Mikipedia Validation embeddings
Figure 13: Distribution of LIDs the autoregressive Phi-3.5--min 3-mmiii-instruue model and fine-tunes (FT).
32

---

20
2:26/8.03/2113 [1Jam--3.--00
Medinn
6.56/6 /.1///2.0 MuRtWoZ)
15
2.67/7.36/2 IFT Reodn)
TwoNN
10
Llama-3.1 3,11-B
FT MultiwOz
FT Reddit
(a) TwoNN estimates on MultiWOZ Validation embeddings
1/8/7-32//.777
5.82/5 47/1//7 MunrNOZ)
6.75 / 66443/3/// 43/2.000 IFT RecdR)
TwoNN
10
Llama-3.1-88
FT MultiWOZ
FT Reddit
TwoNN estimates on SGD Validation embeddings
MUZANOZ)
32.01 /1, 46/46//422 4.23 ReddR)
TwoNN
20
10
Uama-3.1-8B
FT MultiwZ
FT Reddit
(c) TwoNN estimates ICLR 2024 Validation embeddings
40
6,14 MulINaZ)
12 22.52 12: 5.15 IFT RuddR)
30
TwoNN
20
Lamm-3.1-88
FT MultiWOZ
FT Reddit
(d) TwoNN estimates Reddit Validation embeddings
4.29 /Uamm-3311-8))
40
17.92/ / 6L IFT MutthOZ)
17,97 17.22 /4.3 IFT Roodtt)
TwoNN
30
20
10
Lama--3118B
FT MultiWOz
FT Reddit
(e) TwoNN estimates on Wikipedia Validation embeddings
Figure 14: Distribution of LIDs of the autoregressive Llama-3. 3-BBB model and fine-tunes (FT).
33

---

TwoNN dimension estimates for Mult::W0--validdtioon
20
RoBERTa RoBERTa fine tuned on Mult WO2 train
Mean
15
10
Layer
TwoNN dimension estimates for Wikipedi aaaplpeddaaaiiidiee valiidatio
20
RoBERTa RoBERTa fine luned on Mult V02 train
Mean
15
10
12
10
Layer
Figure 15: Comparison of the mean LID development through the different layers of RoBERTa-base and a fine-tuned variant. The LID of embeddings from the distribution used for fine-tuning (here MultiWOZ) differs significantly between the two models, whereas the LID of other data distributions (here Wikipedia) is indistinguishabl..
2.0
3.8
1.4
0.20
0.45
0.25
0.50
0.30
6003 200co
Training accuracy
(b) Validation accuracy
(c) Training mean local estimates
Figure 16: Training model on addition mod 197 with different training data fraction selected from 1; 0.15; 0.2; 0.25; 0.3: 0.5) Five seeds per configuration are shown separately, demon- strating that the onset of grokking and the drop in mean local dimension vary greatly depending on the specific training run. The plots show the development over 15 000 batches. The mean local estimates are computed on the training split for N 3000; L 64.
0.5).
Meon sin
poin acoutcy
(a) MultiWOz Training split
(b) MultiNOZ Validation split
(c) MultiWOZ Test split
Figure 17: Longer TripPy-R training over 50 epochs with linear learning rate schedule. Shown are the mean and standard deviation of the measures over 4 seeds the training runs.
34


Reviews:
[{"review_id": "hwiC33narh", "reviewer": "Reviewer_67F6", "summary": "This paper looks at model, fine-tuning, overfitting, and grokking from a geometric perspective. By computing local intrinsic dimensionality estimates they show that fine-tuning reduces dimensionality specifically for the dataset which is fine-tuned. They also show grokking behaviour is related to reductions in local dimensionality, and that the point when reductions saturate indicates when training on a given dataset is exhausted.", "questions": "As a formatting point, the visuals are currently quite small. In future versions I would appreciate if they could be made larger and floated to the top of the page.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper presents an interesting perspective on an important topic. Unlike a lot of interpretability work which uses behavioural or probing evidence, the work here works to link the geometry of model embedding spaces to model behaviours. Overall the paper is clearly written, and the formalisations are easy to follow. The paper has clear scientific merit and novelty.\n\nWith this being said, there are a few weaknesses. First it's worth noting that the results broadly include no statistical testing. For example, with the results in figure 2, the body text states \"the training mean local dimension starts dropping significantly for those runs that exhibit grokking. Observe that the timing of this drop coincides with the start of the increasing validation accuracy in Figure 2b.\" Correlating local dimension and performance would make this claim more clearly and attach to it a measure of significance. Similarly the first set of results assert a significant difference between local dimension for the three different datasets, and the last set of results relate local dimension over time to overfitting behaviours. All places where support from statistical testing would be welcome.\n\nSecond, the results in the main paper are on largely on Roberta which is comparatively quite a small model by today's standards. The grokking results are also on a two-layer 128 dimensional transformer. Given this and the computational complexity of the methods used it's unclear to me how generalisable the methods here are. Despite this the introduction stresses the importance of understanding large language models and cites work introducing models with billions of parameters. I think the paper would benefit from claims more focussed to the scope of the results.", "was_revised": true, "final_justification": "Overall the paper is well written and of clear scientific merit. The analysis is currently quite computationally expensive, and largely applied to smaller models. The authors acknowledge this and so I think it is worth seeing discussed at the conference.", "rebuttal": "We sincerely thank Reviewer 67F6 for their thoughtful and constructive feedback. We are especially grateful for the positive assessment of the paper’s clarity, novelty, and scientific merit, as well as for highlighting the value of linking the geometry of embedding spaces to model behavior.\n\n## Statistical testing\n\nRegarding the point about statistical testing, we emphasize that the results presented in the grokking, training dynamics, and overfitting sections (4.2, 4.3, 4.4) are **averages over multiple training runs**, accompanied by their respective standard deviations. The relatively small magnitude of these deviations compared to the observed changes supports our characterization of differences as “significant”. However, we agree that formal statistical tests would underline our claims. In the revised version of the paper, we will include appropriate statistical analyses to complement our main findings, particularly in the sections concerning grokking behavior, training, and overfitting dynamics.\n \n## Dimension measures for larger language models\n\nWhile our experiments focus on RoBERTa, GPT-2 and a small transformer used for grokking studies, the geometric perspective and methodology we propose are **broadly applicable across architectures.** \n\nTo support this claim, we are currently running additional experiments on more recent models. Preliminary results on `Phi-3.5-mini-instruct` [b], a decoder model with 3.82B parameters and a hidden dimension of 3072, can be found in the table below. These results demonstrate that the TwoNN estimator, applied locally within our subsampling method, yields **meaningful estimates even for latent spaces whose ambient dimension numbers in the thousands**, a typical setting for modern LLM architectures. Note that, for example, the 70B parameter Llama family, such as \"meta-llama/Llama-3.1-70B-Instruct\" has a hidden dimension size of 8192, which is within reach of our methods. \n\nIn the case of the `Phi-3.5-mini-instruct` model, we implement the fine-tuning via LoRA with rank constraint r=16 on a portion of the MultiWOZ and Reddit training datasets, with other hyperparameters as described in the paper and taking a model checkpoint after 800 batches. During this fine-tuning, we observe the shifts in mean local intrinsic dimension on a subsample of the validation set of various datasets. In addition to the datasets in the paper, we present results for a self-collected dataset of ICLR 2024 titles and abstracts, for the sake of including a more recent dataset.\n\nThe following table contains the (Mean / Median / Std) of the local intrinsic dimension for **base** and **fine-tuned (FT)** versions of the `Phi-3.5-mini-instruct` model.\n\n| Dataset     | Base \"Phi-3.5-mini-instruct\"            | MultiWOZ-FT     | Reddit-FT      |\n|-------------|------------------------|--------------------------|-------------------------|\n| ICLR 2024   | 12.98 / 12.67 / 4.37   | 13.89 / 13.99 / 5.20     | 13.55 / 13.42 / 4.67    |\n| MultiWOZ    | 7.48 / 7.91 / 3.77     | 6.91 / 6.82 / 3.64       | 7.78 / 7.74 / 3.94      |\n| Reddit      | 14.55 / 14.83 / 5.47   | 16.42 / 16.60 / 6.99     | 13.81 / 13.60 / 5.33    |\n| SGD         | 6.14 / 6.81 / 4.01     | 5.74 / 5.73 / 3.97       | 6.18 / 6.34 / 4.26      |\n| Wikipedia   | 15.18 / 14.05 / 5.51   | 18.84 / 18.14 / 5.32     | 18.65 / 17.73 / 5.06    |\n\nNote that our qualitative observations on the behaviour of the autoregressive model’s latent spaces under fine-tuning hold for this larger model as well: The **dimension drops on the datasets that are related to the fine-tuning data** (e.g., the dialogue datasets MultiWOZ and SGD become lower-dimensional in the latent space when tuning the model on MultiWOZ). At the same time, the dimension of unrelated datasets can be seen to increase or stay unchanged, similar to what happens when fine-tuning the notably smaller GPT-2 model.\n\nThese results and the discussion on larger models will be added in the camera-ready version of our paper. We are currently extending our analysis to more recent models (e.g., from the Llama family), and intend to include these results in the final version of our manuscript.\n\nOur choice of RoBERTa was motivated by its **continued use in research and applications** where encoder-based models are needed [a] and autoregressive alternatives are less suitable, such as in information retrieval, which is an important component in Retrieval Augmented Generation. We will clarify this rationale and better situate the scope of our contributions in the revised manuscript.\n\n## Layout\n\nFinally, we thank the Reviewer for the suggestion regarding figure layout and formatting. We will enlarge the visuals and adjust their placement to improve readability in the revised manuscript.\n\nOnce again, we are grateful for the constructive feedback and the encouraging evaluation of our work. \n\n[a] Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (arXiv:2412.13663)\n\n[b] Phi-3 Technical Report (arXiv:2404.14219)"}, {"review_id": "7cMnDF8hKR", "reviewer": "Reviewer_PQs3", "summary": "This paper investigates the internal mechanisms of large language models (LLMs) by analyzing the geometric properties of contextual latent embeddings, focusing on how fine-tuning alters model behavior. Motivated by the lack of unsupervised methods to understand LLM embedding spaces, the authors introduce a framework to measure local intrinsic dimensions using the TwoNN estimator, which captures the low-dimensional manifold structure of token embeddings. Their approach involves sampling token embeddings, computing local neighborhood dimensions, and analyzing shifts during training and fine-tuning. Experiments across dialogue state tracking, emotion recognition, and arithmetic tasks show that changes in local intrinsic dimensions predict training convergence, overfitting, and grokking, with reductions in mean local intrinsic dimensions correlating with performance gains.", "questions": "1. Intuitively, the local dimension might be related to tokens entropy. I hope the author can add some discussion on this, especially considering the recent surge in papers related to entropy and LLM reasoning, e.g.,  [1-2]\n\n[1] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning.\n\n[2] The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models.\n\n\n2. The paper connects local intrinsic dimensions drops to grokking, but the causal relationship is not explained. Why does a decrease in mean LID indicate generalization beyond training data? Similarly, it could be better to add more discussions on other main findings.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. This paper is easy to follow. Its main contributions are clear.\n\n2. The geometric perspective is interesting. The use of local intrinsic dimension to analyze LLM dynamics offers a fresh and unsupervised approach to interpret model behavior. The method bridges intrinsic model mechanisms and geometric properties, providing insights beyond traditional supervised diagnostics.\n\nWeaknesses\n\nAlthough the paper's technical aspects are quite interesting, as a study on understanding language models, I believe the most important aspect is to test its effectiveness on modern models and key issues, which is currently lacking in this paper.\n\n1. The model evaluation is not comprehensive, especially regarding its relevance to recent modern LLM architectures, which remains questionable.\n\n2. As a rapidly evolving field, LLM evaluation is crucial. The datasets tested in this paper are limited, particularly lacking datasets from the past two years.\n\n3. The high cost of metric computation limits its practical applicability.\n\n4. The underlying assumption is TwoNN Assumptions, which assumes locally uniform sampling distributions. The authors do not detail how LLM embedding spaces satisfy it. Are there theoretical justifications for applying TwoNN to LLM latents?", "was_revised": true, "final_justification": "After reading the response and other reviewers' comments, I decide to weakly accpet this paper.", "rebuttal": "We thank Reviewer PQs3 for their constructive feedback. We especially appreciate the recognition of our paper’s clarity, the novelty of the geometric perspective, and our effort to develop an unsupervised approach to interpreting LLM behavior via *local intrinsic dimension (LID)*. We are encouraged that they found our main contributions clear and the methodological framing on understanding language models interesting. Below, we address each of the reviewer’s concerns in detail.\n\n## Addressing the Weaknesses\n\n### (1) Evaluation of modern LLM architectures\n\nWe would like to address the reviewer’s concern regarding the relevance to modern LLM architectures. We chose RoBERTa and its fine-tuned variants because our focus lies in understanding **fine-tuning dynamics in contextual embedding spaces**, settings where pretrained models are adapted to downstream tasks. These architectures **remain widely used in practice**, especially in encoder models [a] and in settings where compute or data constraints prohibit larger model training. \n\nTo demonstrate the **feasibility of our approach on higher-dimensional latent spaces**, we compute our local intrinsic dimension on the `Phi-3.5-mini-instruct` model [b], which has a hidden dimension of 3072.\n\nThe following table contains the (Mean / Median / Std) of the local intrinsic dimension for **base** and **fine-tuned (FT)** versions of the `Phi-3.5-mini-instruct` model. This demonstrates that **our method provides reasonable estimates for higher-dimensional latent spaces produced from modern LLMs**.\n \n| Dataset     | Base \"Phi-3.5-mini-instruct\"            | MultiWOZ-FT     | Reddit-FT      |\n|-------------|------------------------|--------------------------|-------------------------|\n| ICLR 2024   | 12.98 / 12.67 / 4.37   | 13.89 / 13.99 / 5.20     | 13.55 / 13.42 / 4.67    |\n| MultiWOZ    | 7.48 / 7.91 / 3.77     | 6.91 / 6.82 / 3.64       | 7.78 / 7.74 / 3.94      |\n| Reddit      | 14.55 / 14.83 / 5.47   | 16.42 / 16.60 / 6.99     | 13.81 / 13.60 / 5.33    |\n| SGD         | 6.14 / 6.81 / 4.01     | 5.74 / 5.73 / 3.97       | 6.18 / 6.34 / 4.26      |\n| Wikipedia   | 15.18 / 14.05 / 5.51   | 18.84 / 18.14 / 5.32     | 18.65 / 17.73 / 5.06    |\n\n\nMoreover, we are currently working on an extension to even more recent models (e.g., Llama), and we aim to include the respective results in our final manuscript.\n\nOur **code is available** as part of the supplemental material, and our local estimates computation is compatible with models from the HuggingFace `transformers` library. Thus, running our dimension estimation pipeline on larger models is mainly a matter of having compute available. \n**According to Reviewers 67F6 and rATb, our experiments are comprehensive**, and our model selection supports the conclusions drawn in our paper.\nUsers of our method can always supply their own, larger model if applicable.\n\n### (2) Dataset selection and recency\n\nWe would like to address the reviewer’s concern regarding the selection of our datasets. Our goal was to **capture contrasting behaviors in embedding space evolution**, so we deliberately selected **two complementary types of datasets**: \n\n(1) a dataset that was part of the original **pretraining corpus** (Wikipedia), and (2) a dataset that was released **after the pretraining of the model concluded** (Reddit) and hence **not seen during training**. This contrast allows us to probe how local intrinsic dimensions behave in both *seen* and *unseen* data regimes.\n\nOur code is released as part of the supplemental material, and **our local estimates computation pipeline is compatible with the HuggingFace `datasets` package**. Thus, practitioners can easily extend the evaluation to their datasets. We demonstrate this above with our self-collected ICLR 2024 dataset containing recently released machine learning paper titles and abstracts.\n \n### (3) Computational cost of TwoNN estimation\n\nWe address the concern about computational cost in the limitations section. While TwoNN involves pairwise distance calculations, we emphasize that our method employs **lightweight sampling strategies** that drastically reduce computational demands. In practice, we use random token-level sampling, which makes the method **feasible for large-scale experiments**, as demonstrated across all tasks. Our new results on models from the Phi-family show that **our method can be practically applied to modern LLM architectures**.\n\n### (4) Validity of the TwoNN assumption in LLM embedding spaces\n\n\nWe interpret the value of the TwoNN estimator as a local dimension. That this interpretation hinges on certain assumptions is explicitly noted in the limitations section of the paper. However, the **experimental results and their practical implications do not depend on this interpretation**. This is why we view this limitation as relatively unimportant.\n\nStill, we are happy to underpin our particular choice of local dimension estimator with theoretical findings: \n\n- **Locally uniform density of the embedding space:** In Appendix B of “The geometry of hidden representations of large transformer models” (Valeriani et al., NeurIPS 2023), the authors demonstrate empirically that hidden states of Transformer models exhibit **locally constant density**, satisfying this condition. \n- **Poisson point process assumption:** As noted in “The Geometry of Tokens in Internal Representations of Large Language Models” (Viswanthan et. al, arXiv:2501.10573), this condition is usually satisfied in our setup. \n\nWe will make these justifications explicit in the updated manuscript and add references and discussion.\n\n\n## Questions\n\n### (Q1) On the relationship between local dimension and token entropy\n\nWe thank the reviewer for raising this potential connection. Indeed, there may be a relationship between token-level entropy and local intrinsic dimension, and this is a promising direction for future work.\n\nHowever, the recent studies on entropy and reinforcement learning (RLVR) cited ([1], [2]) focus on reward optimization and exploration behavior in instruction-tuned or RL-finetuned models, scenarios that **differ fundamentally** from our setting. These papers were also released after our submission deadline. We agree this connection is worth exploring and have added the general setting of RL-tuning of LLMs to our future work section.\n\n### (Q2) Causal link between LID and generalization/grokking\n\nWe agree that a clearer discussion of why reduced LID corresponds to generalization would improve the paper. Our working hypothesis is that **high-dimensional neighborhoods indicate overparameterized, poorly localized behavior**, while **very low LID reflects overcompression or memorization**. There exists a “sweet spot” in LID, where the representation captures the essential degrees of freedom required for solving the task. Our experiments show that this intermediate range correlates with peak generalization. We will make this intuition more explicit, but one should note that **investigating causality is beyond the scope of this paper**, which is also acknowledged by Reviewer rATb.\n\n## Conclusion\n\nWe believe that we have adequately addressed the concerns regarding evaluation scope, dataset choices, computational feasibility, theoretical grounding, and interpretation of our results. We are committed to incorporating all necessary clarifications and extensions in the final revision and appreciate the reviewer’s suggestions.\n\nIf you find that these responses and planned improvements sufficiently address your concerns, we would be very grateful if you would consider raising your score.\n\n[a] Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (arXiv:2412.13663)\n\n[b] Phi-3 Technical Report (arXiv:2404.14219)"}, {"review_id": "aFrZ9oZUoI", "reviewer": "Reviewer_rATb", "summary": "This paper introduces a novel approach for analyzing large language model (LLM) behavior by examining the geometric structure of their embedding spaces through local intrinsic dimension analysis. Using a localized version of the TwoNN estimator, the researchers demonstrate that while LLM embeddings exist in very high-dimensional spaces (hundreds to thousands of dimensions), they actually occupy lower-dimensional manifold structures that vary across different regions. This geometric analysis provides unsupervised insights into model behavior without requiring labeled data or task-specific supervision, addressing the central question of whether structural changes in embedding spaces can reveal model behavior across language modeling tasks.\n\nThe paper presents four key applications of this framework: (1) detecting fine-tuning effects and training data overlap through dataset-specific dimensional reshaping, (2) predicting the onset of “grokking” (sudden generalization) in synthetic arithmetic tasks before it occurs, (3) monitoring training convergence through stabilizing dimensions in dialogue state tracking, and (4) identifying overfitting patterns through characteristic dimensional changes that reflect the transition from generalization to memorization. These results suggest that local intrinsic dimensions serve as valuable unsupervised signals for practitioners seeking to interpret and monitor LLM training dynamics, potentially complementing traditional evaluation methods and informing future model design approaches.", "questions": "- Implications for Fine-Tuning: The paper compellingly shows that local dimensions can diagnose training phenomena. Beyond diagnosis, what are the direct, actionable implications for improving fine-tuning? For instance, could the mean local dimension be used as a regularization term in the loss function to explicitly encourage dimensional compression and potentially accelerate generalization or prevent overfitting?\n- Exploring Local Heterogeneity: The core method computes a mean local dimension across the token sample. However, the motivation for using a local estimator is the heterogeneity of the embedding space. Have the authors explored analyzing the distribution of local dimensions further? For example, do certain token types (e.g., punctuation, stop-words vs. content-specific nouns) consistently occupy higher or lower dimensional regions, and how do these distributions shift during fine-tuning? This could provide even more granular insights than the mean alone.\n- Connection to Parameter-Efficient Fine-Tuning (PEFT): Methods like LoRA are known to be effective by operating in a low-rank (i.e., low-dimensional) update space. How do you hypothesize your findings on latent space dimension relate to the intrinsic dimension of the parameter updates in PEFT? Could a similar analysis be used to dynamically set the rank r in LoRA based on the geometric needs of the task, as hinted at by the related work from Ed-dib et al. (2024)?\n- Generative vs. Encoder Models: The experiments include an autoregressive model (GPT-2) but the analysis focuses heavily on encoders (RoBERTa, BERT). Are there any observed qualitative differences in how the local dimensions behave between encoder-only and decoder-only architectures during fine-tuning? For example, does the causal attention mechanism in decoders lead to a different geometric signature compared to the bidirectional context of encoders?", "limitations": "Yes, the authors have adequately addressed the limitations of their work in Section 4.6. They are transparent about the computational complexity, the an-is an artifact of the optimization process which also leads to generalization, which is an interesting question worth exploring further. Dependency of the absolute dimensional values on hyperparameters (which makes it a “relative” measure), and the assumptions of the TwoNN estimator. This upfront discussion is commendable.\n\nOne constructive suggestion for improvement would be to briefly touch upon the missing causal link as a limitation. While they demonstrate a powerful correlation, explicitly stating that the causal mechanism (i.e., whether dimensional reduction drives generalization or is merely a symptom of it) is a topic for future work would further strengthen this already solid section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Significance & Originality: The paper addresses fundamental, important questions about LLM training dynamics (fine-tuning, overfitting, grokking, convergence) from a novel and promising angle. Shifting the focus from task-specific metrics to the intrinsic geometry of the latent space is a creative and insightful contribution. The approach of using a local intrinsic dimension is a key strength, as it captures the heterogeneity of the embedding space, which a single global dimension estimate would miss.\n- Quality: The experimental setup is rigorous and well-designed. The authors validate their method across a diverse set of tasks (dialogue, arithmetic, classification) and models (RoBERTa, GPT-2), demonstrating the generalizability of their findings. The inclusion of a thorough sensitivity analysis in the appendix (Appendix A) adds significant credibility to the stability and robustness of the proposed measurement technique.\n- Clarity: The paper is exceptionally well-written and structured. It clearly articulates its central question, methods, and contributions. The visualizations, particularly the comparative plots showing model metrics alongside mean local dimension (Figures 2, 3, 4), are intuitive and effectively communicate the core results.\n- Practical Implications: The conclusion that a drop in mean local dimension correlates with improved generalization provides a powerful, unsupervised heuristic. This could be immensely valuable for practitioners, offering a way to monitor training, detect overfitting, and potentially implement early stopping or other interventions without relying on a labeled validation set, which is often a practical bottleneck.\n\nWeaknesses:\n\n- Computational Overhead: A primary weakness, acknowledged by the authors, is the computational cost of the method. Calculating neighborhoods for tens of thousands of high-dimensional vectors for each checkpoint can be prohibitive, especially for large-scale models or frequent evaluations. This may limit its practical adoption in fast-paced or resource-constrained development cycles.\n- Relativity of Measurements: The exact value of the mean local dimension is dependent on hyperparameters (N, L) and model architecture. This makes it difficult to compare absolute dimension values between different models or different hyperparameter settings. The paper’s insights are based on observing the relative change (the “delta”) during a single training run, which is valid but limits the scope of cross-model analysis.\n- Causality vs. Correlation: While the paper demonstrates a strong correlation between dimensional shifts and model behavior, the causal link remains an open question. Does the dimensional reduction cause generalization (e.g., by forcing the model to find a more compressed, efficient representation), or is it merely an artifact of the optimization process that leads to generalization? While a full causal explanation is beyond the scope of this paper, a deeper discussion of this point would be welcome.", "was_revised": true, "final_justification": "I will keep the score the same and really appreciate detailed responses from the authors on my questions.", "rebuttal": "We sincerely thank Reviewer rATb for the thoughtful and constructive review. We are grateful for the recognition of the strengths of our work, particularly your appreciation of the originality of our framing, the rigorous experimental design, and the clarity of the presentation. We are pleased the reviewer found **practical utility in our method** and **robustness in our analysis**.\nWe would like to take this opportunity to address the specific points you raised:\n\n## Addressing the Weaknesses\n\n### Computational Overhead\n\nWe agree that computational cost is a potential barrier to large-scale adoption. However, we would like to emphasize that we already **mitigate this issue through random subsampling of tokens**, which significantly **reduces overhead** while **maintaining stable estimates**. Our sensitivity analysis (in Appendix A) empirically supports this approach, demonstrating that local dimension estimates are robust under sampling. We will further clarify this point in the revised manuscript to better guide practitioners on balancing accuracy and efficiency.\n\n### Relativity of Measurements\n\nWe acknowledge that absolute local dimension values depend on the choice of hyperparameters and model architecture. However, our extensive **sensitivity analysis** (Appendix A) shows that our findings are robust across a wide range of neighborhood sizes and subsampling rates. We will make this robustness more prominent in the main text.\n\n### Causality vs. Correlation\n\nRegarding the distinction between causation and correlation in dimensionality reduction and generalization, while disentangling this relationship is beyond the scope of our current work, we agree that the question is critical. We will revise Section 4.6 to explicitly state this limitation and outline directions for future causal investigations. We are also happy to include a brief speculative discussion on possible mechanisms (e.g., information compression) by which **reduced dimensionality might encourage better generalization**.\n\n## Questions\n\n### Implications for Fine-Tuning / Regularization via LID:\n\n\nThis is an insightful question. While our current dimension estimation method is not differentiable (due to nearest-neighbor graph construction and the subsequent TwoNN estimator), we see this as an exciting **opportunity for future work**. Designing a differentiable surrogate or proxy loss to encourage local compression during training could open up new avenues for model regularization. We will expand our discussion of this direction in Section 5.\n\n### Heterogeneity of Local Dimensions Across Token Types\n\nWe have indeed explored the heterogeneity of local dimension distributions by grouping tokens according to POS tags. For example, we observed that on average, **punctuation and stop words tend to occupy higher-dimensional regions**, while content-bearing words such as nouns and verbs exhibit lower local dimensions. We will include a discussion of this phenomenon in the revised manuscript and add a supporting figure to the Appendix.\n\n### Connection to Parameter-Efficient Fine-Tuning (PEFT)\n\nWe appreciate this connection and have explored it preliminarily. The drop in mean local dimension on the fine-tuning set **can be detected on LoRA-tuned models as well**, in line with our observations in Section 4.1. Interestingly, we did not observe consistent differences in local dimensional dynamics between full and LoRA-based fine-tuning. \n\nHowever, we agree that adapting the LoRA rank dynamically based on local geometry is a promising idea. We will add a note on this to the discussion section, citing the relevant work by Ed-dib et al. (2024), and marking it as an exciting direction for future research.\n\n### Encoder vs. Decoder Architectures:\n\n\nWhile much of our analysis focuses on encoder models, we **also included decoder-only models** (Appendix C, Figures 12 and 13) and observed comparable dimensional drops on the fine-tuning set. We will clarify this in the main text and ensure that the decoder results in Appendix C.1 are better connected to the core narrative.\n\nWhen comparing the distribution plots of a given dataset between the encoder-only RoBERTa and decoder-only GPT-2 model, one observes **stark differences**. For instance, the distribution of local estimates on the Reddit dataset in the GPT-2 latent space appears to be bimodal, which is less pronounced in the RoBERTa latent space. Investigating the reasons for these differences is an interesting direction for future research.\n\nAdditionally, we are currently working on an extension to even more recent decoder models (e.g., the Phi and LLaMA model family), and we aim to include the respective results in our final manuscript.\n\n## Limitations\n\n### Causal Limitation Not Explicitly Stated\n\n\nWe will explicitly note this limitation regarding causality between dimensionality and generalization in the revised limitations section.\n\nWe thank Reviewer rATb again for the detailed and supportive review."}, {"review_id": "90axFpVdCU", "reviewer": "Reviewer_oyKS", "summary": "The authors study the local intrinsic dimension of language models using a localized version of two-NN and study representation changes during training/fine-tuning. They offer an explanation for many behaviors seen in learning dynamics, such as overfitting, grokking, etc.\n\nThe authors study 4 setups: change in ID from fine-tuning, grokking, “exhausting training capabilities”, classification/overfitting.\n\nFine-tuning: the change in local geometry is dataset dependent - fine-tuning on dataset X does not alter the local intrinsic dimension given dataset Y.\n\nGrokking: the change in local intrinsic dimension coincides with when grokking occurs in a toy arithmetic setup.\n\nExhausting training capabilities: monitoring the change in intrinsic dimension over the course of training can inform when training is done. Notably, the change in intrinsic dimension looks very similar across train/validation/test splits, indicating that practitioners can rely on intrinsic dimension to know when training is done, without needing a validation split as most typical training regimes currently rely on.\n\nOverfitting: in a classification setting, an increase in intrinsic dimension coincides with when", "questions": "How were the models and datasets chosen for each setup?\n\nNot a question but a comment - I think the authors will enjoy these works as well: https://arxiv.org/pdf/2410.01444,  https://arxiv.org/abs/2503.21073, which also studies the intrinsic dimension of hidden states/token embeddings.", "limitations": "See “Weaknesses” – also, the authors list some limitations which I agree with: high computational cost (which, if I understand correctly, is why the authors do not study >1B models), and that their intrinsic dimension measure is a “relative” measure and not an absolute measure, making it difficult to know how to compare the measure across models.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "I am a big fan of studying the representations or geometry of latent states to better understand learning dynamics, and I am happy to see contributions in this space. \n\nThe paper’s biggest strength is perhaps also its weakness – the authors use their approach to study multiple phenomena commonly seen during training (fine-tuning, grokking, monitoring training progress, overfitting) through multiple setups, which is great. However, per phenomenon, they only study a single setting, making it difficult to know whether their claims will generalize, making it difficult to draw conclusions per setting. \n\nIt is also unclear how the authors went about choosing their models, tasks, and datasets per setup. The choice of models seemed rather arbitrary, and I am curious why they did not study more recent models. Even for models of smaller scale, I believe there are more recent models that the authors could have studied. The same applies to datasets - many of the datasets seemed to be very specific (ex: EmoWoz). At the very least, an explanation for the design decision per setup would have been helpful.\n\nThere also seem to be minor cases when their claims don’t hold. For instance, Figure 13 in the appendix seems to go against the author’s claim, in which intrinsic dimension is only affected by the data within the domain of the fine-tuning data - this doesn’t seem to be the case in Figure 13?\n\nIn Section 4.4, when discussing classification/overfitting, the authors consider an increase in validation loss to be overfitting.\nHowever, if I understand correctly, the authors do not use this logic in Section 4.3, and claim that even though validation loss is increasing, their intrinsic dimension measure continues to decrease and is a better measure for training progress. This discrepancy is a bit confusing to me.\n\nIn Section 4.2, when discussing grokking, because grokking can be thought of as a phase transition, it is not surprising that the intrinsic dimension and change in validation accuracy coincide. In fact I do not think the intrinsic dimension “predicts” when grokking will occur, and suggest that the authors curtail this claim.\n\nIn summary, while I am excited to see this work, I am not quite convinced that intrinsic dimension provides a useful measure to monitor learning dynamics for practitioners, for the reasons listed above. If the paper showed the same general pattern in multiple experiments per setting, I would be a lot more comfortable drawing concrete conclusions.", "was_revised": true, "final_justification": "My main concern was generalizability of the experiments. The authors have conducted additional analyses, demonstrating that their findings still hold. All of my other concerns have been promised to be addressed in their next revision (claims about predicting grokking, better transparency around how they selected their models and datasets, etc.).", "rebuttal": "We thank Reviewer ​​oyKS for their feedback and for highlighting the potential of our work to advance the study of geometric representations in language models. We are particularly grateful for the appreciation of our effort to connect intrinsic dimensionality with diverse learning dynamics such as fine-tuning, grokking, and overfitting, and for acknowledging the **value of studying the representation geometry of LLMs**.\nBelow, we address each concern in detail.\n\n## Addressing the Weaknesses\n\n### (1) Limited number of experiments per phenomenon\n\nA broader empirical basis per phenomenon would certainly strengthen the generality of our claims. While we focused the main text on one representative setup per phenomenon for clarity and space, we do have **additional results that support the generality of our findings**. \n\nFor example, in the context of overfitting, we conducted further fine-tuning experiments on language modeling, where continued training past optimal validation performance led to an increase in local dimension after the initial drop. This further confirms that a dimension increase is a useful indicator of overfitting. We will include these results in the appendix of the revised version.\n\n### (2) Clarification of dataset and model choices\n\nThank you for requesting more transparency here. We will clarify these design decisions in the revision. Briefly:\n\n- Our dataset selection was guided by the goal of interpreting training and fine-tuning dynamics in controlled, well-understood domains.\n**SDG** and **MultiWOZ** remain as the **largest openly available task-oriented dialogue datasets**, and they come with rich **annotations** (for example, EmoWOZ as an augmented version of MultiWOZ with additional emotion labels per user utterance).\nDialogue provides a test-bed for many natural language processing phenomena, allowing us to study discourse-related challenges like natural language understanding, state tracking, and emotion classification in a realistic yet compact setting.\n\n- The fine-grained and high-quality annotation of MultiWOZ/EmoWOZ allows us to **train models with different goals on the same underlying dataset**. We decided to cover sequence tagging and tracking via dialogue state tracking in the Trippy-R model, and sequence classification via emotion classification in the ERToD model.\nThis, in turn, allowed us to **isolate specific learning signals** in fine-tuning and overfitting settings.\n\n\n- For models, we **selected representative examples** of transformer-based encoder-only (BERT/RoBERTa) and a decoder-only model (GPT-2) to capture both sides of the architectural spectrum. These models are **well-established** and allow for interpretability across training regimes, since, in particular, RoBERTa-style models [a] are still widely applied as encoder models.\n\n- To support our claims of wider applicability, we are currently running additional experiments on more recent models. Preliminary results on `Phi-3.5-mini-instruct` [b], a decoder model with 3.82B parameters and a hidden dimension of 3072, can be found in the table below. These results demonstrate that **the TwoNN estimator, applied locally within our subsampling method, yields meaningful estimates even for latent spaces whose ambient dimension numbers in the thousands**, a typical setting for modern LLM architectures. \n\nIn the case of the `Phi-3.5-mini-instruct` model, we implement the fine-tuning via LoRA with rank constraint r=16 on a portion of the MultiWOZ and Reddit training datasets, with other hyperparameters as described in the paper and taking a model checkpoint after 800 batches. During this fine-tuning, we observe the shifts in mean local intrinsic dimension on a subsample of the validation set of various datasets. In addition to the datasets in the paper, we present results for a self-collected dataset of ICLR 2024 titles and abstracts, for the sake of including a more recent dataset.\n\nThe following table contains the (Mean / Median / Std) of the local intrinsic dimension for **base** and **fine-tuned (FT)** versions of the `Phi-3.5-mini-instruct` model.\n\n| Dataset     | Base \"Phi-3.5-mini-instruct\"            | MultiWOZ-FT     | Reddit-FT      |\n|-------------|------------------------|--------------------------|-------------------------|\n| ICLR 2024   | 12.98 / 12.67 / 4.37   | 13.89 / 13.99 / 5.20     | 13.55 / 13.42 / 4.67    |\n| MultiWOZ    | 7.48 / 7.91 / 3.77     | 6.91 / 6.82 / 3.64       | 7.78 / 7.74 / 3.94      |\n| Reddit      | 14.55 / 14.83 / 5.47   | 16.42 / 16.60 / 6.99     | 13.81 / 13.60 / 5.33    |\n| SGD         | 6.14 / 6.81 / 4.01     | 5.74 / 5.73 / 3.97       | 6.18 / 6.34 / 4.26      |\n| Wikipedia   | 15.18 / 14.05 / 5.51   | 18.84 / 18.14 / 5.32     | 18.65 / 17.73 / 5.06    |\n\nNote that our qualitative observations on the behaviour of the autoregressive model’s latent spaces under fine-tuning **hold for this larger model as well**: The dimension drops on the datasets that are related to the fine-tuning data (e.g., the dialogue datasets MultiWOZ and SGD become lower-dimensional in the latent space when tuning the model on MultiWOZ). At the same time, the dimension of unrelated datasets can be seen to increase or stay unchanged, similar to what happens when fine-tuning the notably smaller GPT-2 model.\nMoreover, we are currently working on an extension to even more recent models (e.g., from the Llama family), and we aim to include the respective results in our final manuscript.\n\n\n\n### (3) Appendix Figure 13 and fine-tuning generalization\n\nThe result in Figure 13 actually **reinforces our core claim**: it shows that the drop in local dimension primarily occurs *within the domain of the fine-tuning data*, while *unrelated inputs* remain largely unaffected or even increase in their mean local dimension.\nThis is what we expect if the model **adapts its representation geometry in a task-specific way**. We will clarify this interpretation in the caption and appendix discussion to avoid confusion. In particular, we will make it clear that the discussion in Section 4.1 *applies primarily to masked language models*, and that decoder-only models appear to exhibit the phenomenon of increasing local dimension on non-training-related data.\n\n### (4) Discrepancy between Section 4.3 and 4.4, re: overfitting criteria\n\n\nThis is an important observation, and we appreciate the opportunity to clarify. In Section 4.3, we observe that validation loss increases while the non-differentiable downstream metric *(Joint Goal Accuracy)* still improves, indicating that when *JGA* is your target metric, **validation loss is not a reliable stopping criterion in this case**. \n\nIn contrast, in Section 4.4, the validation loss and accuracy (weighted F1) are tightly aligned; the respective maximum and minimum occur around the same training step. Hence, the loss serves as a valid stopping signal there, and the **local dimension measures support this point**. The difference in evaluation criteria is task-specific, and we will clarify this point more explicitly in the text.\n\n### (5) Interpretation of grokking as a phase transition\n\nWe agree that grokking is often interpreted as a phase transition. Our novel contribution here is to make this intuitive notion *quantifiable*: our local intrinsic dimension estimates provide a **smooth, quantitative trace of the internal phase transition dynamics**. \n\nCrucially, our method enables this measurement **using training data alone**, and still **anticipates the grokking event** observed in validation accuracy. To the best of our knowledge, this is the first time such alignment can be observed at this granularity. We will revise the text to avoid overclaiming about prediction and emphasize the interpretability advantage.\n\n## Questions\n\n### (Q1) Choice of models and datasets\n\n\nPlease see our earlier explanation [see (2) above]. We will add a dedicated section explaining our design choices per setup in the revised paper.\n\n### (Q2) Suggested related work\n\nWe thank the reviewer for these additional references, and we will incorporate them into our Related Work section. While these works also explore intrinsic dimensions, they **differ in key aspects**: e.g., one focuses on sequence-level embeddings, while the other does not track dynamics during training. \n\nOur method, in contrast, enables fine-grained, *contextualized*, *local* estimates over the course of optimization, which is essential for studying grokking or overfitting behavior. We are happy to discuss how our findings relate to and complement these efforts.\n\n## Limitations\n\n### Limitations and computational cost\n\nOur approach is computationally intensive, which we discuss in our Limitations section, but at the same time, we provide a partial **solution to the computational overhead via our sampling procedure**. Moreover, we extensively **analyse the stability of our estimates under sampling** in our sensitivity analysis in Appendix A1.\n\n\nThe preliminary experiments with the `Phi-3.5-mini-instruct` model (3.82B parameters, 3072-dimensional latent space, for details see our rebuttal to Reviewer 67F6) demonstrate that this **sampling approach applies to models with >1B parameters** and **embedding dimension in the thousands**.\n\nSo while we largely agree with the reviewer’s assessment of the limitations, we believe that with minor modifications in the camera-ready version of the paper, these will be adequately addressed. \n\n## Conclusion\n\nWe thank the reviewer again for their time and engagement with our work. We hope that our clarifications and small additions to the revised manuscript address your concerns. Should these revisions meet your expectations, we would kindly ask you to consider increasing your score.\n\n[a] Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (arXiv:2412.13663)\n\n[b] Phi-3 Technical Report (arXiv:2404.14219)"}]

Author rebuttal:

Meta-review: The paper investigates the internal mechanisms of LLMs by analyzing the geometry of contextual latent embeddings via local intrinsic dimension (LID). The authors introduce a framework to measure LID using the TwoNN estimator, which approximates the token embeddings’ intrinsic dimension, and further analyze shifts during training. Experiments across dialogue state tracking, emotion recognition, and arithmetic tasks show that drops in mean LID accompany improved generalization and characterize: training exhaustion, overfitting, and grokking, respectively.

The paper offers a clear, generally applicable geometric signal that is unsupervised and tracks important training phenomena. The rebuttal materially improved the case. Authors provided supports on the claim generality, provided new results using bigger and more recent LLMs, clarified dataset/model choices, and theoretical assumptions for TwoNN. Authors also committed to adding statistical tests, and will further revise the paper for improved clarity.

With expanded results and the promised analyses, I believe that the contribution is solid and of interest to the communities.

__Strength__
- The bridging of intrinsic model mechanisms and geometric properties of latent states is interesting.
- The work has clear scientific merit and novelty.
- The paper is clearly written and the formalisations are easy to follow.

__Weakness__
- The main weaknesses pointed by reviewers on whether their claims will generalize, has mostly been addressed in the rebuttal
- The prediction of grokking event is seen as an over-strong framing. However, the author will provide revision to focus on the interpretability advantage.
Decision: Accept (poster)