Title: Less is More: Local Intrinsic Dimensions of Contextual Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: latent space geometry

contextual embeddings

LLMs

intrinsic dimension

training dynamics

generalization

overfitting detection

grokking
Abstract: Understanding the internal mechanisms of large language models (LLMs) remains a challenging and complex endeavor. 
Even fundamental questions, such as how fine-tuning affects model behavior, often require extensive empirical evaluation. 
In this paper, we introduce a novel perspective based on the geometric properties of contextual latent embeddings to study the effects of training and fine-tuning. 
To that end, we measure the local dimensions of a contextual language model's latent space and analyze their shifts during training and fine-tuning.
We show that the local dimensions provide insights into the model's training dynamics and generalization ability.
Specifically, the mean of the local dimensions predicts when the model’s training capabilities are exhausted, as exemplified in a dialogue state tracking task, overfitting, as demonstrated in an emotion recognition task, and grokking, as illustrated with an arithmetic task.
Furthermore, our experiments suggest a practical heuristic: reductions in the mean local dimension tend to accompany and predict subsequent performance gains.
Through this exploration, we aim to provide practitioners with a deeper understanding of the implications of fine-tuning on embedding spaces, facilitating informed decisions when configuring models for specific applications. 
The results of this work contribute to the ongoing discourse on the interpretability, adaptability, and generalizability of LLMs by bridging the gap between intrinsic model mechanisms and geometric properties in the respective embeddings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the local intrinsic dimension of language models using a localized version of two-NN and study representation changes during training/fine-tuning. They offer an explanation for many behaviors seen in learning dynamics, such as overfitting, grokking, etc.\n\nThe authors study 4 setups: change in ID from fine-tuning, grokking, “exhausting training capabilities”, classification/overfitting.\n\nFine-tuning: the change in local geometry is dataset dependent - fine-tuning on dataset X does not alter the local intrinsic dimension given dataset Y.\n\nGrokking: the change in local intrinsic dimension coincides with when grokking occurs in a toy arithmetic setup.\n\nExhausting training capabilities: monitoring the change in intrinsic dimension over the course of training can inform when training is done. Notably, the change in intrinsic dimension looks very similar across train/validation/test splits, indicating that practitioners can rely on intrinsic dimension to know when training is done, without needing a validation split as most typical training regimes currently rely on.\n\nOverfitting: in a classification setting, an increase in intrinsic dimension coincides with when\n\nSTRENGTHS AND WEAKNESSES:\nI am a big fan of studying the representations or geometry of latent states to better understand learning dynamics, and I am happy to see contributions in this space. \n\nThe paper’s biggest strength is perhaps also its weakness – the authors use their approach to study multiple phenomena commonly seen during training (fine-tuning, grokking, monitoring training progress, overfitting) through multiple setups, which is great. However, per phenomenon, they only study a single setting, making it difficult to know whether their claims will generalize, making it difficult to draw conclusions per setting. \n\nIt is also unclear how the authors went about choosing their models, tasks, and datasets per setup. The choice of models seemed rather arbitrary, and I am curious why they did not study more recent models. Even for models of smaller scale, I believe there are more recent models that the authors could have studied. The same applies to datasets - many of the datasets seemed to be very specific (ex: EmoWoz). At the very least, an explanation for the design decision per setup would have been helpful.\n\nThere also seem to be minor cases when their claims don’t hold. For instance, Figure 13 in the appendix seems to go against the author’s claim, in which intrinsic dimension is only affected by the data within the domain of the fine-tuning data - this doesn’t seem to be the case in Figure 13?\n\nIn Section 4.4, when discussing classification/overfitting, the authors consider an increase in validation loss to be overfitting.\nHowever, if I understand correctly, the authors do not use this logic in Section 4.3, and claim that even though validation loss is increasing, their intrinsic dimension measure continues to decrease and is a better measure for training progress. This discrepancy is a bit confusing to me.\n\nIn Section 4.2, when discussing grokking, because grokking can be thought of as a phase transition, it is not surprising that the intrinsic dimension and change in validation accuracy coincide. In fact I do not think the intrinsic dimension “predicts” when grokking will occur, and suggest that the authors curtail this claim.\n\nIn summary, while I am excited to see this work, I am not quite convinced that intrinsic dimension provides a useful measure to monitor learning dynamics for practitioners, for the reasons listed above. If the paper showed the same general pattern in multiple experiments per setting, I would be a lot more comfortable drawing concrete conclusions.\n\nQUESTIONS:\nHow were the models and datasets chosen for each setup?\n\nNot a question but a comment - I think the authors will enjoy these works as well: https://arxiv.org/pdf/2410.01444,  https://arxiv.org/abs/2503.21073, which also studies the intrinsic dimension of hidden states/token embeddings.\n\nLIMITATIONS:\nSee “Weaknesses” – also, the authors list some limitations which I agree with: high computational cost (which, if I understand correctly, is why the authors do not study >1B models), and that their intrinsic dimension measure is a “relative” measure and not an absolute measure, making it difficult to know how to compare the measure across models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach for analyzing large language model (LLM) behavior by examining the geometric structure of their embedding spaces through local intrinsic dimension analysis. Using a localized version of the TwoNN estimator, the researchers demonstrate that while LLM embeddings exist in very high-dimensional spaces (hundreds to thousands of dimensions), they actually occupy lower-dimensional manifold structures that vary across different regions. This geometric analysis provides unsupervised insights into model behavior without requiring labeled data or task-specific supervision, addressing the central question of whether structural changes in embedding spaces can reveal model behavior across language modeling tasks.\n\nThe paper presents four key applications of this framework: (1) detecting fine-tuning effects and training data overlap through dataset-specific dimensional reshaping, (2) predicting the onset of “grokking” (sudden generalization) in synthetic arithmetic tasks before it occurs, (3) monitoring training convergence through stabilizing dimensions in dialogue state tracking, and (4) identifying overfitting patterns through characteristic dimensional changes that reflect the transition from generalization to memorization. These results suggest that local intrinsic dimensions serve as valuable unsupervised signals for practitioners seeking to interpret and monitor LLM training dynamics, potentially complementing traditional evaluation methods and informing future model design approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Significance & Originality: The paper addresses fundamental, important questions about LLM training dynamics (fine-tuning, overfitting, grokking, convergence) from a novel and promising angle. Shifting the focus from task-specific metrics to the intrinsic geometry of the latent space is a creative and insightful contribution. The approach of using a local intrinsic dimension is a key strength, as it captures the heterogeneity of the embedding space, which a single global dimension estimate would miss.\n- Quality: The experimental setup is rigorous and well-designed. The authors validate their method across a diverse set of tasks (dialogue, arithmetic, classification) and models (RoBERTa, GPT-2), demonstrating the generalizability of their findings. The inclusion of a thorough sensitivity analysis in the appendix (Appendix A) adds significant credibility to the stability and robustness of the proposed measurement technique.\n- Clarity: The paper is exceptionally well-written and structured. It clearly articulates its central question, methods, and contributions. The visualizations, particularly the comparative plots showing model metrics alongside mean local dimension (Figures 2, 3, 4), are intuitive and effectively communicate the core results.\n- Practical Implications: The conclusion that a drop in mean local dimension correlates with improved generalization provides a powerful, unsupervised heuristic. This could be immensely valuable for practitioners, offering a way to monitor training, detect overfitting, and potentially implement early stopping or other interventions without relying on a labeled validation set, which is often a practical bottleneck.\n\nWeaknesses:\n\n- Computational Overhead: A primary weakness, acknowledged by the authors, is the computational cost of the method. Calculating neighborhoods for tens of thousands of high-dimensional vectors for each checkpoint can be prohibitive, especially for large-scale models or frequent evaluations. This may limit its practical adoption in fast-paced or resource-constrained development cycles.\n- Relativity of Measurements: The exact value of the mean local dimension is dependent on hyperparameters (N, L) and model architecture. This makes it difficult to compare absolute dimension values between different models or different hyperparameter settings. The paper’s insights are based on observing the relative change (the “delta”) during a single training run, which is valid but limits the scope of cross-model analysis.\n- Causality vs. Correlation: While the paper demonstrates a strong correlation between dimensional shifts and model behavior, the causal link remains an open question. Does the dimensional reduction cause generalization (e.g., by forcing the model to find a more compressed, efficient representation), or is it merely an artifact of the optimization process that leads to generalization? While a full causal explanation is beyond the scope of this paper, a deeper discussion of this point would be welcome.\n\nQUESTIONS:\n- Implications for Fine-Tuning: The paper compellingly shows that local dimensions can diagnose training phenomena. Beyond diagnosis, what are the direct, actionable implications for improving fine-tuning? For instance, could the mean local dimension be used as a regularization term in the loss function to explicitly encourage dimensional compression and potentially accelerate generalization or prevent overfitting?\n- Exploring Local Heterogeneity: The core method computes a mean local dimension across the token sample. However, the motivation for using a local estimator is the heterogeneity of the embedding space. Have the authors explored analyzing the distribution of local dimensions further? For example, do certain token types (e.g., punctuation, stop-words vs. content-specific nouns) consistently occupy higher or lower dimensional regions, and how do these distributions shift during fine-tuning? This could provide even more granular insights than the mean alone.\n- Connection to Parameter-Efficient Fine-Tuning (PEFT): Methods like LoRA are known to be effective by operating in a low-rank (i.e., low-dimensional) update space. How do you hypothesize your findings on latent space dimension relate to the intrinsic dimension of the parameter updates in PEFT? Could a similar analysis be used to dynamically set the rank r in LoRA based on the geometric needs of the task, as hinted at by the related work from Ed-dib et al. (2024)?\n- Generative vs. Encoder Models: The experiments include an autoregressive model (GPT-2) but the analysis focuses heavily on encoders (RoBERTa, BERT). Are there any observed qualitative differences in how the local dimensions behave between encoder-only and decoder-only architectures during fine-tuning? For example, does the causal attention mechanism in decoders lead to a different geometric signature compared to the bidirectional context of encoders?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations of their work in Section 4.6. They are transparent about the computational complexity, the an-is an artifact of the optimization process which also leads to generalization, which is an interesting question worth exploring further. Dependency of the absolute dimensional values on hyperparameters (which makes it a “relative” measure), and the assumptions of the TwoNN estimator. This upfront discussion is commendable.\n\nOne constructive suggestion for improvement would be to briefly touch upon the missing causal link as a limitation. While they demonstrate a powerful correlation, explicitly stating that the causal mechanism (i.e., whether dimensional reduction drives generalization or is merely a symptom of it) is a topic for future work would further strengthen this already solid section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the internal mechanisms of large language models (LLMs) by analyzing the geometric properties of contextual latent embeddings, focusing on how fine-tuning alters model behavior. Motivated by the lack of unsupervised methods to understand LLM embedding spaces, the authors introduce a framework to measure local intrinsic dimensions using the TwoNN estimator, which captures the low-dimensional manifold structure of token embeddings. Their approach involves sampling token embeddings, computing local neighborhood dimensions, and analyzing shifts during training and fine-tuning. Experiments across dialogue state tracking, emotion recognition, and arithmetic tasks show that changes in local intrinsic dimensions predict training convergence, overfitting, and grokking, with reductions in mean local intrinsic dimensions correlating with performance gains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper is easy to follow. Its main contributions are clear.\n\n2. The geometric perspective is interesting. The use of local intrinsic dimension to analyze LLM dynamics offers a fresh and unsupervised approach to interpret model behavior. The method bridges intrinsic model mechanisms and geometric properties, providing insights beyond traditional supervised diagnostics.\n\nWeaknesses\n\nAlthough the paper's technical aspects are quite interesting, as a study on understanding language models, I believe the most important aspect is to test its effectiveness on modern models and key issues, which is currently lacking in this paper.\n\n1. The model evaluation is not comprehensive, especially regarding its relevance to recent modern LLM architectures, which remains questionable.\n\n2. As a rapidly evolving field, LLM evaluation is crucial. The datasets tested in this paper are limited, particularly lacking datasets from the past two years.\n\n3. The high cost of metric computation limits its practical applicability.\n\n4. The underlying assumption is TwoNN Assumptions, which assumes locally uniform sampling distributions. The authors do not detail how LLM embedding spaces satisfy it. Are there theoretical justifications for applying TwoNN to LLM latents?\n\nQUESTIONS:\n1. Intuitively, the local dimension might be related to tokens entropy. I hope the author can add some discussion on this, especially considering the recent surge in papers related to entropy and LLM reasoning, e.g.,  [1-2]\n\n[1] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning.\n\n[2] The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models.\n\n\n2. The paper connects local intrinsic dimensions drops to grokking, but the causal relationship is not explained. Why does a decrease in mean LID indicate generalization beyond training data? Similarly, it could be better to add more discussions on other main findings.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks at model, fine-tuning, overfitting, and grokking from a geometric perspective. By computing local intrinsic dimensionality estimates they show that fine-tuning reduces dimensionality specifically for the dataset which is fine-tuned. They also show grokking behaviour is related to reductions in local dimensionality, and that the point when reductions saturate indicates when training on a given dataset is exhausted.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper presents an interesting perspective on an important topic. Unlike a lot of interpretability work which uses behavioural or probing evidence, the work here works to link the geometry of model embedding spaces to model behaviours. Overall the paper is clearly written, and the formalisations are easy to follow. The paper has clear scientific merit and novelty.\n\nWith this being said, there are a few weaknesses. First it's worth noting that the results broadly include no statistical testing. For example, with the results in figure 2, the body text states \"the training mean local dimension starts dropping significantly for those runs that exhibit grokking. Observe that the timing of this drop coincides with the start of the increasing validation accuracy in Figure 2b.\" Correlating local dimension and performance would make this claim more clearly and attach to it a measure of significance. Similarly the first set of results assert a significant difference between local dimension for the three different datasets, and the last set of results relate local dimension over time to overfitting behaviours. All places where support from statistical testing would be welcome.\n\nSecond, the results in the main paper are on largely on Roberta which is comparatively quite a small model by today's standards. The grokking results are also on a two-layer 128 dimensional transformer. Given this and the computational complexity of the methods used it's unclear to me how generalisable the methods here are. Despite this the introduction stresses the importance of understanding large language models and cites work introducing models with billions of parameters. I think the paper would benefit from claims more focussed to the scope of the results.\n\nQUESTIONS:\nAs a formatting point, the visuals are currently quite small. In future versions I would appreciate if they could be made larger and floated to the top of the page.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}