Title: Correlation Dimension of Autoregressive Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: correlation dimension

fractal dimension

large language models

self-similarity

complexity

degeneration

hallucination

LLM evaluation
Abstract: Large language models (LLMs) have achieved remarkable progress in natural
language generation, yet they continue to display puzzling behaviors—such as
repetition and incoherence—even when exhibiting low perplexity. This
highlights a key limitation of conventional evaluation metrics, which
emphasize local prediction accuracy while overlooking long-range structural
complexity.  We introduce correlation dimension, a fractal-geometric measure
of self-similarity, to quantify the epistemological complexity of text as
perceived by a language model. This measure captures the hierarchical
recurrence structure of language, bridging local and global properties in a
unified framework.  Through extensive experiments, we show that correlation
dimension (1) reveals three distinct phases during pretraining, (2) reflects
context-dependent complexity, (3) indicates a model's tendency toward
hallucination, and (4) reliably detects multiple forms of degeneration in
generated text.  The method is computationally efficient, robust to model
quantization (down to 4-bit precision), broadly applicable across
autoregressive architectures (e.g., Transformer and Mamba), and provides
fresh insight into the generative dynamics of LLMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Correlation Dimension (CD) as a global-structure metric for autoregressive LLMs. Instead of measuring token-level loss, CD treats the log-probability vector sequence \n𝑥𝑡 produced during generation as a trajectory in high-dimensional space and estimates its correlation integral \n𝑆\n(\n𝜀\n)\n∝\n$𝜀^𝑑$\n . The slope \n𝑑 serves as a proxy for the effective dynamical degrees of freedom: low \n𝑑 signals trajectory collapse (e.g., repetition), high \n𝑑 indicates random noise, and natural language tends to stabilise around \n𝑑\n ⁣\n≈\n ⁣\n6\n ⁣\n−\n ⁣\n7. Experiments on Pythia checkpoints, six open LLMs, and GPT-3.5/4-mini demonstrate:\n\n1. a three-phase training curve (short-range learning → long-range capture → compression/overfit);\n\n2. sensitivity to synthetic grammar complexity (Lin-Tegmark);\n\n3. unified detection of repetition, disorder, and semantic sparsity;\n\n4. strong correlation (ρ ≈ 0.95) with HelloEval robustness scores during long-text “name-stream” stress tests.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The explanation is intuitive. Small distance → The interval [s, t) can be regarded as a \"skippable segment\", corresponding to text jumps of different scales (skip); the threshold size forms a hierarchy from word level to sentence level, which can cover the global complexity measurement characteristics to a certain extent.\nHigh dimension → The state trajectory occupies a higher dimensional manifold (complex/random); low dimension → The trajectory collapses to a low dimensional attractor (repetitive or pattern-poor). It can monitor phased learning and potential pattern collapse without increasing training overhead, providing signals for early stopping and model selection.\n2. The application of long text generation health has certain practical value. Online sliding window calculation d in the reasoning flow: If the continuous window d ↓ < 3 ⇒ trigger rewriting/temperature heating, and warn of \"deviant\" paragraphs. d also decreases for repetition, disorder, and information sparseness → simpler than multiple dedicated probes.\n\nWeaknesses:\n1. When there are lists or rhymed texts, the dimensionality difference between models is large and the explanation is not sufficient.\n2. Core estimator (Grassberger–Procaccia) is classical; novelty lies in application, not algorithm. Prior LLM work has used spectrum-based complexity metrics. A side-by-side ablation (same texts) would clarify relative advantages.\n\nQUESTIONS:\n1. How do you access the full logits of the API closed model for degradation detection & long text stress experiments? My understanding is that this indicator requires access to the full logits, and the API closed model is difficult to apply?\n2. The author shows through the RLHF model that the indicator can compare the same model version across the entire process of \"pre-training-instruction fine-tuning-RLHF\", but this is not entirely intuitive. Current reasoning models have complex reward rules injected into the reinforcement learning stage and have different generation strategies. I want to know what the actual impact of RL strategy and the so-called \"text structure\" on the \"complexity\" of the evaluation of this indicator is? Can you explain why the impact of RL strategy is limited in the experiment?\n3. If you randomly permute \n𝑥𝑡​ order (destroying long-range structure) while keeping unigram distribution, how does CD react? This would support the “global hierarchy” claim.\n\nLIMITATIONS:\nMulti-language and code generation have not been tested yet → The transferability of indicators is unknown.\n\nThe threshold range needs to be set manually, and there is currently no adaptive algorithm.\n\nAuthors should add discussion and mitigation on potential bias against low-resource languages with inherently repetitive morphology (e.g., language-conditioned normalisation).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhile perplexity is widely used as a metric for evaluating LLMs, it has limitations in capturing the long-range structural complexity of natural language, such as repetition when perplexity is low. This paper introduces a novel evaluation metric based on correlation dimension, a measure of self-similarity from fractal geometry, to quantify hierarchical and repetitive structures in language.\nThe authors demonstrate through experiments that this method can consistently detect various forms of text degeneration, including repetition, incoherence, and blandness, which have traditionally been difficult to quantify.\n\nSTRENGTHS AND WEAKNESSES:\n##### Strengths\n- This paper proposes a new interesting evaluation metric based on fractal geometry (correlation dimension) that captures long-range structural complexity and generation dynamics of LLMs, which conventional metrics fail to measure.\n\n- Authors demonstrate that the proposed metric can unify and quantify multiple types of text degeneration (e.g., repetition, incoherence, and blandness) that are hard to assess using existing methods.\n\n- The paper is well written and easy to follow.\n\n- The approach is theoretically grounded, taken from established mathematical frameworks such as fractal geometry and dynamical systems theory.\n\n##### Weaknesses\n- While perplexity is broadly applicable and robust across different datasets and floating-point precision formats (e.g., FP16), the correlation dimension may yield different optimal values depending on these factors, potentially limiting its general applicability.\n\n- To strengthen the generality and robustness of the findings, it would be helpful to show how the correlation dimension behaves on other text domains (e.g., WikiText-103, BookCorpus), and whether phenomena such as the three-stage evolution are also observed in these settings.\n\n- (minor) The abbreviation \"SEP\" (Stanford Encyclopedia of Philosophy) is defined only in the appendix. For readability, the full name should also be spelled out in the main text.\n\nQUESTIONS:\nPlease see the weaknesses listed above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new metric to evaluate LLMs. Their metric is derived from the self-similarity principle and enables the measurement of recursions in natural text. Their experiments demonstrate that this approach handles different types of text effectively, measuring their complexity while also serving as a tool to evaluate imperfections in LLM text distributions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This is an interesting paper addressing the highly important topic of benchmarking and understanding the quality and structure of LLMs. Its applicability extends beyond autoregressive models (where perplexity is available). The paper addresses many questions that arise during reading through a comprehensive range of experiments.\n\nWeaknesses: The primary question that remains unanswered for me is how to utilize this metric effectively and how to derive meaningful conclusions from it. While this is a specific concern, I believe it reflects how most practitioners who evaluate LLMs approach such tools. For instance, when evaluating Diffusion-based LLMs, there is no convenient method to assess text quality beyond employing numerous metrics (Zipf Coefficient, Self-BLEU, and other metrics depicted in Table 1) that yield disparate results. Consequently, practitioners must consider all these metrics collectively to gain meaningful insights. I have concerns that this metric operates in a similar manner.\n\nMore concretely, suppose I have an arbitrary model (for example, my hypothetical Turbo-Scientific-13B model), and I wish to determine whether it outperforms various Llama models of different scales (where the definition of \"better\" may be arbitrarily complex). Based on the current presentation, it remains unclear to me whether this comparison is feasible. I observe that different models converge toward a specific value of your metric when perplexity is low; however, in an arbitrary case, I lack knowledge of the appropriate convergence value. If my model's metric is higher or lower than the value to which other models converge, what does this indicate? Is it favorable, unfavorable, or indicative of something else entirely?\n\nI acknowledge that you may respond that this question has already been addressed in the paper. However, I would like to emphasize that if this is the case, the answer is not sufficiently prominent. I would appreciate a practical tutorial for practitioners on how to use and interpret this metric, particularly in the absence of extensive pre-existing evaluations (such as knowledge of the convergence value associated with decreasing perplexity across models).\n\nA second, more minor weakness is that it would be valuable to understand how this metric correlates with downstream performance quality. I believe this is ultimately our primary concern, and it would be interesting to determine whether this metric can provide insights into downstream performance.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel framework for quantifying the implicit textual complexity by leveraging “correlation dimension”, a concept from fractal geometry. In particular, the authors adapt the Grassberger–Procaccia algorithm [1] (originally formulated for Euclidean spaces) to the statistical manifold of probability distributions.  The core idea is to treat the sequence of next-word probability distributions (as produced by an autoregressive LLM) as a trajectory in this manifold, and to measure its fractal (self-similar) structure. Empirically, the authors compute the correlation dimension of language data using LLMs [2, 3 ,4].  They find a universal fractal dimension of roughly 6–7 for diverse natural languages (e.g., English, Chinese, Japanese, German) and genres, indicating pervasive global self-similarity. The finding of a common “language dimension” and the comparison to theoretical random processes are particularly noteworthy.\n\n*References*\n\n[1] *Characterization of strange attractors.* Grassberger et al. In Physical Review Letters 1983.\n\n[2] *Pythia: A suite for analyzing large language models across training and scaling.* Biderman et al. In ICML 2023.\n\n[3] *The llama 3 herd of models.* Grattafiori et al. arXiv 2024.\n\n[4] *Qwen2.5 technical report.* Yang et al. arXiv 2024.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. The theoretical demonstration is well-grounded. Extending the Grassberger–Procaccia algorithm to probability distributions is carefully justified.\n2. This paper is generally well-written and easy-to-follow.\n3. The authors explore on a novel perspective of language modeling by linking it to dynamical systems theory and fractal geometry.\n\n**Weaknesses**:\n1. The paper could more explicitly contrast with any related fractal or information-theoretic analyses of text (e.g., multifractal analyses, entropy measures) to highlight what is unique.\n\n2. The practical usages of proposed method are somewhat speculative at this stage. The paper reveals an interesting phenomenon (i.e., language self-similarity) but stops short of demonstrating clear downstream applications. For example, it is not yet shown how knowing this dimension improves language modeling or analysis tasks.\n\nQUESTIONS:\n1. The reviewer is curious about how would systematically disrupting long-range correlations (beyond simple word shuffling) affect the dimension?\n2. Would the correlation dimension framework extend naturally to structured representations (e.g., syntactic tree), and if so, do the authors foresee the dimension increasing or decreasing in that case?\n3. Could the authors relate the correlation dimension to other measures of linguistic complexity? For example, measures like entropy rate, mutual information scaling, or classical fractal exponents (Hurst exponent) have been used for text.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}