Correlation Dimension of Autoregressive Large Language Models
Authors: Xin Du
Kumiko Tanaka-Ishii
Venue: NeurIPS 2025 poster ()
Keywords: correlation dimension
fractal dimension
large language models
self-similarity
complexity
degeneration
hallucination
LLM evaluation
Primary area: deep_learning
TL;DR: We propose correlation dimension as a practical, model-agnostic metric that captures structural complexity and detects degeneration in large language model outputs beyond what perplexity reveals.

Abstract: Large language models (LLMs) have achieved remarkable progress in natural
language generation, yet they continue to display puzzling behaviors—such as
repetition and incoherence—even when exhibiting low perplexity. This
highlights a key limitation of conventional evaluation metrics, which
emphasize local prediction accuracy while overlooking long-range structural
complexity.  We introduce correlation dimension, a fractal-geometric measure
of self-similarity, to quantify the epistemological complexity of text as
perceived by a language model. This measure captures the hierarchical
recurrence structure of language, bridging local and global properties in a
unified framework.  Through extensive experiments, we show that correlation
dimension (1) reveals three distinct phases during pretraining, (2) reflects
context-dependent complexity, (3) indicates a model's tendency toward
hallucination, and (4) reliably detects multiple forms of degeneration in
generated text.  The method is computationally efficient, robust to model
quantization (down to 4-bit precision), broadly applicable across
autoregressive architectures (e.g., Transformer and Mamba), and provides
fresh insight into the generative dynamics of LLMs.

Correlation Dimension of Auto-Regressive Large Language Models
Xin Du Waseda University
Kumiko Tanaka-Ishi
Waseda University
duxinCaoni. waseda. jp
kumiko@waseda. jp
Abstract
Large language models (LLMs) have achieved remarkable progress in natural language generation, yet they continue to display puzzling behaviors-sune as repetition and incoherence- when exhibiting low perplexity, This highlights a key limitation of conventional evaluation metrics, which emphasize local predic- tion accuracy while overlooking long-range structural complexity. We introduce correlation dimension, a the epistemological complexity of text as perceived by a language model. This measure captures the hierarchical recurrence structure of language, bridging local and global properties in a unified framework. Through extensive experiments, we show that correlation dimension (1) reveals three distinct phases during pretraining, (2) reflects context-dependent complexity, (3) indicates model's tendency toward hallucination, and (4) reliably detects multiple forms of degeneration in generated text. The method is computationally efficient, robust to model quantization (down to 4-bit precision), broadly applicable across autoregressive architectures (e.g.. Transformer and Mamba), and provides fresh insight into the generative dynamics of LLMs.
measure of self self-similarity, to quantify
Introduction
Latest advances in large language models (LLMs) have demonstrated sophisticated capabilities, including mathematical reasoning and planning. These abilities suggest that LLMs internally pro- cess information through complex, nonlinear, and potentially hierarchical mechanisms [3), despite operating through simple, token-by-token by-ttoeen predictions. Understanding precisely how LLMs achieve such macroscopic behaviors from microscopic predictive steps remains important open question, essential for grasping the full potential and limitations of these models. Although previous research shows that minimizing next-token prediction loss- perp-eeeetxitt theoretically powerful (36), it still falls short in fully explaining unexpected model behaviors, such as hallucinations and repetitive, bland outputs, even when perplexity values are low. Thus, new tools for characterizing the complex behavior of LLMs are urgently needed.
Current evaluation approaches broadly fall into two categories. The first includes methods based on local textual properties, such as lexical or syntactic statistics (e.g., n-gram frequencies). These metrics intuitive and interpretable but often fail to capture semantic ambiguities deeper structural complexities. The second category consists of global metrics, such as mean perplexity or semantic similarity measures. While these methods provide comprehensive quantitative evaluations, they frequently lack interpretability and connection to underlying local textual properties. This divide between local interpretability and global comprehensiveness reflects the broader challenge of bridging microscopic (token-level) and macroscopic (long-range, structural) perspectives of LLM behavior.
In this work, we propose correlation dimension, measure drawn from fractal geometry and dy- namical systems theory, to bridge this gap. Correlation dimension quantifies self-similarity
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

fundamental characteristic of complex systems that exhibit invariant patterns across scales, Originally developed for analyzing deterministic chaotic systems [20), correlation dimension has since been
successfully adapted to stochastic processes and real-world complex phenomena 44 31 Applied to language models, correlation dimension effectively quantifies the intrinsic complexity and recurrence structure of generated texts as perceived by the model. For instance, texts with randomly shuffled words appear highly complex and yield high dimensionality, whereas simple repetitive patterns exhibit low dimensionalityy.
We specifically propose computing correlation dimension using sequences of next-token log- probability vectors, which are readily available for any autoregressive language model, Recurrences are defined via the Euclidean distance between these vectors. Unlike perplexity, correlation dimension reflects deeper structural properties of the text generation process. For example, a model experiencing mode collapse might produce contextually irrelevant but statistically plausible text, exhibiting low perplexity yet high correlation dimension. Conversely, models lacking adequate memory to handle rongrrange dependencies might show low dimensionality despite high perplexity, clearly indicating their limited structural comprehension.
Correlation dimension serves as both practical and theoretically grounded metric. It requires little computational overhead beyond perplexity calculations and runs at inference time, making it easy integrate into existing inference infrastructures such as vllm. At a theoretical level, provides nuanced insights into model behavior, revealing the long-range complexity structures that perplexity alone cannot uncover. Practically, offers a straightforward yet powerful method to evaluate model robustness, indicate potential problems such as hallucination, and capture subtle forms of degeneration such as incoherence, and blandness.
Throughout the paper, we illustrate how correlation dimension inherently connects local recurrence structures and global textual complexity (Section provides insights into intrinsic properties of texts and model behavior (Section 4. and effectively detects degeneration issues in long-text generation (Section 5 Overall, our approach naturally integrates interpretability and comprchensiveness, offering robust metric grounded in the fundamental properties of complex dynamical systems.
2 Related Works
2.1 Statistical Self-Similar Phenomenon in Language
Self-similar is a fundamental property observed various complex systems, characterized by invariant patterns across multiple scales. A special class of self-similarity, known as scale-free properties, refers specifically to patterns that are consistent when viewed at different scales (52].
Unlike precise scale-free structures defined mathematically (e.g., the Koch snowflake), statistical self-similarity denotes approximate scale invariance identified through statistical analysis of empirical data (49).
Statistical self-similarities have been widely documented in natural language, manifesting through
phenomena such Zipf's law [58] 37 40 for word frequencies, Herdan- law [26) 24 for
vocabulary growth, and long-ram range correlations [1]
14 50 9
for word occurrence or word rank
across text spans.
Previous approaches have explored statistical self-similarity and fractal dimension at the abstract semantic level, extending beyond traditional lexical-based metrics (e.g., word frequency). For instance, M [2] utilized latent semantic analysis (LSA) embeddings of text paragraphs [10] and discovered geometric self-similarity (fractal structures) within semantic spaces, estimating fractal dimensions approximately between 8 and 20. 44] also observed fractal patterns in word vectors with Box-counting dimension of about 20. These findings are based on static word vectors that do not vary with context, and therefore have limited utility for characterizing language models.
Recently, [3] investigated the self seff-simllarity of texts using autoregressive LLMs, and found consistent semantic self-similarity across multiple languages. However, the linguistic significance of the self- similarity and the explicit role of language models in capturing such phenomena remains unexplored. In contrast, this paper proposes a new method tailored for analyzing L.MMs, and provides a range of insights into the generative dynamics of LLMs.

---

Garsian IDD
But then towards the papol2 end, Newton added his new line of argument, which employed philosophical analylla togothes with some Experimental evidence to support the
Pytlis-125
10
Fakand--10B
(a) Local skips
53
V11.9-388
10
Newton's 's view that the actubily contains series of colors its "qualities,, Are these qualities "sensible" their presence can be detected Anly through the use of one more prisms but never thfoughh inspection means These are apt to strike us as
7.0
sunlight through ordinary
10
10-
10
Distance
(b) Long-range skips (whole sentences)
(c) Recurrence plot
(d) Correlation integral curves
Figure 1: Experimental results on the "Newton's 's Philosophy" article of the Stanford Encyclopedia of Philosophy. (a-b) Examples of textual skips predicted by the Pythia-12B model, illustrating both local and long-range recurrences the text. Segment of the recurrence plot for the log-probability vectors generated by the model; black dots indicate pairs of points within specified distance threshold. (d) Correlation integral curves six pre-trained language models (solid lines), compared to i.i.d. Gaussian noise in R 50000 (dashed lines).
Characterizing LLMs via Self-Similarity Previous studies have evaluated language models via
statistical similarii.. 47 compared generated texts from language models to natural texts across
five scale-free properties, revealing significant deviations in scaling exponents. Building this, [39]
introduced quantitative statistical tests to measure such deviations systematically and examined how
different sampling strategies (e.g., nucleus sampling vs. beam search) affect self-similarity. (6] set up a mutual-information scaling law for long-context language modeling.
Notably, 2 recently measured the fractal dimension and Hurst exponent of cumulative log-perplexity series, demonstrating that these scaling exponents correlate with downstream performance of LLMs. However, their approach primarily captures long-range dependencies in the model's 's predictive error sequences. In contrast, our method (detailed in Section] 3) structure inherent in language itself, uncovering fundamentally different insights about the model behavior and text complexity.
examines the intrinsic generative recurrence
Recurrence Structure and Correlation Dimension of Language
Natural language exhibits self--imilarrty across various linguistic scales, ranging from morphological, lexical, and syntactic to semantic structures and long-range dependencies, Unlike mathematically precise fractals, real-world self- self-similarity is typically approximate and statistical, manifesting through recurrent statistical patterns rather than exact repetitions.
The correlation dimension provides quantitative framework to measure such statistical self-similarity in sequences by analyzing their recurrence structure. A recurrence defined as event where the trajectory of system approximately revisits a previous state within predefined distance threshold e. As increases, more recurrences naturally emerge. Self-similar systems typically follow a power-law relationship between the recurrence frequency S(E) and the distance threshold e, formally defined as follows:
Given infinite sequence {xx) in metric space (e.g., To its correlation dimension is the exponent characterizing the sculing behavior of the correlation integral S(e):
Definition 1 (Correlation dimension 20
S(E)
as --00 0,
(I)
where the correlation integral S(E) is defined as the frequency of point pairs separated by distances less than E:
S(s) = lim 2
141
- Σ 111, - x,,1
<e}-
t000 E(t 11
ISKjS!
Here, ||:I1 denotes the Euclidean norm, and 1(-} is the indicator function that equals 1 if the condition true and 0 otherwise.
Correlation dimension was originally developed to characterize deterministic attractors (e.g., Hénon maps 25 but subsequently generalized for analyzing stochastic processes such as fractional Brown- ian motion [7 and complex networks (3

---

Correlation Dimension Applied to Language. Representing text as sequence of numerical vectors allow the identification and measurement of recurrences and subsequently their correlation dimension. However, selecting meaningful, stable, and interpretable representation for natural language is nontrivial.
We propose utilizing the Euclidean distance between logarithmic next-token probability vectors derived from language models. Specifically,, the prg-obbbabiltyy vector time denoted Tt. is calculated as:
ny (w) = log Po(we = wlirees" w Ye E S2
where W; represents the token at position t, Pe represents the model-predicee probability (0 specifies the model), and 52 the vocabulary set. Unless otherwise specified, the context length c is set to infinity, i.e., no context length limit; its impact is further examined in Section 4.2 and discussed in Appendix A
Textual Skips as Recurrences. Recurrences in next-token log-probability vectors can be interpreted as textual skips. If two states 24 and 2s (s < t) are close, the text segment (s. t) could theoretically be omitted without significantly altering subsequent text generation. Skips occur at multiple scales, as illustrated Figure aa-b): from local (single-word skips) to global (sentence-level skips). Smaller distance thresholds (e) identify local skips, while larger thresholds detect longer-range skips. The hierarchy aligns naturally with Chomsky's s generative grammar [7]. where such skips may correspond to omitted subtrees within the hierarchical structure of language.
Figure (c) provides an example recurrence plot based on log-probability vectors from the text Newton's s Philosophy [29), clearly visualizing recurrences at a certain threshold. The correlation integrals S(e), illustrated in Figure [d), d), demonstrate near-linear scaling for multiple pre-trained language models (GPT2 [43], Pythia [5). Falcon3 [51], OpeALLaMA [I9], Yi1.5 [54], and Mamba
(23). ). yielding a consistent correlation dimension around 7. In contrast, Gaussian random vectors in equivalent dimensional spaces exhibit distinct different scaling behavior, underscoring language- specific recurrence structures.
3.1 Sufficiency of Next-Token Log-probabilities
The next-token pe--robabblity vectors represent only partial information of language model's full state, defined theoretically as the distribution over all future token sequences. Thus, one might question the sufficiency of next-token probabilities alone for characterizing long-range text generation.
Time-delayed embeddings, motivated by Takens' embedding theorem [48] and its stochastic ex- tensions [45] (46), are common methods for reconstructing full states from partial observations. Such embeddings concatenate multiple sequential log--roballe probability vectors, 3/2 = [xy] #1+1; [t+k], potentially capturing higher-order dependencies.
However, empirical observations reveal negligible differences in the correlation dimension when comparing simple next-token log- bog-probability vectors (A: = 1) to embeddings with higher orders (k: > 1), after accounting for the noise inevitably introduced the delayed embeddings. This some- what surprising result suggests that single-step log-probability vectors inherently encode significant long-term structural information about language evolution. This phenomenon mirrors findings in knowledge distillation (27). where single-step probability distributions effectively summarize model knowledge. Additional details and analysis are provided in Appendix
3.2 Empirical Convergence of Correlation Dimension
Considering an ideal "perfect" language model capable of precisely predicting next-token distribu- tions, its correlation dimension would reflect intrinsic textual complexity. However, slight imper- fections in predictions significantly affect correlation dimension measurements, particularly due to rare outcomes influencing the measure disproportionately compared to typical loss calculations. Pre- cisely, let ((0) Eugg. log Po Po
be the cross-entropy loss function of a language model parameterized by 0, where IP and Po represent the empirical distribution and the model-predicted distribution, respectively, and wige = [w1, ... [ار] represents the sequence of tokens until time I.
Then, can see that the contribution of rare outcomes w to the total loss is very small, proportional

---

Others
Qwen
Falcon3
>0 Random texts
8
GPT
Llama3.2
Qwen2.5
OPT
Yi-1.5
OpenLLaMA
7
34B
10B
Natural language languages)
6.5
Programming language (C, Java, Python)
6
1:5B
Trigram
N-grams
Bigram
Simple repetitoon
100 Negative Perplexity
Polya urn model
Figure Left: Mean correlation dimension of dataset measured with various pre-trained LLMs. Right: A conjectured spectrum of correlation dimension for different types of texts / language models.
to its frequency:
01(0)
F(wlagr-1)
Ve E Ω.
(4)
i log Pe (e(www(::1)
In other words, the log- bog-probability vectors at rare words w may vary significantly, without affecting the loss function, but this variation will be reflected in the correlation dimension regardless of the rareness of the word.
Nonetheless, empirical experiments indicate convergence in correlation dimensions of well-trained language models narrow range as perplexity decreases. Figure 2 illustrates correlation dimension for various pre- ire-trained LLMs across the Stanford Encyclopedia of Philosophy (SEP) (55] dataset which is summarized in Appendix C As the perplexity decreases, correlation dimensions stabilize around a consistent value near 6.5. This empirical convergence underscores correlation dimension's ss reliability as stable, interpretable metric for evaluating the structural complexity of natural language and the performance of language models.
The right half of Figure 2 presents the spectrum of correlation dimension values across various types of texts and statistical processes. Randomly shuffled texts exhibit high correlation dimensions, typically above ten, whereas self-reinforcing processes such the Polya urn model [35] display much lower values, below two. n-gram processes with small also yield lower correlation dimensions compared natural language. Moreover, programming languages (Python, Java, and C) show consistent correlation dimension around 5. We provide the full results in Appendix C.2 (other natural languages) and C.3 (programming languages).
Characterizing Language Models Using Correlation Dimension
Estimating the correlation dimension of text using language models raises critical questions re- garding the reliability and interpretability of this measure when the underlying model is imperfect or insufficiently trained. Ideally, with a perfect language model, the correlation dimension would repre- sent the intrinsic complexity of the text itself. However, real-world language models inevitably contain imperfections, prompting us investigate whether the correlation dimension remains meaningful under such conditions.
In this section, we demonstrate that the correlation dimension is robust measure of the perceived complexity of texts by LLMs, even when the LLMs are imperfect insufficiently trained. Specifically, we explore three key aspects: (1) how the correlation dimension reflects the intrinsic complexity of
a text as determined by its underlying hierarchical structure (Section 4.1): (2) how the correlation dimension is influenced by constraints on the contextual information available to the model (Section
4.2). and (3) how three distinct stages emerge the training process LLMSs as indicated by the
correlation dimension, phenomenon not observed in perplexity (Section 4.3 After that, present
5

---

9
0.001
1-9
q
) G
10
0.01
0.03
G
S
0.2
1-q
0.3
G
10
q
0.5
10
" #10011000
10
10
10a
Distance
(a)
(b)
Figure 3: (a) Illustration of the Lin- Lin-Tegmark grammar generation process gaverned by the transition matrix G, parameterized by (b) Correlation integral curves for texts generated with different q values, measured using the OpenI.LaMA-13B model.
a case study of occasions where significant divergence of correlation dimension observed between different models on knowledge-intensive texts, and how this divergence indicates LLMs' hallucination behavior (Section
4.1 Correlation Dimension and Textual Complexity
To analyze how correlation dimension captures inherent textual complexity, we generate sequences
using the Lin- -egmaar grammar [33], a probabilistic context-free grammar defined over binary alphabet {0, 1}. The grammar's complexity is parameterized by single Bernoulli parameter
10, as shown Figure a)) controlling the mutual information decay between tokens: G
q 1-4 4) where Gij denotes the probability of generating child node valued j given
1-q 4
parent node valued i. The resultant text complexity varies from highly predictable sequences (as approaches or 1) highly unpredictable sequences (as q approaches 0.5).
Figure ()) illustrates the correlation integral curves computed by the OpenL 13B model for different values of g. As q increases towards 0.5, the correlation dimension increases markedly from near zero to values above ten, demonstrating its sensitivity to textual complexity. Despite the nonlinear curves due the model not being explicitly trained on this grammar, the correlation dimension consistently captures the inherent complexity defined by the grammar.
Effect of Contextual Constraints on Correlation Dimension
We next explore how limiting contextual access affects the correlation dimension measured by pretrained language models. Varying the context length parameter c in Eq. (3) imposes restrictions on the model's available context, directly influencing its complexity perception. A context length of e = 1 reduces the model effectively to a bigram approximation, while longer contexts progressively enable deeper linguistic comprehension.
Figure a) shows correlation integral curves for the Pythia- -BB model measured on the SEP dataset at varying context lengths. Figure 4 (b) depicts average correlation dimensions against perplexity for multiple models (Pythia-Ie IB [5]. Qwen2.5-1.5B [53], and Llama3.2-1B (2I)). Notably, we observe a two-stage pattern: initial correlation increases sharply from approximately 3 to about 8 context length extends to 32 tokens, followed by a gradual reduction to around 6.5 at longer contexts.
This pattern indicates that initial increases in context length enhance the model's perception of complexity, revealing more contextual variation. Subsequently, the model identifies redundant contextual variations, compressing perceived complexity and converging to stable dimension around 6.5. This trend consistently appears across different models, validating the correlation dimension as reliable metric of the complexity perceived by language models.

---

100
10
Llama3.2-1B
Qwen2.5-1.5B
16
10-2
c= 2
Pythia-
64
S(e)
c
128
10
10-6
128
512
1024
c = 512
2048
10-8
10
10a
10
10
Distance E
Negative Perplexity
(a)
(b)
Figure 4: (a) Correlation integral curves on an article from the SEP dataset, measured by Pythia- IB model at different context lengths. (b) Mean correlation dimension the SEP dataset (vertical axis), measured using three models at different context lengths, with respect the negative perplexity (horizontal axis).
4.3 Three-Stage Evolution in LLM Pre- Tre-Trainnn
Pre-training language models typically focus on monotonically reducing perplexity, implying a linear improvement. However, find that correlation dimension reveals distinct, nonlinear evolution consisting of three stages during training.
We monitored correlation dimension across multiple checkpoints of various models, including the Pythia family (12B, 2.8B, IB, 160M, 14M), Ope (3B, 13B), Amber-7B (Figure 5). The evolution distinctly unfolds in three stages: (1) an initial rapid drop in correlation dimension due to learning short-range (bigram-level) structures, (2) a subsequent increase as models begin capturing longer-range dependencies, and (3) final gradual decline indicating improved generalization via context compression. This three-stage evolution is clearly observed in the Pythia family, for which the early-stage checkpoints are available; for the other models, the evolution is verified in the last stage. Notably, these shifts occur even though perplexity continuously decreases.
Interestingly, the third stage, indicative of improved generalization, does not universally occur. Smaller models, like Pythia- 14M and -160M, instead exhibit sudden increase in correlation dimen- sion at later stages, correlating with degraded performance in contextual learning tasks. Near the end of the training process, the correlation dimensions of these models exhibit a sudden rise to around In the lower half, show the accuracy of the two models in simple in-context learning task, where the model is asked to repeat a sequence of symbols. A clear correlation is seen between the accuracy and the increase in correlation dimension, suggesting the loss of generalization ability of the model. This observation underscores the potential of correlation dimension indicate generalization failures and stability in language models, insights inaccessible through perplexity alone.
10
1000
16000
16
13000
143000
8
143000
16
256
1000
4100
Pythin- 12B
4
256
Pythin-28B
Accuracy (%)
100
Pythis-1B
25p
Pythia- 160M
OpenkLaMAA33
Pythia-141
LaA
4
Amher-78
10'
-10"
-10"
110
-10
-10
109
100
10
10
10'
Negative Perplexity
Negative Perplexity
(a)
Figure (a) Evolution of the mean correlation dimension for various language models SEP dataset during pre-training. (b) Notable case of two smaller models (Pythia-14M 14M and 160M) that show marked increase in correlation dimension (top) near the end of training, coinciding with significant drop in in-context learning accuracy (bottom).

---

4.4 Hallucination vs. Memorization: Diverging Correlation Dimension on Knowledge-Intensive Texts
When processing texts containing domain-specific knowledge. LLMs may exhibit markedly different correlation dimensions, depending on whether the model truly recalls the knowledge or hallucinates producing syntactically valid but factually incorrect text. A model that successfully retrieves knowledge from memory tends to show higher correlation dimension, whereas a model that hallucinates typically exhibits lower correlation dimension such texts.
30
We present case study using the SEP article "process-theism." which contains a long list of relatively obscure scholars' names. Table I compares the correlation dimensions of the Qwen2.5 and Falcon3 model families on the entire SEP dataset (second column) and on the specific "process-theism" text (third column). Furthermore, we asked each model to complete the name list in the text and assessed whether the model successfully recalled the correct names hallucinated; the results are summarized in the fourth column (see Appendix C.4 for details).
As shown in the table, Falcon3-7B (6.68) and Falcon3-10B (8.49) exhibit substantially higher correlation dimen- sions than the other models. Model size alone the determining factor: al- though Qwen2.5-32D has far more pa- rameters, its correlation dimension re- mains low (4.42). A clear relationship emerges between correlation dimension and the model's ability to recall the cor-
Table 1: Comparison of correlation dimension between Qwen2.5 and Falcon3 text "process theism".
Normal text (ave.)
Knowledge- intensive text
Recalling or Hallucinating
Model
Qwen2.5-0.58
5.88
3.32
hallucinate
Qwen2.5-7B
6.27
3.56
hallucinate
Qwen2.5-32B
6.32
4.42
hallucinate
Falcon3-1B
6,03
3.28
hallucinate
Falcon3-3B
6.11
3.14
hallucinate
rect ganes-every model with a corre- lation dimension below 5.0 produced hallucinations this task.
Falcon3-7B
6.55
6.68
recall
Falcon3-108
6.56
8.49
recall
These results suggest that LLMs may internally signal whether they are hallucinating, and that correlation dimension provides a quantitative measure of this tendency. A possible interpretation is that recalling factual names requires the model to engage long-range dependencies, resulting in a high correlation dimension. contrast, hallucination relies primarily on format-level imitation. leading to a markedly lower correlation dimension.
Correlation Dimension for Quantifying Degeneration in Text Generation
A critical challenge in language model
Table 2: Schematic summary of various LLM evaluation meth- ods (rows) and their ability detect different types of degen- eration (columns). Triangles indicate potential usefulness that has not heen empirically validated.
generation is maintaining coherence, di- versity, and relevance throughout ex- tended text sequences. A well-known phe- nomenon, termed degeneration, describes scenarios where texts become repetitive, incoherent, or bland 28 While ex- plicit repetition is easily detectable, sub- tler forms of degeneration such incoher- ence blandness lack clear definitions or reliable quantification methods.
Metric
Repetition
Incoherent
Bland
Local generation probability
Perplexity Cond. Entropy
A
X
A
X
A
Word or N-gram statistics
Zipf Coefficient
58
A
In this section, we propose using corre- lation dimension a unified metric to quantify degeneration. Conceptually, de- generation is viewed as sudden col- lapse from a higher-dimensional trajec- tory in the model's state space into diver-dimensional attractor. Such col- lapses are generally irreversible, mirror- ing the boundary crisis phenomenon in chaotic dynamical systems (22). Table
Heap Coefficient
24
A
Rep-N 28
Distinct-N 32
X
BLEE 57
X
A
Semantics
BERTScore 56
X
A
MAUVE 41
A
CorrDim (ours)
compares several popular evaluation methods (rows) against specific degeneration types (columns),

---

highlighting correlation dimension's 's unique capability to detect all considered forms: repetition, incoherence, and blandness.
5.1 Repetition Detection: Semantic vs. Surface-level
Texts that exhibit repetitive patterns tend to have low correlation dimensions. As shown in Table
Table 3: Correlation dimension for repetition detec- tion, compared with lexicon-based metric (Rep-N).
3 (upper half), the correlation dimensions of ex- plicitly repetitive texts fall below 2.0 far lower than those of normal texts (around 6.5). Tradi- tional repetition-deetectio metrics such as Rep-N also identify these patterns through lexical statistics, as indicated in the rightmost column.
CorrDim (mean)
Rep-2 (mean)
Text
Normal texts (SEP dataset)
6.27
0.45
Explicitly repetitive patterns
1.83
0.99
10 Japanese novels
Unlike Rep-N, however, the correlation dimen- sion measures a text's semantic complexity rather than its surface-level word repetition. To show this distinction, we conducted a case study using Japanese, where texts can be written in two paral- lel script systems: (1) kanji (Chinese characters) combined with kana (Japanese phonetic symbols), and (2) kana only. The latter be derived from the former by replacing each kanji with its phonetic kana equivalent. Table 3 (lower half) shows that the correlation dimensions are highly consistent between the two scripts, even though the kana-only script has vocabulary roughly times smaller. In contrast, the lexicon-based metric, Rep-N, shows large differences between the two.
Normal script (kanji kana)
6.44
0.60
Syllabic script (kana only)
6.57
0.78
Mean relative difference
5.7%
29.8%
These findings demonstrate that correlation dimension is insensitive to superficial morphological variations and instead captures the intrinsic semantic repetition language.
5.2 Detecting Degeneration Beyond Repetition
In addition to repetition, correlation dimension be used to detect other forms of degeneration which are more subtle. To demonstrate this, we created a controlled dataset comprising responses twenty generic questions. Each question had ten normal responses and intentionally degenerate texts rexts-eepe,,,ive, incoherent, or bland-genenre generatedd by the GPT-4o model (details in Appendix D). We computed correlation dimensions using the Falcon3-10B model (51).
The results, shown in Table 4 indicate signifi- cantly lower correlation dimensions for degenerate texts compared to normal responses, confirmed by Wilcoxon signed-rank tests (all p-values < 0.01). These findings validate the correlation dimension's sensitivity to different degeneration modes, includ- ing those traditionally difficult to quantify, such as incoherence and blandness.
Table 4: Correlation dimension or perplexity of de- generate texts compared to normal texts, measured using the Falcon3-101 model, with p-values from Wilcoxon signed-rank test in the second column.
Group
CorrDim
Perplexity
mean
p-value
mean
Normal
5.04
10.79
For comparison, the table also reports the mean per- plexity (right-most column) of each group, mea- sured by the Falcon3-109 model. While perplexity can distinguish certain types of degeneration show much lower values for repetitive texts and higher values for incoherent texts compared to normal response- the direction of change is inconsistent across degeneration types. This indicates that perplexity is not an intrinsic measure of degeneration.
Repetitive
3,80
9,5E-7
1.25
Incoherent
3.96
2.9E-6
13.24
Bland
4.51
1.1E-3
4.24
5.3 Stress- Testing Language Models with Random Texts
To assess model susceptibility to degeneration, we designed stress-test involving random texts specifically, a list of randomly generated English names separated by commas, totaling approximately 1024 tokens. Models were tasked with completing the sequence from varying input lengths (n), generating 1024 - tokens. Random sequences typically have high initial correlation dimension due to unpredictability; thus models prone degeneration exhibit abrupt drops in dimension as repetition or incoherence emerges.

---

Figure 6 illustrates correlation dimension trends for three models as the input length increases (and the output length 1024 decreases). Robust models like Yi1.5-34B consistently increased their correlation dimension, whereas weaker models such as Qwen2-7B-Instie displayed marked dimension drops, indicating degeneration.
Further quantification is presented in Table which reports mean correlation dimensions at input length 512 alongside scores from the HelloEval lext-completion benchmark [42]. A high correlation (Spearman's intrinsic measure of model robustness in generating long, coherent text without relying on complex evaluation tasks.
0.952) is evident, underscoring correlation dimension's 's validity as an
Table 5: Mcan correlation dimensions gener- ated texts each model, compared with the Hel- loEval text completion scores.
Qwen2-7B8 Boven227B-instree
GLM--9B-
10
Yr1.5-34B
Model
Evll
CorrDim
Qwen2-77 IB-Inrtttnttt
5.12
3.54
Llama3.1-81
-5.61
4.15
InterLLM..577B
6.39
4.61
Mistral-7B-v0.2
13.05
5.23
GLM4-9B -B-tht
12.32
6,45
256
512
768
1024
LongWriter WWriter-GGLM--99
17.67
8.01
Input tokens
IntemL.M.55220
36.68
8,89
Yi1.5-34B
44.73
8.89
Figure Correlation dimension generated texts with respect to the length input texts.
Spearman's 's
0.952
Practical Issues
Efficient Calculation Computation of the correlation integral in Eq. ① requires evaluating pairwise distances between log-probability vectors. This operation entails O(N2) additional memory and O(N²Ω) computational cost, where N (~ 10a) and Ω (~ 10a) denote the sequence length and vocabulary size, respectively. To reduce the cost, we employ two techniques GPU kernel fusion and vocabulary reduction (see Appendix -hhich together achieve more than 10x speedup and incur zero additional memory overhead beyond standard LLM inference.
Inference Precision Modern LLMs typically operate at very low precision during inference to improve memory efficiency and throughput, with parameters often quantized to fewer than bits. Remarkably, the correlation dimension remains highly stable even under such extreme quantiza- tion, provided that Euclidean distances between proobbbbbyy vectors are computed in FP32. In experiments with GPTQ [18] and AWQ (3) quantized models, the average change in correlation dimension across the SEP dataset was below 3%. See Appendix B.3 for detailed results.
Closed Models Our approach requires access to an LLM's full logits, which is typically unavailable in closed models such GPT-4. Nevertheless, because the calculation of correlation dimension relies solely on log-probabilitte that are already produced during inference and requires no additional memory, believe this measure be readily integrated into standard commercial APIs.
7 Conclusion
We presented correlation dimension as a principled, model-agnostic metric for characterizing the long-range structure of texts as perceived by LLMs. By operating on next-token log pooobbbbyyyyity vectors, bridges local recurrence with global complexity and runs efficiently at inference time. Empirically, correlation dimension (i) reflects intrinsic textual complexity, (ii) exhibits predictable stage evolution during pre-training and under context constraints, and (iii) reliably detects degeneration (repetition, incoherence, blandness) beyond perplexity. These findings indicate that correlation dimension complements standard metrics by revealing aspects of model behavior that local accuracy alone misses. Future work includes formal analysis of estimation properties, extensions to conditional or multi-modal settings, and deployment as online signal for training diagnosties and generation control.
10

---

Acknowledgments
This work was supported by JST CREST, Japan, Grant Number JPMJCR2114.
References
[1] Kanjiconv: A tool for converting between kanji and kana, October 2025.
[2] Ibrahim M Alabdulmohsin, Vinh Tran, and Mostafa Dehghani. Fractal patterns may illuminate the success of next-token prediction. Advances in Neural Information Processing Systems, 37:112864-11288 2024.
[3] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 1, learning hierarchical language structures. arXiv preprint arXiv:2305. 136733, 2023.
[4] Edouard G. Altmann, Giampaolo Cristadoro, and Mirko D. Esposti. On the origin of long-range correlations in texts. Proceedings of the National Academy of Sciences, 109(29):11582-11587 2012.
[5] Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling, In International Conference on Machine Learning. pages 2397 -430.. PMLR, 2023.
[6] Zhuo Chen, Zhuotao Jin, Di Luo, Marin Soljačić, et al. L2m: Mutual information scaling law for long-context language modeling. arXiv preprint arXiv:2503, 047050 2025.
[7] Noam Chomsky, Aspects of the Theory of Syntax. Number 11. MIT press, 2014.
[8] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16639) 2022.
[9] Enrico De Santis, Giovanni De Santis, and Antonello Rizzi. Multifractal characterization of texts for pattern recognition: On the complexity of morphological structures in modern and ancient languages. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):00143-10166 2023.
[10] Scott Deerwester, Susan T Dumais, George W Furnas, Thomas Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391-407 1990.
[11] Isidoros Doxas, Simon Dennis, and William Oliver. The dimensionality of language. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 29, 2007.
[12] Isidoros Doxas, Simon Dennis, and William L Oliver. The dimensionality of discourse. Pro- ceedings of the National Academy of Sciences, 107(11):4886 4871, 2010.
[13] Xin Du and Kumiko Tanaka-Ishii. Correlation dimension of natural language in statistical manifold. Physical Review Research, 6:L022028. May 2024.
[14] Werner Eheling and Alexander Neiman. Long-range correlations between letters and sentences texts. Physica A, 215:233-241, May 1995.
[15] Werner Ebeling and Thorsten Pöschel. Entropy and long-range correlations in literary English. Europhysics Letters, 26(4):241-246 1993.
[16] Kenneth Falconer. Fractal geometry: mathematical foundations and applications. John Wiley & Sons, 2004.
[17] Patrick Flandrin. the spectrum of fractional brownian motions. IEEE Transactions on information theory, 3571):197-199 1989.

---

[18] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: accurate post- cost-training quantization for generative pre-trained transformers. CoRR, abs/2210.17323, 2022.
[19] Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023.
[20] Peter Grassberger and Itamar Procaccia. Characterization of strange attractors. Physical review letters, 50(5):346, 1983.
[21] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 240 2024.
[22] Celso Grebogi, Edward Ott, and James A Yorke. Crises, sudden changes in chaotic attractors, and transient chaos. Physica D: Nonlinear Phenomena, 7(1-3):181-20) 1983.
[23] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, In First Conference on Language Modeling, 2024.
[24] Harold Stanley Heaps. Information retrieval: Computational and theoretical aspects. Academic Press, Inc., 1978.
[25] Michel Hénon. A two-dimensional mapping with a strange attractor. The theory of chaotic attractors, pages 94-102, 2004.
[26] Gustav Herdan. The advanced theory of language as choice and chance. Springer- Seringer-Verlag New York, 1966.
[27] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural 2015.
network. CoRR,
[28] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[29] Andrew Janiak. Newton's 's Philosophy. Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2021 edition, 2021.
[30] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1-38, 2023.
[31] Lucas Lacasa and Jesús GadeeeGardenes Correlation dimension of complex networks. Physical review letters, 10(16))168703, 2013.
[32] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao. and Bill Dolan. A diversity-promootin objective function for neural conversation models. In Proceedings of NAACL-HLT, pages 110-119, 2016.
[33] Henry W. Lin and Max Tegmark. Critical behavior in physics and probabilistic formal languages. Entropy, 19(7), 2017.
[34] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device I1m compression and acceleration. In P. Gibbons, G. Pekhimenko, and C. De Sa, editors, Proceedings of Machine Learning and Systems, volume 6, pages 87-100, 2024.
[35] Hosam Mahmoud. Polya urn models. Chapman and Hall/CRC, 2008.
[36] Eran Malach. Auto-regressive next-token predictors are universal learners. In International Conference Machine Learning, pages 34417-34431. PMLR, 2024.
[37] Benoit Mandelbrot et al. An informational theory of the statistical structure of language. Communication theory, 84(21):48-502,, 1953.
12

---

[38] John M Marstrand. Some fundamental geometrical properties of plane sets of fractional dimensions. Proceedings of the London Mathematical Society. 3(1):257-302, 1954.
[39] Clara Meister and Ryan Cotterell. Language model evaluation beyond perplexity. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5328-5339, Online, August 2021. Association for Computational Linguistics.
[40] Steven T Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21:1112-1 12-1130, 2014.
[41] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816-4888 2021.
[42] Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, et al. Hellobench: Evaluating long text generation capabilities of large language models. arXiv preprint arXiv:2409. 16/91, 2024.
[43] Alec Radford, Jeffrey Wu, Rewon Child, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[44] Leonardo Costa Ribeiro, Américo Tristão Bernardes, and Heliana Mello. On the fractal patterss of language structures. Plos one, 18(5)o0225630, 2023.
[45] Tim Sauer, James A Yorke, and Martin Casdagli. Embedology. Journal of statistical Physics, 65:579-616, 1991.
[46] Jaroslav Stark, David S Broomhead, Mike E Davies, and J Huke. Delay embeddings for forced systems. ii stochastic forcing. Journal of Nonlinear Science, 13:519-577, 2003..
[47] Shuntaro Takahashi and Kumiko Tanaka-Ishii. Evaluating computational language models with scaling properties of natural language. Computational Linguistics, 45(3):81-511 2019.
[48] Floris Takens. Detecting strange attractors in turbulence. In Dynamical Systems and Turbulence, Warwick 1980: proceedings of a symposium held at the University of Warwick 1979/80, pages 366-381. Springer, 1980.
[49] Kumiko Tanaka-Ishii. Statistical Universals of Language: Mathematical Chance vs. Human Choice. Springer Nature, 2021.
[50] Kumiko Tsnaka-Ishii and Tatsuru Kobayashi. Taylor's law for linguistic sequences and random walk models. Journal of Physics Communications, 2(1 1):1 2111):155244, 15024, 2018.
[51] Falcon-LLM Team. The falcon 3 family of open models, December 2024.
[52] Stefan Thurner, Rudolf Hanel, and Peter Klimee. Introduction to the theory of complex systems. Oxford University Press, 2018
[53] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men. Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
[54] Yi. The yi model, 2024. hutpe://hhgginn/ccccc.oo//////- 2024.
visited in January
[55] Edward N. Zalta and Uri Nodelman. The stanford encyclopedia of philosophy, 2024.
13

---

[56] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
[57] Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 1097-1100, 2018.
[58] George Kingsley Zipf. Human behavior and the principle of least effort. Addison- Wesley Press, 1949.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims are supported by experimental results in Sections through
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There are no theoretical results in the paper.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The only hyperparameters used in the experiments the range of S(z) to estimate the slope, which is described in Appendix A All details of the dataset are explicitly described in the appendices.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use primarily open- soen-source datasets and models, and we provide sufficient information in the appendices for reproducing the experiments.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: There is no training in the paper. All LLMs used are publicly available.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We reported the p-values of the Wilcoxon signed-ranke test Section
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Resources required are presented in Appendix A
15

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification: have read the NeurIPS Code of Ethics and believe that the research conducted in this paper conforms to it.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We will not release any models or datasets that have a high risk for misuse.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the creators of the datasets and models used in the paper.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included details of the created datasets in the appendices.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We do conduct research with human subjects.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not conduct research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: LLM a core part of our method and we have described the usage of LLMs in the paper.
16

---

A Settings for Estimating Correlation Dimension
While correlation dimension is defined for an infinitely long sequence, finite number of steps available in practice. For finite sequences, there is no guarantee that the correlation sum S(z) follows power-law relationship with respect to E. Nevertheless, measured from a finite sequence still provides an informative description of the complexity of the LLM and the underlying language dynamics.
find that the correlation dimension
The correlation dimension d is obtained as the slope of log S(s) versus log e, and the observable range of log S(E), - depends on N. To make d comparable across sequences of different
lengths, we limit the range of S(=) to NIN-IJ+ B (n 1.0 by default), i.e., we clip the left tail of the log-log plot below ten counts and discard the upper half. When the range becomes too narrow for small N (i.e., N < 500), 1j is increased to maintain a reasonable range for slope estimation.
Short and Long Sequences We adopt slightly different settings for estimating the correlation dimension of long and short sequences. For long sequences, such as those in the SEP dataset, which exceed the context limit of typical LLMss, we use moving-window approach estimate next-token prediction probabilities with an exact, predefined context length (c.g., 512 tokens).
For short sequences, this approach would discard large portion of the data. Therefore, do not restrict the context length for short sequences. Since such sequences are typically within the model's context limit, the model and compute the correlation dimension using the output probabilities at all time steps. Thus, the log- probability vectors estimated with progressively increasing context lengths. This setting applies to the experiments in Section 5
access all tokens at once. In this case, input the entire sequence
In practice, the difference between the two settings is often negligible, and we do not differentiate the two settings in the main text for simplicity.
Synthetic Texts When measuring the correlation dimension of synthetic texts- sexts-specifically. the explicitly repetitive patterns described in Section - -we further restrict the range of the distance threshold values above 10 to prevent the correlation integral from being dominated by numerical errors. Because the synthetic texts follow exact repetitive patterns, the log-probability vectors lie very close to another small distance presholds-ttpicaant (with dimension greater than 10%). At such small thresholds, numerical errors in nigh-dimensional space can accumulate and dominate the correlation integral.
below 10a in high-dimensional space
B Efficient Calculation of Correlation Dimension
We introduce two techniques to enable efficient computation of the correlation dimension: (a) kernel fusion (Section B.T) and (b) vocabulary reduction (Section B.2). Kernel fusion achieves zero additional memory footprint beyond standard LLM inference and up to 3x speedup, particularly for long sequences. Vocabulary reduction decreases memory usage and computational cost by over 10x, with only minor loss in accuracy. These two techniques can be combined to achieve both benefits simultaneously.
In addition, we examine the effects of model quantization on correlation dimension estimation (Section B.3). We find that the correlation dimension remains remarkably stable across different quantization methods, even at extremely low precision (c.g., 4 bits).
B.1 Fused Calculation of Pairwise Distance and Correlation Integration
The computation of correlation dimension consists of two steps: (1) calculating pairwise distances between all log-probability vectors, and (2) counting the number of pairs within each distance threshold e to obtain S(s). Calculating the pairwise Fuclidean distances for a sequence of length N requires computing and storing distance matrix of size O(N2) high precision (e.g., FP32), which is prohibitively expensive for long sequences.
17

---

A straightforward approach to reduce memory usage is to divide the distance matrix into blocks and process them sequentially. The number of pairs with distances smaller than a threshold can be counted within each block and then accumulated to obtain the total count for the full distance matrix, However, after computing a block distance matrix, one still needs to copy it from high-speed local memory (i.e., SRAM) back to low-speed global memory (i.e., HBM or CPU memory) before performing step (2). When N is large, this copy operation becomes a major bottleneck.
Inspired by Ateeton g we propose fusing the two steps into a single CUDA kernel. After computing each block distance matrix, the number of pairs within distance threshold E directly computed in-place, without copying (blocks of) the large distance matrix out of SRAM.
Algorithm 1 Fused Blockwise Distance-and-Couml for Correlation Integral
Require: Log-probabilityy matrix X R NxD, thresholds (ex )
Ensure: Counts S(xx) for all A:
I: Initialize global counters S5 4-
2: for each index tile (t.j) with j S. do
3: Load X; and X, tiles into SRAM/shared memory
4: Compute pairwise distances des on-the-fty fly within the tile
Immediately compare de with {ex} and perform AtomicAdd to S. not write the tile distances dij back to global memory
7: end for
8: return {Sx}
Table 6: Runtime comparison between our method (bottom row) and four baselines on a log-probability vector sequence of length 50,000 and dimension 10,000. Distance computation performed in FP32 precision. For the blockwise baseline, we use block size of 512x512.
Methods
Additional memory
Clock time (s)
Speedup
torch. torch.pdist (entire matrix at once)
4.8 GiB
44.3
0.07x
torch, cdist (entire matrix once)
9.3 GiB
3.3
1.0x
torch. cdist (blockwise)
1.0 MiB
6.9
0.48x
torch. cdist (blockwise, upper triangular only)
MiB
0.92x
Fused (our method)
1.8
1.83x
Table 6 presents empirical comparison between method and four baselines using off-the-shelf PyTorch implementations. By fusing the two steps into a single CUDA kernel, our method achieves nearly a 2x speedup over the fastest baseline, while requiring no additional memory.
B.2
Vocabulary Reduction
A key property of fractal dimension is its almost-sure invariance under linear projection.
Theorem 2 (Fractal Projection Theorems [38 [6]). Consider fractal set embedded in R. with Hausdorff dimension (or box-counting dimension) d (S D). Given a random linear projection from RDD to T (m < D), then with probability the projected set's Hausdorff dimension d satisfies:
d= d fd. m,
if 2 d, imm d.
In other words, we can construct a linear projection that maps the log prg--rbbability vectors in R to lower-dimensional vectors, and estimate the correlation dimension using the reduced vectors. However, the thcorem assumes infinitely long sequences of points, which real data do not satisfy. Therefore, the choice of línear projection must be handled carefully.
We propose simple modulo-based function to group dimensions (i.e., unique tokens) in R and sum the vectors within each group to form smaller vector. The linear projection vh, RR + R is defined as:
(x);
Σ Kj
(5)
jEmod (8)
18

---

where x = [a1, ... [pp] is log proo-abbbllltty vector, (x); denotes the i-th element of the reduced vector, and mod, (i) is the set of indices j such that mod v = i. In practice, the vocabulary size !2 approximately 10".
article from the SEP dataset, where measured the correlation dimen- sion using the Qwen2.5-7B model with unlimited context length. As shown, using 10,000 achieves values very close the true correlation dimension, while reducing memory and computation costs by roughly 10x.
Table shows an example using the thewton-philosppe
Table 7: Change in correlation di- mension after vocabulary reduction.
Dimension
CorrDim
151643 (=Q2
5.92
30000
5.79
Although randomly selected linear projections are sometimes rec-
10000
ommended avoid caro-probabilitty exceptions in Theorem 2 find that such randomness introduces uncertainty and signihcant bias in correlation dimension estimates when the vocabulary size is large. Therefore, we prefer the deterministic modulo function defined in Eq. (5).
3000
5.87
1000
6.38
300
6.77
B.3 Effects of Model Quantization
Although the correlation dimension is defined in the limit as + O, observe that it remains remarkably stable under model quantization, in- cluding AWQ 34 and GPTQ [18], which com- press model parameters to bits. Table [rrrpotts the correlation dimensions estimated using quan- tized models for 60 articles in the SEP dataset. The second to fourth columns present the relation dimensions obtained with the original (FP16), AWQ-, and GPTQ-quantied versions of the Qwen2.5-322 3-32B-Innttuul model. As shown, both quantization methods yield only minor vari- ations in correlation dimension, with mean ab- solute change of just 0.14 across all articles.
Table Effects of model quantization on correlation dimension estimation. For each article in the SEP dataset. we measured the correlation dimension of the first 10,000 tokens using the Qwen2.5-33B-IIstrru model with unlimited context length.
Text
FP16
AWQ
GPTQ
aesthetics- aestheties-1Sth-germal africana
6.06
6.21
6.06
5.31
5,39
5.53
trinity weyl
5.60
5.66
5.66
6.57
6.62
6.45
Mean Mean Absolute Change
5.86
5.90
5.93
0.14
0.14
19

---

Supplementary Results for Sections
and
C.1
English Texts
The SEP Dataset We constructed the Stanford Encyclopedia of Philosophy (SEP) dataset by using the 60 longest articles on the SEP website [35]. The article identifiers are listed in the left-most column of Table / than 20,000 words. When measuring the correlation dimension, we truncated each article the first 20,000 tokens. This may introduce slight differences between models with different tokenizers, but the effect was small for most models examined.
For each article, we removed the prologue and catalog. All 60 articles are longer
10 prrocess-theiste
6
Qwen2.5-7B
Falcon3-10B
22
10
Qwen2.5-32B
Figure 7: Comparison of correlation dimensions of the SEP articles across three models.
Results for three representative models models--Qwen2.5---e Qwen2.5-32B, Qwen2.5-7B, and Falcon3-108 are shown in Table The second to fourth columns report the articles' correlation dimensions; the right-most column lists perplexities for Qwen2.5-32B. Correlation dimensions exhibit strong consistency across models, especially within the same family (Qwen2.5), as illustrated in Figure 7
The largest discrepancy in correlation dimension occurs for the article "process-theistee highlighted Figure ang shaded in Table This divergence reflects a qualitative behavioral difference between models snowledge-intensivee text: whether the model recalls knowledge tends hallucinate. We report this formally in Section 4.4 with additional details in Appendix C.4
20

---

Table 9: Results on the SEP dataset acquired from three pre-trained LLMs. Context size set to 512 tokens.
Article
CorrDim
Perplexity Qwen2.5-32B
Qwen2.5-32B
Qwen2.5-7B
Falcon3-108
misonee--dilemme
3.28
3.11
4.10
3.58
process-theism
4.42
3.56
8.49
5.68
cosmological-argument
4.93
5.26
5.30
5.83
information
5.22
5.28
5.41
6.31
ethics-chinese
5.39
5.07
6.18
5.52
recursive-function
5.40
5.76
5.88
4.96
reasoning-automaaee
5.48
5.32
5.44
5.06
infinity
5,51
5.33
5.48
5,68
buddhism-tiane
5.53
5.50
5.37
9.25
evil
5.64
6.01
6.37
4.69
logic-temporal
5.66
4.77
4.83
4.99
mohist-canons
5.72
6.06
6.14
3.75
linguistics
5.73
6.00
5.88
9.54
publichealth-ethe
5.79
5.96
5.77
8.87
mill-moral-por pollitcal
5.79
6.12
6.06
4.38
gasset
5.85
6.33
6.55
11.22
computational-compe complexxtty
5.91
5.95
6.90
5.32
africana
5.91
5.32
7.05
12.21
prihiipia-mathemaaica
5.93
5.39
6.17
5.36
levinas
5.96
6.14
6.73
15.92
formal-belief
6.03
5.99
7.19
5.82
saacetime-singularee
6.09
6.66
6.81
5.42
pegpuuaatiioallinnnussttt chaos
6.10
6.01
5.85
9.71
6.13
5.84
5.81
6.70
shaftesbury
6.17
5.93
6.14
8.62
possibilism-actualle
6.18
5.68
5.98
6.72
trinity
6.19
6.63
6.16
7.97
dynamic-epistemic
6.21
5.70
5.71
5.44
proof-theory
6.22
5.60
6.45
6.36
logic--for-games
6.30
5.52
5.53
9,92
game-theory
6.36
6.03
6.60
3,84
consciousness-temporll
6.38
6.13
6.83
8.63
sidgwick
6.40
6.55
6.78
10.32
ppistemie-game
6.42
6.31
7.54
5.43
amatencss-language
6.47
6.10
6.62
4,98
peilonyphilosophy
6.51
6.63
6.75
6.19
habermas
6.52
7.34
7.20
9.04
descartes-epistemology
6.62
6.56
6.32
6.48
russell-moral
6.65
6.62
6.98
7.04
mereology
6.67
6.96
6.71
7.71
comnectives-logic
6.68
7.39
7.99
6.63
weyl
6.75
6.87
6.15
5.62
cunsciousees--ineettomnait
6.78
6.62
7.69
9.57
attention
6.79
6.56
6.65
7.10
margaret-cavendish
6.81
6.81
6.75
8.16
gm-actionn-distance
6.83
7.12
7.18
4.70
normativity
6.84
7.06
6.01
5,63
saannology-bbyysiinne
6.84
6.94
6.95
5.94
aesthetics- Mertiicss--tt--eee 3eth-german
6.88
6.89
6.63
7.75
logic-inductive
6.97
6.76
7.16
5.10
chance-randomness
7.00
7.07
6.67
6.88
idealism
7.03
7.01
7.29
6.94
egalitarianism reid
7.07
6.20
7.46
3.14
7.10
6.34
8.32
5,43
early-modern -ndeer--iediia
7.15
6.35
6.74
4.70
heidegger- aesthetics
7.49
7.34
7.05
7.42
pythagoreanism
7.71
7.48
7.58
4.10
heidegger
7.84
8.69
7.59
6.59
al- Hl-farabi-metaphyyiii
8.33
9.31
8.70
6.04
ontological-commimennt
8.33
8.10
7.25
5.19
Mean
6.31
6.27
6.56
6.71
Pearson's p
0.90
0.63
-0.01
(w.nt. CorrDim of Qwen2.5-322
21

---

C.2 Other Natural Languages
In addition English, we conducted experiments on seven other natural languages: French, German, Spanish, Italian, Dutch, Chinese, and Japanese. Correlation dimension was measured using the multilingual Qwen2.5-7B model. For each language, we selected 10 books from Project Gutenberg. We used an unlimited context length and measured the correlation dimension on the first. 10,000 tokens of each book.
Table reports results for the eight languages. Each book is identified by its Project Gutenberg index, accessible at lnttps://ww/ wtt.s////ww.ttteberr......
toookks/bbok-tii idd where (book-id> the index. A consistent dimension of about 6.0 is observed for normal texts in all languages, suggesting language-independent universal complexity across natural languages. Small variations between books are primarily attributable to genre and style. For example, the German selection includes multiple philosophical works by Hegel and therefore exhibits higher values than the Spanish selection, which is primarily novels.
The dimension values slightly lower than those in Figure which is natural consequence of using an unlimited context length, as discussed in Section
Table 10: Corrolation dimensions of books in Project Gutenberg in different languages.
English
French Book
German
Spanish Book
Book ID
CorrDim
CorrDim
Book ID
CorrDim
ComDim
74
5.75
9643
5.51
34811
5.74
2000
6.07
76
13819
6.18
44921
6.56
25640
5.82
9830
4.70
13952
6.33
14075
6.03
44584
5.18
86
6.63
13951
6.69
23756
6.29
33275
7.07
215
6.76
14158
7.16
8126
5.70
37590
5.51
37106
14163
5,40
5.63
47092
3.41
805
5.08
796
6.17
6698
7.32
28281
5.28
996
5.52
14287
6.39
40739
4.95
49660
5.35
1156
5,90
9262
6.92
46259
5.74
17013
5.77
2701
6.08
5423
4.27
31114
7.31
14329
4.65
Mean
6.09
6.10
6.13
5.41
Italian
Dutch
Chinese
Japanese
Book ID
CorDDim
Book ID
ConDim
Book ID
CorDDim
Book ID
CorDii
46957
5.61
19591
4.71
23835
6.26
31757
6.47
20062
5.90
25138
6.07
23950
6.18
34013
6.36
10215
13214
6.42
23910
5.72
34636
6.15
48490
5.87
19563
6.38
24226
5.55
35327
6.87
43022
6.18
19774
4.87
27582
6.52
37626
6.36
46082
4.27
21875
6,97
23962
5.80
32978
6.77
43023
6.19
17706
6.13
25350
5,50
33307
6.35
43024
6.19
27124
5.66
25142
6.01
36459
6.53
48445
5.21
19161
5.50
24264
5.49
32941
6.44
19024
5.89
26564
5.02
23841
6.26
31617
6.14
Mean
5.63
5.77
5.93
6.44
22

---

C.3 Programming Languages
We also conducted experiments on three programming languages: Python, Java, and For each language, we selected 30-50 sufficiently long source files from their standard libraries. We used the Qwen2.5-Coder -B model measure correlation dimensions. Because many source files are relatively short, we used an unlimited context length instead of fixed one to avoid excessive truncation. In addition, we removed comments and docstrings to eliminate the influence of natural language content.
Results for the three programming languages are shown in Table 11 dimensions for all three languages are approximately 5.0-significantly lower than those of natural texts (around 6.5).
As shown, the correlation
Table Correlation dimensions programs written different programming languages: (left), Java (middle), and Python (right).
Java
Python
CorDii
Source code
CorrDim
CorDim
afsd_ _nff44xxr..
6.27
text DocimalFormat java
test, jest decimal.py
3.85
12is_super.e
4.71
util Collections.javs
5.53
Jos_
4.94
btrfs disk-io.
4.47
math. Minabbl-igInneggrjjss
4.23
test pickletestor.py
5.35
sepl__nds, cliente
4,08
lang Math.java muth. BigIntegerjava
pickletools.py
12is_ data.e
4.56
5.71
test lest, mulppppeeiinneeesnngpyy
4.86
file.
4.96
lang jhmmjava
test daterimetesterrpy
ext4 _al_naami.c
5,70
jaavacter,java
8.13
tkintor iniit_
4.99
otrss_extent_io.
4.92
util. AiLAAnayaajvo
4.22
doctest.py
5.15
xende
4.78
text_ SimpleDate/Fommal java
5.45
sesLJestünspper
4.49
refoccoocction
4.55
text CompaatNumbooFoomsst..o
4.92
jest_tcst_ooo codecss py
4.08
jis_jfs dmap.c
Formatter.jaa
email beader, putser_prrrrppy
hurfs inode.c
5.10
text, MessageForrat.jjo
4.36
test, Jest, dataclasses.pp
4.99
ocfs2 refcountreee.c
4.75
lang String, java
test Jest pos__essttttngpppppp typing-py
restrr_dttgrup...
5,33
util. HashMaa.jaaa
4.54
-yddcciall.py
4.41
namespace.e
5.19
uiil ResourveBundleejav
5.23
test_estt joLLiesL_eummppy
4.61
4.69
io File,java
5.12
inspect.py
6.08
jectle
4.89
io ObjectuuputStremm.
3.94
cot_iestosspp
4.64
frrss_rree-ppp stee--puce
sacce-cache. cache.c
5.15
Calendur.avva
test, Lest subpracess.py
nefs2 dimgluu.e
4.52
lang System.java
5.07
test test doctest.py
4.15
ols ji_nnl__u cp95cc
6.34
security KeyStore.java
3.94
pydoc data ternaatopicspppp
5.96
dlm_Jocks
4.37
time LocalDate java
5.71
3.94
ex14 rbblllocc.
5.08
lang ClassLonerrjava lang Alnttnctttrnnnuiildeerrrr
6.07
test MoLLlesLdess deseery
4.45
mifs__fllo.cc
4.92
5.47
test Jest prggarse..pp
6.16
burfs exteett-eett
4.97
util Spliterators
3.54
testiestt tesLiesttiipillpp
4.22
capscc
io_ ObjectInpuStrremmjjun
test Jest jest_looginngpy
volumeso
5.01
lang. Thread.java
5.70
turtle
6.69
btrfs, qgroup.c
4.48
net_ URL_java
4.32
test, jest unioodd py
5.55
burfs block-groupe.
5.21
util_ Scamner.java
4.38
test destlesttmll eane_etree.py
4.67
nfsd nis4staiee
5.40
util, GiegeorianCalemmaajjoo
4.70
test Jest buffergy test test statisties.py
3.95
ext4 extents.e
5.38
time Timee java
4.76
5.28
ocis2 xallre
4.60
lang StrictMath java
6.25
pyduc.py
5.60
5.09
DualPiverQuick java
unittest mock. py
nfs nfssAxdr
5.48
util Igpaneeefmprcillalldddee
4.65
test Cee_est__soo jest_socket.ppy
4.26
burfs etrecc
5.44
util Locale java
4.79
tree_tree-ooge
4.99
4.95
nameic
5.46
muth BigDecimal java
3.75
nfs nfs4proce
496
lang Jaavacterjjva
8.13
jfs jfs_jfs dtreec
5.08
util. TreeMap.jav java
4.46
nls cp949.c
6.50
lang Class
ext4_supee sct__superee
4.27
octs2_dir.
4.71
126s segmeetet
4.73
nls pll___p_____ cp936c
6.49
alk_nl__c932..e
6.12
Mean
5.07
Mean
5.10
Mean
5.00
C.4 Hallucination vs. Memorization: Experimental Details
In Section 4.4 and Appendix C we reported a notable divergence in correlation dimension between models on knowledge-intensive SEP article "process-theism." The text contains an extensive list of philosophers' names, many of whom are relatively obscure. We conducted simple experiment: we prompted the models to continue the name list and observed that those with lower correlation dimensions tended to hallucinate and generate non-existent names.
23

---

We began with the following input text and asked each model to complete the list:
Philosophers and theologians who have published a monograph defending some variety of process theism informed by Whitehead or Hartshorne include: Henry Nelson Wieman Bernard Meland, Paul Weiss, Norman Pittenger, Daniel Day Williams John Moskop. William L. Reese. John B. Cobb, Jr., Schubert Ogden. Edgar A. Towne, Eugene H. Peters, Bowman Clarke, Joseph Bracken, Burton Z. Cooper, Marjorie Hewitt Suchocki, Gene A. Reeves, Lewis S. Ford André Gounelle, Rem B. Edwards, (start generation)
The ground-truth continuation is:
Delwin Brown, David A. Pailin, Franklin I. Gamwell, Forrest Wood, David Ray Griffin, James A. Keller, Jorge Luis Nobo Tyron Inbody, Carol P. Christ, George L. Goodwin, Barry Whitney, Santiago Sia, Jay McDaniel, George W. Shields Donald Viney, Catherine Keller, Daniel A. Dombrowski, Anna Miistr,n Kurian Kachappilly, Gregory A. Boyd, Roland Faber, Thomas Jay Oord, Donna Bowman, Derek Malone-France, Monica A. Coleman, and Julia Enxing.
We report generation results for seven models: Qwen2.5-0.5B, Qwen2.5-7B. Qwen2.5-32B, Falcon3-1B 1B, Falcon3-3B, Falcon3-7B, and Falcon3- 10B. Names that appear in the gold answer are marked in green. These names did not appear in the input context; hence, they are not inferred from context but recalled from the model's internal knowledge. All models were evaluated with temperature of 1.0. We present representative generations that contain at least five names.
Generation Results
Qwen2.5-0.5B (CorrDim: 3.32);
G. K. Dugald Stewart, Robert J. Leary, E. F. Robinson, Bernard P. Rieff, Robert A. Miller, R. K. Gombrich, Alan Tucker, B. Williams V. Fowden, Brown, William E. Clark, David E. Stoloff, Eric Damer, James R. Miller, D. A. Burchardt, J.
Qwen2.5-7B (CorrDim: 3.56):
Robert Schofield, Robert McAfee Brown, Keith Yandell, Robert John Russell, John H. Cobb, Jr., Arthur Roberts. Robert Solomon, William E. Connolly, and Bruce L. Reichen- bach.
Qwen2.5-32B (ConDim: 4.42):
Eugene F. Rogers, Jr . C. Robert Mesle, Keith R. Davies, William S. Hatcher, Roland Faber, Thomas J. J. Altizer, Mark McSwain, Charles Hartshorne, and David Ray Griffin.
Falcon3-1B (CorrDim: 3.28):
Peter J. Haught, William McMillen, B. W. McMillen, Paul H. Metzger, Brian Joseph
Murphy, George E. Spivey, Peter L. Wurm, Carl W. Smith, William H. Miller, and C. S. Lewis.
Falcon3-3B (CorDDim: 3.14):
John C. Miller, Jr., Bernard Auchincloss, William L. McBride, Gerald L. T. Mitchell, George A. Meeks, William C. Schade, Thomas B. Allen, Jr., Michael W. Williams, David L. Boorse, John Henry Elliott, J. I. Packer. J. H. Oder, R. A. Butler, Bernard W. Miller,
Robert E. Houtman, William L.. Lane
Falcon3-7B (CorrDim: 6.68):
David Ray Griffin, C. Robert Mesle. David R. Tracy, Robert C. Neville, Roland Faber, Jay McDaniel, and John D. Capes.
Falcon3-10B (CorrDim: 8.49):
Delwin Brown, David Ray Griffin, Hank Keeton, George L. Robinson, George Reynolds, Franklin I. Gamwell, Alan I. Megill, Roland Faber, and Catherine Keller.
As shown, Falcon3-7B, and Falcon3-10B correctly recalled several names, whereas the other models generated plausible but non-existent ones. The Qwen2.5-32B model recalled two names correctly, but the two names more well-known and casier to recall than the other names. Therefore, in avoiding

---

"01" hallucination, model size alone is not decisive: Falcon3-10B achieved much higher precision than Qwen2.5-32B, despite being roughly three times smaller. We hypothesize that correlation dimension is better indicator of a model's s tendency to hallucinate in specific contexts- low correlation dimension suggests that the model's generation collapses into a simple, format-driven pattern.
D Repetition Detection: Experimental Details and Supplementary Results
D.1 Repetition Detection
Explicitly Repetitive Patterns have Low CorrDim Re We generated texts by explicitly repeating set of patterns and measured their correlation dimensions, as described in Section D.1 Results are shown in Table [2] The patterns to repeat are shown in the first column of the table, and the second column shows the mean correlation dimension of the repeated text. As shown, correlation dimensions of these explicitly repetitive patterns are about 1.5-2.5, significantly lower than those of normal texts (around 6.5).
dimension close to indicates that the LLM's internal state evolves like a random walk while retaining steady memory of previous states.
key consideration for correct estimation here is restrict the measurement range sufficiently large distance thresholds (Appendix A): for highly regular patterns, state distances become so small that numerical errors from -igh-dimesiional log-probability vectors dominate. This adjustment is typically unnecessary for normal texts.
Table 12: Repetition detection explicitly repetitive patterns.
Repetition Pattern
Mean CorrDim
2.17
"012"
2.29
"ab"
1.58
"4%""
1.94
"#X0"
1.73
""
1.79
1"
1.73
0 1 2"
1.87
b"
1.69
X"
1.71
Q"
1.71
# % * 1"
1.71
Mean
1.83
Japanese Scripts:
CorrDim Invariance under Kanji-Kana Conversion We compared two Japanese script systems. The first is the standard, peahhoorpphhi-plu--yylleee system, consisting of kanji (Chinese characters) and kana (Japanese phonetic symbols). In the second, all kanji are replaced by kana, so the entire text is written only in kana. The kana-only script has an order-of-magnitude smaller vocabulary.
We used the ten Japanese books listed in Table 10 from Project Gutenberg. Using the kanjiconv tool I. we converted each book from the standard script to the syllabic (kana-only) script.
Table 1 reports the correlation dimensions (second and third columns) of the ten books in both scripts, computed with the Qwen2.5-7B model. The dimensions are highly consistent between the two scripts, even though the kana-only version has a much smaller vocabulary and a higher repetition rate (higher Rep-N; fourth and fifth columns).
This suggests that correlation dimension captures semantic complexity rather than surface morpho- logical features, detecting semantic repetition rather than morphological repetition.
D.2 Generating Degenerate Texts
The twenty prompts used in the experiment in Section 5.2 are listed in Table 14
25

---

Table 13: Comparison between correlation dimension and Rep-N for two Japanese scripts.
Book ID
CorrDim
Rep-N
normal (kanji kana)
syllabic (kana only)
normal (kanji kana)
syllabic (kana only)
31617
6.14
6.72
0.57
0.76
31757
6.47
6.66
0.58
0.73
32941
6.44
5.76
0.66
0.84
32978
6.77
7.68
0.66
0.85
33307
6.35
6.44
0.58
0.75
34013
6.36
6.60
0.57
0.75
34636
6.15
6.61
0.59
0.77
35327
6.87
6.77
0.62
0.82
36459
6.53
6.28
0.65
0.82
37626
6.36
6.15
0.51
0.69
Mean
6.44
6.57
0.60
0.78
Table 14: List of prompts used in the experiment of degeneration detection.
No.
Prompt
1
Describe the primary goals of an effective team.
2
Explain the basic steps involved a standard project workflow.
3
Outline advantages of using modern technology in daily life.
4
Discuss the key features of a reliable customer service program.
5
Summarize the benefits of maintaining consistent work schedule. Describe how typical training session should be conducted.
6
7
Explain why clear communication is important in organizations. Outline the main characteristics of a successful leadership style.
8
9
Discuss the factors that contribute to smooth operational process.
10
Summarize the core principles of quality assurance.
Describe the role of feedback performance improvement.
12
Explain the importance setting realistic goals.
13
Outline the steps for conducting a standard evaluation.
14
Discuss how data is used to inform business decisions.
15
ummarize the benefits of structured planning approach. Describe the essential elements professional code of conduct.
16
17
"Explain how consistency can improve team efficiency.
18
Outline the main objectives of routine maintenance program.
19
Discuss the value of transparency in reporting results. Summarize the advantages applying proven best practices.
20
To elicit different types of degenerate texts with GPT-4o, provided distinct instructions and collected the model's responses as follows.
Normal Responses
You are a creative text generator. Your task to produce richly detailed, vivid, and engaging passage of at least 1,500 words (but no more than 2,000) that brings its subject to life with specific imagery, sensory detail, unique examples, and dynamic narrative. Avoid generic descriptions, stock phrases, or "safe" language. Instead, strive for:
Concrete specifics: name real or invented places, objects, characters, or processes.
Sensory richness: evoke sight, sound, smell, taste, and touch wherever possible.
Fresh metaphors and similes: craft original comparisons rather than clichés.
Varied sentence rhythms: mix short, punchy lines with longer, flowing sentences.
Emotional or intellectual hooks: give the reader something surprising, thought-provoking. or emotionally resonant.
Generate a single coherent text that alive and unmistakably your own.
26

---

Bland responses
You are text generator. Produce a bland yet coherent passage of 1,500-2,000 words that avoids exact repetition of phrases words.
Incoherent responses
You are text generator. Please produce richly worded, vivid, and non-repetitive passage (1,500-2,000 words) that is intentionally disjointed lacking logical coherence the reader feels the narrative fragmented.
Repetitive responses We used the responses generated under Normal responses and then extracted the first sentence, and repeated to form a passage of approximately the same length.
D.3 Random Text for Stress-Testing LLMs
For stress testing, we used a string of random names separated by commas as the input text. The names are common English names, sampled i.i.d. from a list of unique names. A portion of the resulting text is shown below:
Quinlan, Anthony, Henry, Felicity, Taylor. Raymond, Xander. Christopher, King, Amanda, Flora. Nicole, Anthony, Frank, Quiana. Owen, Finley, Paige, Victoria, Aaron, Ulrika, Sarah, Ignacio, Emily, Yuna, Imogen, Cameron, Claire, William, Preston, Ulrika, Sabrina, Ncil, Zara, Joseph, Orion, Vivian, Quinn, Wyatt, Paul, Sophia, Brian, Flynn, Hayden, Charles, Grace, Carter, Heather, Quest, Jacob, Jordan, Frances, Griffin, Yasmin, Quiana, Penelope, Emma, Sabrina, Elizabcth, Joseph, Zion, Quinlan, Omar, Ruby, Virginia, Ursula, Flynn, Alexander, Ian, Griffin, Frances. Yasmine, Warren, Isaiah, Ryan, Kyle, Xanthe, Lucy, Georgia, Gregory, Ophelia, Georgia, Patricia, Xiomara, Kayla, Finley, Zayden, Noah, Caitlin, Brittany, Connor, Quinton, Urban, King, Blake, Joshua,
E
Consistency with Time-Delayed Embeddings
potential concern with the correlation dimension estimation method in Section Bi that we used only the probabilistic information of the next tokens, which is not complete representation of the model's state. A state of a dynamical system is defined as point the phase space, which contains all the information that governs the future evolution of the system.
No delay
Delay
-3
Delay =
55
6.0
6.0
6.0
-7
1.5
2.0
2.5
3.0
Distance E
Figure Correlation integral curves for time-delayed embeddings with different delays (3 or 5), compared with that for the original sequence.
common method to reconstruct the phase space from partial observations to measure the dimen- sion time-delayed embeddings. While the theoretical effectiveness of time-delayed embedding is
27

---

guaranteed by the Takens' theorem and variants 48 45 46], the method is empirically sensitive to noise and the choice of the embedding dimension, failing to deliver satisfactory results except for simple, low-dimensional systems. In this work, we aim at characterizing language models that are very high-dimensional and random nature. For such systems, the time-delayed embeddings are often overwhelmed by observational noise and the dimension is overestimated.
Nevertheless, observe that the correlation dimension values exhibit good consistency even if the time-delayed embedding is used. For the log-probability vector times series r = 22, .. we acquired two time-delayed sequences (3) and (:(5) with delays of 3 and 5, respectively. For a delay
x. [ - al. i.e., the the concatenation of vectors from t f + 1. The correlation integral curves for these time-delayed sequences are shown Figure g While the time-delayed
embeddings have 3x 5x the dimension of the original sequence, the correlation integral curves similar to that of the original sequence at small e, except that the curves are shifted to the right. The slopes of the curves increased at large because of accumulated noise and is outside the range of interest.
This indicates that the next-token probabilities contain information beyond the next token, and is sufficient to use the next-token probabilities to characterize the dimension of the model's evolution.
28


Reviews:
[{"review_id": "4XfSSVKYyp", "reviewer": "Reviewer_Bw4t", "summary": "The paper introduces a novel framework for quantifying the implicit textual complexity by leveraging “correlation dimension”, a concept from fractal geometry. In particular, the authors adapt the Grassberger–Procaccia algorithm [1] (originally formulated for Euclidean spaces) to the statistical manifold of probability distributions.  The core idea is to treat the sequence of next-word probability distributions (as produced by an autoregressive LLM) as a trajectory in this manifold, and to measure its fractal (self-similar) structure. Empirically, the authors compute the correlation dimension of language data using LLMs [2, 3 ,4].  They find a universal fractal dimension of roughly 6–7 for diverse natural languages (e.g., English, Chinese, Japanese, German) and genres, indicating pervasive global self-similarity. The finding of a common “language dimension” and the comparison to theoretical random processes are particularly noteworthy.\n\n*References*\n\n[1] *Characterization of strange attractors.* Grassberger et al. In Physical Review Letters 1983.\n\n[2] *Pythia: A suite for analyzing large language models across training and scaling.* Biderman et al. In ICML 2023.\n\n[3] *The llama 3 herd of models.* Grattafiori et al. arXiv 2024.\n\n[4] *Qwen2.5 technical report.* Yang et al. arXiv 2024.", "questions": "1. The reviewer is curious about how would systematically disrupting long-range correlations (beyond simple word shuffling) affect the dimension?\n2. Would the correlation dimension framework extend naturally to structured representations (e.g., syntactic tree), and if so, do the authors foresee the dimension increasing or decreasing in that case?\n3. Could the authors relate the correlation dimension to other measures of linguistic complexity? For example, measures like entropy rate, mutual information scaling, or classical fractal exponents (Hurst exponent) have been used for text.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. The theoretical demonstration is well-grounded. Extending the Grassberger–Procaccia algorithm to probability distributions is carefully justified.\n2. This paper is generally well-written and easy-to-follow.\n3. The authors explore on a novel perspective of language modeling by linking it to dynamical systems theory and fractal geometry.\n\n**Weaknesses**:\n1. The paper could more explicitly contrast with any related fractal or information-theoretic analyses of text (e.g., multifractal analyses, entropy measures) to highlight what is unique.\n\n2. The practical usages of proposed method are somewhat speculative at this stage. The paper reveals an interesting phenomenon (i.e., language self-similarity) but stops short of demonstrating clear downstream applications. For example, it is not yet shown how knowing this dimension improves language modeling or analysis tasks.", "was_revised": true, "final_justification": "I have updated my score to reflect my decision, considering the sufficient rebuttal the authors have provided.", "rebuttal": "We sincerely thank you for taking the time to provide constructive feedback. We\nnotice that some of your remarks appear to reflect findings or content from a\ndifferent work, rather than the current one. To avoid any potential confusion,\nlet us briefly summarize the unique contributions of this manuscript:\n\n1. We show that the correlation dimension of text arises from multiscale \"skip\nstructures\" in the language generation process.\n\n2. We validate that the proposed method remains robust when applied to\nunder-trained models and we uncover multiple distinct phases of LLM pretraining\n(Section 4).\n\n3. We demonstrated that correlation dimension serves as a comprehensive\nindicator of degeneration in long-text generation, capturing repetition,\nincoherence, and blandness under a single metric (Section 5).\n\nWe hope this brief summary helps to clarify the focus of this submission. Below,\nwe respond to each of your comments in detail.\n\n\n--------------------------------------------\n> Strength: The theoretical demonstration is well-grounded. Extending the\nGrassberger–Procaccia algorithm to probability distributions is carefully\njustified. This paper is generally well-written and easy-to-follow. The authors\nexplore on a novel perspective of language modeling by linking it to dynamical\nsystems theory and fractal geometry.\n\nThank you for your encouraging feedback. We are pleased to hear that you found\nour theoretical foundation solid and the writing clear. Your recognition of the\nnovel connection between fractal geometry and language modeling is greatly\nappreciated.\n\n\n--------------------------------------------\n> Weakness 1: The paper could more explicitly contrast with any related fractal or\ninformation-theoretic analyses of text (e.g., multifractal analyses, entropy\nmeasures) to highlight what is unique.\n\nWe agree. In the revised manuscript, we will expand our discussion to include\ncomparisons with existing complexity measures, such as multifractal spectra,\nentropy rates, and spectrum-based metrics (as also noted by another reviewer).\n\n\n--------------------------------------------\n> Weakness 2: The practical usages of proposed method are somewhat speculative\nat this stage. The paper reveals an interesting phenomenon (i.e., language\nself-similarity) but stops short of demonstrating clear downstream applications.\nFor example, it is not yet shown how knowing this dimension improves language\nmodeling or analysis tasks.\n\nWe agree that demonstrating downstream relevance is important. We believe our\ncorrelation dimension (CorrDim) provides a complementary metric for evaluating\ngeneralization in LLMs, especially when used alongside perplexity.\n\nAs illustrated in Figure 2, models in the lower-right corner (low perplexity,\nlow CorrDim) tend to generalize better. Furthermore, in Figure 5, we observe a\ndecreasing CorrDim trend during the final phase of pretraining, suggesting that\nCorrDim reflects increasing generaliation over time.\n\nTo further support this interpretation, we conduted a case study on the outlier\ntext *process-theism* (Table 4, Appendix B). There, CorrDim differed\nsignificantly between Qwen2.5-7B (3.56) and Falcon3-10B (8.49). Upon analysis,\nwe found that the higher CorrDim in Falcon3-10B was due to memorization of\ntraining content, whereas Qwen2.5-7B exhibited better generalization. This\nsuggests CorrDim may serve as a useful diagnostic tool for memorization vs.\ngeneralization, which is a key concern in LLMs.\n(For full details, please refer to our response to Reviewer JCEK in the \"Weakness 1\" section)\n\n\n--------------------------------------------\n> Question 1: The reviewer is curious about how would systematically disrupting\nlong-range correlations (beyond simple word shuffling) affect the dimension?\n\nDisrupting long-range dependencies tends to increase the correlation dimension,\nsince CorrDim is sensitive to the global recurrence (or skip structure) in the\ngeneration trajectory.\n- When the sequence is fully shuffled by word, all long-range correlations are\ndestroyed, and the CorrDim typically rises above 10, indicating random-like,\nhigh-dimensional behavior.\n- In cases of partial disruption (e.g., shuffling paragraphs), long-range structure\nis weakend but not erased. Here, CorrDim increases, but remains below 10.\n\nThis trend is demonstrated in our Lin-Tegmark grammar experiments (Figure 3b),\nwhere we gradually increase the dependency parameter $q$ from 0.001 to\n0.5---reducing long-range dependencies---the CorrDim rises from below 1 to above\n10.\n\n\n--------------------------------------------\n> Question 2. Would the correlation dimension framework extend naturally to\nstructured representations (e.g., syntactic tree), and if so, do the authors\nforesee the dimension increasing or decreasing in that case?\n\n\nYes, we believe the framework can be extended to structured representations such\nas syntactic trees, with some caveats. CorrDim is fundamentally driven by\nmultiscale recurrence patterns, which can be viewed as analogous to removing or\nskipping subtrees in a hierarchical representation of text.\n\nIf a syntactic representation is designed to preserve these skip structures,\nwe would expect the resulting CorrDim to be comparable. However, common\nsyntactic formalisms (e.g., Universal Dependencies) may not fully capture\nsuch generative recurrences. In such cases, the measured CorrDim may not\nalign with that of the original sequence.\n\n\n--------------------------------------------\n> Question3. Could the authors relate the correlation dimension to other\nmeasures of linguistic complexity? For example, measures like entropy rate,\nmutual information scaling, or classical fractal exponents (Hurst exponent) have\nbeen used for text.\n\nThank you for this insightful question. We will add a dedicated section in the\nrevised manuscript to clarify how CorrDim relates to these measures. Here is a\nbrief summary.\n\n- Entropy rate (closely related to perplexity): CorrDim tends to converge to\naround 6.5 when entropy rate is low---typically in well-trained LLMs. However,\nwhen entropy is high (e.g., in early training), CorrDim and entropy rate diverge\nand capture different aspects of the model.\n\n- Mutual information (MI) scaling: Higher CorrDim is generally associated with\nfaster decay, as we demonstrated in the Lin-Tegmark experiments (Figure 3).\nHowever, MI and CorrDim capture complementary facets:\n  - Fixed points, limit cycles, and quasi-periodic tori all exhibit zero MI decay,\n  yet their CorrDim values differ: 0, 1, and 2, respectively.\n  - Thus, CorrDim reflects geometric complexity, while MI scaling reflects predictability.\n\n- Hurst exponent: Hurst exponent measures linear autocorrelation in 1D\nsequences. It aligns with CorrDim only when temporal dependencies are linear\n(e.g., fractional Brownian motions). For nonlinear and high-dimensional dynamics\n(like language generation), CorrDim is generally more sensitive and informative,\ncapturing nonlinear dependencies that Hurst exponent may miss."}, {"review_id": "wxnwrIXXqr", "reviewer": "Reviewer_fXFE", "summary": "The authors propose a new metric to evaluate LLMs. Their metric is derived from the self-similarity principle and enables the measurement of recursions in natural text. Their experiments demonstrate that this approach handles different types of text effectively, measuring their complexity while also serving as a tool to evaluate imperfections in LLM text distributions.", "questions": "See weaknesses", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths: This is an interesting paper addressing the highly important topic of benchmarking and understanding the quality and structure of LLMs. Its applicability extends beyond autoregressive models (where perplexity is available). The paper addresses many questions that arise during reading through a comprehensive range of experiments.\n\nWeaknesses: The primary question that remains unanswered for me is how to utilize this metric effectively and how to derive meaningful conclusions from it. While this is a specific concern, I believe it reflects how most practitioners who evaluate LLMs approach such tools. For instance, when evaluating Diffusion-based LLMs, there is no convenient method to assess text quality beyond employing numerous metrics (Zipf Coefficient, Self-BLEU, and other metrics depicted in Table 1) that yield disparate results. Consequently, practitioners must consider all these metrics collectively to gain meaningful insights. I have concerns that this metric operates in a similar manner.\n\nMore concretely, suppose I have an arbitrary model (for example, my hypothetical Turbo-Scientific-13B model), and I wish to determine whether it outperforms various Llama models of different scales (where the definition of \"better\" may be arbitrarily complex). Based on the current presentation, it remains unclear to me whether this comparison is feasible. I observe that different models converge toward a specific value of your metric when perplexity is low; however, in an arbitrary case, I lack knowledge of the appropriate convergence value. If my model's metric is higher or lower than the value to which other models converge, what does this indicate? Is it favorable, unfavorable, or indicative of something else entirely?\n\nI acknowledge that you may respond that this question has already been addressed in the paper. However, I would like to emphasize that if this is the case, the answer is not sufficiently prominent. I would appreciate a practical tutorial for practitioners on how to use and interpret this metric, particularly in the absence of extensive pre-existing evaluations (such as knowledge of the convergence value associated with decreasing perplexity across models).\n\nA second, more minor weakness is that it would be valuable to understand how this metric correlates with downstream performance quality. I believe this is ultimately our primary concern, and it would be interesting to determine whether this metric can provide insights into downstream performance.", "was_revised": true, "final_justification": "I have read the authors' rebuttal, and they have addressed all my concerns. I remain positive regarding the paper and recommend it for acceptance.", "rebuttal": "> This is an interesting paper addressing the highly important topic of\nbenchmarking and understanding the quality and structure of LLMs. Its\napplicability extends beyond autoregressive models (where perplexity is\navailable). The paper addresses many questions that arise during reading through\na comprehensive range of experiments.\n\nThank you for your positive feedback. We appreciate your insightful suggestions\nand will incorporate them into the revised version.\n\n--------------------------\n> When evaluating Diffusion-based LLMs, there is no convenient method to assess\ntext quality beyond employing numerous metrics (Zipf Coefficient, Self-BLEU, and\nother metrics depicted in Table 1) that yield disparate results.\n\nYou are correct. Our method currently applies only to autoregressive LLMs, as it\nrelies on next-token probabilities to identify **skip structures** in texts\n(Figure 1a-b). Extending it to diffusion-based models is possible, but it will\nrequire a method to identify the skip structures from the diffusion process. We\nconsider this an important future direction.\n\n\n--------------------------\n> The primary question that remains unanswered for me is how to utilize this\nmetric effectively and how to derive meaningful conclusions from it.\nI believe it reflects how most practitioners who evaluate LLMs approach such tools.\n\nThank you for raising this important point, which was also echoed by other\nreviewers. We completely agree.\n\nIn brief, we believe correlation dimension is indicative of a model's tendency\nto **generalize** (low CorrDim) versus **memorize** (high CorrDim). Therefore, a\nmodel with lower correlation dimension at similar perplexity should be\nconsidered to generalize better, making CorrDim a potentially useful metric for\nLLM practitioners.\n\nWe have conducted an additional case study to support this claim. As shown in\nTable 4 (Appendix B), the correlation dimensions of Qwen2.5-7B (3.56) and\nFalcon3-10B (8.49) showed an exceptionally large difference on the text\n\"process-theism\" because the two models perceived the long list of scholars'\nnames differently. To demonstrate that this high CorrDim in Falcon3-10B is due\nto memorization of the list during pre-training, we asked the two models to\ncomplete the name list, as follows:\n\n- **Input:** “… André Gounelle, Rem B. Edwards, (start generation)”\n- **Gold answer**: Delwin Brown, **David A. Pailin**, **Franklin I. Gamwell**, Forrest Wood, **David Ray Griffin**, …\n- **Falcon3-10B output**: **David A. Pailin**, **Franklin I. Gamwell**, George L. Kline, André Gounelle, Nancy Murphy, **David Ray Griffin**, …\n- **Qwen2.5-7B output**: Thomas J. Altizer, Richard H. Connell, John Basinger, Daniel S. Brown, Robert N. Bellah, …\n\nAs seen, Falcon3-10B memorizes the names in the text (which did not occur in the\ncontext), while Qwen2.5-7B generalizes to produce generic, plausible names.\n\nThis finding echoes with the results in Figure 5a (Section 4.3), where\ncorrelation dimension decreases in the third training stage as the model learns\nto generalize. In Figure 5b, we also showed how increased correlation dimension\n(near the end of pre-training) coincides with a loss of in-context learning\naccuracy, which requries generalization ability.\n\nPlease note that we do not claim that generalization is always better than\nmemorization. However, in some challenging tasks, such as complex reasoning,\ngeneralization (i.e., lower CorrDim) is preferred.\n\n\n\n--------------------------\n> Practitioners must consider all these metrics (Zipf Coefficient, Self-BLEU, and\nother metrics depicted in Table 1) collectively to gain meaningful insights.\nI have concerns that this metric operates in a similar manner.\n\nWe agree that correlation dimension, like any single metric, has limitations and\ncannot capture all aspects of LLM behavior. However, as a complexity-inspired\nmetric, we believe it offers unique insights---particularly into model\ngeneralization, as discussed above.\n\nMore importantly, correlation dimension is the only metric we are aware of that\ncaptures the dynamical-system nature of autoregressive LLMs. It is invariant\nacross different tokenization schemes or vocabularies---an advantage over other\nmetrics such as the Zipf coefficient, Self-BLEU, and Rep-N.\n\nTo show this, we conducted a control experiment on the script system in Japanese\nlanguage. We tested three parallel Japanese texts, each rendered in two scripts:\n\n- Kanji + kana: the standard modern mix of ideographic kanji and phonographic kana\n- Kana only: a purely syllabic representation using only kana\n\n“Parallel” here means that both versions convey the exact same meaning,\ndiffering only in script. The correlation dimensions are as follows:\n\n| Book | Kanji & Kana (vocabulary size > 2000) | Kana only (vocabulary size < 200) |\n|------|----------|--------------|\n| Shisei | 6.72 | 6.65 |\n| Doko e | 7.47 | 7.72 |\n| Kesshouki | 6.51 | 6.62 |\n\nAs seen, the correlation dimension remained largely unchanged despite the\nvocabulary size was reduced by an order of magnitude, highlighting correlation\ndimension as a robust metric that captures the underlying dynamical system of\nautoregressive LLMs.\n\nIn this case, none of the other metrics (Zipf coefficient, Self-BLEU, Rep-N)\nwould work: the Zipf coefficient requires a large vocabulary to manifest,\nSelf-BLEU and Rep-N are sensitive to tokenization.\n\n\n--------------------------\n> More concretely, suppose I have an arbitrary model (for example, my hypothetical\nTurbo-Scientific-13B model), and I wish to determine whether it outperforms\nvarious Llama models of different scales (where the definition of \"better\" may\nbe arbitrarily complex). Based on the current presentation, it remains unclear\nto me whether this comparison is feasible. I observe that different models\nconverge toward a specific value of your metric when perplexity is low; however,\nin an arbitrary case, I lack knowledge of the appropriate convergence value. If\nmy model's metric is higher or lower than the value to which other models\nconverge, what does this indicate? Is it favorable, unfavorable, or indicative\nof something else entirely?\n\nThank you for your detailed question. As mentioned earlier, we believe a lower\ncorrelation dimension is indicative of a model's tendency to generalize when\ncompared at similar perplexity. In most cases, generalization ability is\npreferred, especially in complex reasoning tasks where the model must generlize\nto unseen contexts.\n\nThat said, if your hypothetical Turbo-Scientific-13B model exhibits a lower\ncorrelation dimension than the Llama models at similar perplexity, it is likely\nto generalize better than the Llama models.\n\nFor example, in Figure 2, the Qwen2.5 family (released in 2024) lies to the\nlower right of the Qwen family (2023), Pareto-dominating its predecessor. This\nis expected, as the Qwen2.5 models are trained on more data and employ more\nadvanced training techniques.\n\n\n\n> I acknowledge that you may respond that this question has already been addressed\nin the paper. However, I would like to emphasize that if this is the case, the\nanswer is not sufficiently prominent. I would appreciate a practical tutorial\nfor practitioners on how to use and interpret this metric, particularly in the\nabsence of extensive pre-existing evaluations (such as knowledge of the\nconvergence value associated with decreasing perplexity across models).\n\nWe will add a dedicated section in the revised manuscript to clarify how\ncorrelation dimension can be used in practice for LLM evaluation. We\nappreciate you pointing out that oversight.\n\n\n--------------------------\n> A second, more minor weakness is that it would be valuable to understand how\nthis metric correlates with downstream performance quality. I believe this is\nultimately our primary concern, and it would be interesting to determine whether\nthis metric can provide insights into downstream performance.\n\nWe agree that the ultimate goal of LLM evaluation is to understand and predict\nhow well a model performs on downstream tasks. However, this is extremely\nchallenging for LLMs, and most existing LLM evaluations are based on\nfine-grained benchmarks.\n\nTo reach a definitive conclusion, we believe it would better to establish a more\ndirect theoretical link and CorrDim and the LLM generation process---an\nimportant direction that lies beyond the scope of this paper. We are interested\nin exploring this in future work and hope that our current findings provide a\nfoundation for new research on LLM generalization from a complex-system\nperspective."}, {"review_id": "SgPxNogyNn", "reviewer": "Reviewer_mbgA", "summary": "While perplexity is widely used as a metric for evaluating LLMs, it has limitations in capturing the long-range structural complexity of natural language, such as repetition when perplexity is low. This paper introduces a novel evaluation metric based on correlation dimension, a measure of self-similarity from fractal geometry, to quantify hierarchical and repetitive structures in language.\nThe authors demonstrate through experiments that this method can consistently detect various forms of text degeneration, including repetition, incoherence, and blandness, which have traditionally been difficult to quantify.", "questions": "Please see the weaknesses listed above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "##### Strengths\n- This paper proposes a new interesting evaluation metric based on fractal geometry (correlation dimension) that captures long-range structural complexity and generation dynamics of LLMs, which conventional metrics fail to measure.\n\n- Authors demonstrate that the proposed metric can unify and quantify multiple types of text degeneration (e.g., repetition, incoherence, and blandness) that are hard to assess using existing methods.\n\n- The paper is well written and easy to follow.\n\n- The approach is theoretically grounded, taken from established mathematical frameworks such as fractal geometry and dynamical systems theory.\n\n##### Weaknesses\n- While perplexity is broadly applicable and robust across different datasets and floating-point precision formats (e.g., FP16), the correlation dimension may yield different optimal values depending on these factors, potentially limiting its general applicability.\n\n- To strengthen the generality and robustness of the findings, it would be helpful to show how the correlation dimension behaves on other text domains (e.g., WikiText-103, BookCorpus), and whether phenomena such as the three-stage evolution are also observed in these settings.\n\n- (minor) The abbreviation \"SEP\" (Stanford Encyclopedia of Philosophy) is defined only in the appendix. For readability, the full name should also be spelled out in the main text.", "was_revised": true, "final_justification": "Based on the rebuttal and the other reviews, particularly in terms of the scope, potential impact, and novelty of the work, I will keep my current scores, leaning slightly toward acceptance. I assume that the authors will address the questions and concerns raised by the reviewers in the final version of the paper.", "rebuttal": "> This paper proposes a new interesting evaluation metric based on fractal\ngeometry (correlation dimension) that captures long-range structural complexity\nand generation dynamics of LLMs, which conventional metrics fail to measure.\nAuthors demonstrate that the proposed metric can unify and quantify multiple\ntypes of text degeneration (e.g., repetition, incoherence, and blandness) that\nare hard to assess using existing methods. The paper is well written and easy to\nfollow. The approach is theoretically grounded, taken from established\nmathematical frameworks such as fractal geometry and dynamical systems theory.\n\nThank you. We are glad you found our work interesting. We appreciate your\nsuggestsions and will incorporate them into the revised manuscript.\n\n\n------------------------------\n> Weakness 1: While perplexity is broadly applicable and robust across different\ndatasets and floating-point precision formats (e.g., FP16), the correlation\ndimension may yield different optimal values depending on these factors,\npotentially limiting its general applicability.\n\nThank you for raising this thoughtful concern. You are absolutely right that correlation dimension reflects both the properties of the text and the model, and therefore may vary accordingly. In fact, this sensitivity is part of its strength---it allows CorrDim to capture meaningful distinctions in structure and generalization behavior across models and datasets.\n\nThat said, we agree it is important to assess the method’s robustness with respect to technical factors such as floating-point precision. We sincerely appreciate your suggestion, which prompted us to run additional experiments using Qwen2.5 models quantized to 4 bits with the activation-aware quantization (AWQ) method. We found that the mean correlation dimension remains remarkably stable, with only minimal changes:\n\n| Model | Mean CorrDim (4-bit AWQ) | Mean CorrDim (FP16) |\n|-------|----------------|---------------|\n| Qwen2.5-1.5B-Instruct-AWQ | 6.06  | 6.00  |\n| Qwen2.5-7B-Instruct-AWQ   | 6.44  | 6.34  |\n| Qwen2.5-14B-Instruct-AWQ  | 6.43  | 6.33  |\n\nThese results suggest that CorrDim is robust even under aggressive quantization, further supporting its potential for broader applicability. We will include these findings in the revised manuscript, and we are grateful to you for highlighting this important point.\n\n\n> Weakness 2: To strengthen the generality and robustness of the findings, it would be\nhelpful to show how the correlation dimension behaves on other text domains\n(e.g., WikiText-103, BookCorpus), and whether phenomena such as the three-stage\nevolution are also observed in these settings.\n\nWe agree. We evaluated ten articles from WikiText-103 and ten novels from\nBookCorpus and both datasets showed CorrDim similar to that reported in the paper:\n\n| Dataset | Mean CorrDim (std) |\n|---------|--------------------|\n| WikiText-103 |  6.41 (0.99) |\n| BookCorpus (novels) | 7.03 (1.08) |\n\nIn addition, both datasets exhibit the same three-stage training evolution (as in Figure 5a).\nWe will include a new figure for these results.\n\nIn addition, following Reviewer JCEK’s recommendation, we extended our evaluation to:\n- Seven other natural languages: French, German, Spanish, Italian, Dutch, Chinese, and Japanese\n- Three programming languages: Python, Java, and C\n- Models with and without instruct/RL fine-tuning\n\nThe findings are consistent:\n- For natural languages, correlation dimensions consistently range between 6 and 7;\n- For programming languages, they are slightly lower—around 5, reflecting their more rigid structure;\n- Instruction-tuned and RL-fine-tuned models show only modest increases in CorrDim.\n\nWe provide a detailed breakdown of these results in our response to Reviewer JCEK (in Limitation 1). We believe these expanded experiments further support the robustness and generality of the correlation dimension as a structural diagnostic across languages, domains, and training paradigms.\n\n------------------------------\n> Weakness 3: (minor) The abbreviation \"SEP\" (Stanford Encyclopedia of\nPhilosophy) is defined only in the appendix. For readability, the full name\nshould also be spelled out in the main text.\n\nThank you for catching that. We will spell out Stanford Encyclopedia of\nPhilosophy (SEP) on first use in the main text."}, {"review_id": "XyYU3vcTia", "reviewer": "Reviewer_JCEK", "summary": "The authors introduce Correlation Dimension (CD) as a global-structure metric for autoregressive LLMs. Instead of measuring token-level loss, CD treats the log-probability vector sequence \n𝑥𝑡 produced during generation as a trajectory in high-dimensional space and estimates its correlation integral \n𝑆\n(\n𝜀\n)\n∝\n$𝜀^𝑑$\n . The slope \n𝑑 serves as a proxy for the effective dynamical degrees of freedom: low \n𝑑 signals trajectory collapse (e.g., repetition), high \n𝑑 indicates random noise, and natural language tends to stabilise around \n𝑑\n ⁣\n≈\n ⁣\n6\n ⁣\n−\n ⁣\n7. Experiments on Pythia checkpoints, six open LLMs, and GPT-3.5/4-mini demonstrate:\n\n1. a three-phase training curve (short-range learning → long-range capture → compression/overfit);\n\n2. sensitivity to synthetic grammar complexity (Lin-Tegmark);\n\n3. unified detection of repetition, disorder, and semantic sparsity;\n\n4. strong correlation (ρ ≈ 0.95) with HelloEval robustness scores during long-text “name-stream” stress tests.", "questions": "1. How do you access the full logits of the API closed model for degradation detection & long text stress experiments? My understanding is that this indicator requires access to the full logits, and the API closed model is difficult to apply?\n2. The author shows through the RLHF model that the indicator can compare the same model version across the entire process of \"pre-training-instruction fine-tuning-RLHF\", but this is not entirely intuitive. Current reasoning models have complex reward rules injected into the reinforcement learning stage and have different generation strategies. I want to know what the actual impact of RL strategy and the so-called \"text structure\" on the \"complexity\" of the evaluation of this indicator is? Can you explain why the impact of RL strategy is limited in the experiment?\n3. If you randomly permute \n𝑥𝑡​ order (destroying long-range structure) while keeping unigram distribution, how does CD react? This would support the “global hierarchy” claim.", "limitations": "Multi-language and code generation have not been tested yet → The transferability of indicators is unknown.\n\nThe threshold range needs to be set manually, and there is currently no adaptive algorithm.\n\nAuthors should add discussion and mitigation on potential bias against low-resource languages with inherently repetitive morphology (e.g., language-conditioned normalisation).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The explanation is intuitive. Small distance → The interval [s, t) can be regarded as a \"skippable segment\", corresponding to text jumps of different scales (skip); the threshold size forms a hierarchy from word level to sentence level, which can cover the global complexity measurement characteristics to a certain extent.\nHigh dimension → The state trajectory occupies a higher dimensional manifold (complex/random); low dimension → The trajectory collapses to a low dimensional attractor (repetitive or pattern-poor). It can monitor phased learning and potential pattern collapse without increasing training overhead, providing signals for early stopping and model selection.\n2. The application of long text generation health has certain practical value. Online sliding window calculation d in the reasoning flow: If the continuous window d ↓ < 3 ⇒ trigger rewriting/temperature heating, and warn of \"deviant\" paragraphs. d also decreases for repetition, disorder, and information sparseness → simpler than multiple dedicated probes.\n\nWeaknesses:\n1. When there are lists or rhymed texts, the dimensionality difference between models is large and the explanation is not sufficient.\n2. Core estimator (Grassberger–Procaccia) is classical; novelty lies in application, not algorithm. Prior LLM work has used spectrum-based complexity metrics. A side-by-side ablation (same texts) would clarify relative advantages.", "was_revised": true, "final_justification": "This work is well-constructed and interesting. Further exploration of its relationship to the model's generalization and extrapolation capabilities would be even more valuable. Given that its application value has yet to be fully verified, my current rating is objective.", "rebuttal": "We greatly appreciate your insightful comments and valuable feedback. We will revise our paper to address all concerns. Below, we respond point-by-point (quoted comments are lightly rephrased due to length limitation).\n\n---\n> Strength 1. The explanation is intuitive. Small distance → The interval [s, t) can be regarded as a \"skippable segment\", corresponding to text jumps of\ndifferent scales (skip) ... It can monitor phased learning and potential pattern\ncollapse without increasing training overhead, providing signals for early\nstopping and model selection.\n\nYour understanding is exactly right. We’re delighted that our explanation came across intuitively.\n\n---\n> Strength 2. The application of long text generation health has certain practical value.\nOnline sliding window calculation d in the reasoning flow: If the continuous\nwindow d ↓ < 3 ⇒ trigger rewriting/temperature heating, and warn of \"deviant\"\nparagraphs. d also decreases for repetition, disorder, and information\nsparseness → simpler than multiple dedicated probes.\n\nWe agree that a sliding‐window approach is practical for monitoring correlation dimension in real time. Thank you for this suggestion—we will include it as an explicit use case in the revised manuscript.\n\n---\n> Weakness 1. When there are lists or rhymed texts, the dimensionality difference between\nmodels is large and the explanation is not sufficient.\n\nThank you for highlighting this important issue. We will add a dedicated section discussing it. In brief, we believe the observed difference is a feature rather than a bug. Specifically, the difference reflects a contrast between generalization (low CorrDim) vs. memorization (high CorrDim) behaviors.\n\nIn Appendix B, we reported that on a list of scholars’ names, Falcon3-10B yields CorrDim = 8.49, whereas Qwen2.5-7B gives 3.56.\nTo investigate the origin of the large difference, we asked the models to complete the name list, as follows:\n- **Input:** “… André Gounelle, Rem B. Edwards, (start generation)”\n- **Gold answer**: Delwin Brown, **David A. Pailin**, **Franklin I. Gamwell**, Forrest Wood, **David Ray Griffin**, …\n- **Falcon3-10B output**: **David A. Pailin**, **Franklin I. Gamwell**, George L. Kline, André Gounelle, Nancy Murphy, **David Ray Griffin**, …\n- **Qwen2.5-7B output**: Thomas J. Altizer, Richard H. Connell, John Basinger, Daniel S. Brown, Robert N. Bellah, …\n\nFalcon3-10B has clearly memorized the list during training, while Qwen2.5-7B treats it as random names and “simplifies” the pattern, hence the lower CorrDim. This case illustrates that correlation dimension not only reflects the text’s structural complexity but also exposes the difference between a model’s memorization and its generalization behaviors.\n\n---\n> Weakness 2. Core estimator (Grassberger–Procaccia) is classical; novelty lies\nin application, not algorithm. Prior LLM work has used spectrum-based complexity\nmetrics. A side-by-side ablation (same texts) would clarify relative advantages.\n\nThank you for raising this point and suggesting missed references. We will add discussion of spectrum‐based metrics and perform ablations on identical texts to compare their behavior with CorrDim.\n\n---\n> Question 1. How do you access the full logits of the API closed model for degradation\ndetection & long text stress experiments?\n\nThere are two ways to apply our method to evaluate a model: (1) running the\ntarget model itself, or (2) using another model to analyze the text generated by\nthe target model. These correspond to Sections 4 and 5 of the paper,\nrespectively.\n\nFor closed models, approach (1) is unavailable since we don't have access to the\nfull logits, but approach (2) remains feasible. For example, in Section 5.1 we\nused the open model Falcon3-10B to evaluate text generated by the closed GPT-4o\nmodel.\n\n---\n> Question 2. I want to know what the actual impact of RL strategy and the\nso-called \"text structure\" on the \"complexity\" of the evaluation of this\nindicator is? Can you explain why the impact of RL strategy is limited in the\nexperiment?\n\nWe agree that instruction fine-tuning can affect model behavior.\nWe ran a preliminary experiment on the SEP dataset (60 texts) comparing base vs. instruct versions of Qwen2.5 and Falcon3 models:\n\n| Model       | Base CorrDim | Instruct CorrDim | Increase |\n|-------------|----------------|---------------|---------|\n| Falcon3-10B |  6.56          | 6.93          | 0.37 |\n| Falcon3-7B  |  6.56          | 6.81          | 0.25 |\n| Falcon3-3B  |  6.11          | 6.44          | 0.33 |\n| Falcon3-1B  |  6.04          | 6.33          | 0.29 |\n| Qwen2.5-32B |  6.32          | 6.45          | 0.13 |\n| Qwen2.5-14B |  6.26          | 6.33          | 0.07 |\n| Qwen2.5-1.5B |  5.97         | 6.00          | 0.03 |\n\nInstruction fine-tuning consistently raises CorrDim, especially for larger models. We suspect this reflects stylistic shifts from plain text to instruction-driven inputs, but the precise mechanism remains to be explored. We will include these results and discuss possible causes.\n\n---\n> Question 3.  If you randomly permute $x_t$ order (destroying long-range structure) while\nkeeping unigram distribution, how does CD react? This would support the “global\nhierarchy” claim.\n\nAfter shuffling token order, CorrDim jumps above 10, indicating exactly destroying of long-range structure. We will add this experiment. Thank you for your suggestion.\n\n---\n> Limitation 1. Multi-language and code generation have not been tested yet → The transferability of indicators is unknown.\n\nWe performed quick experiments on other languages and code:\n1. Natural languages: 80 Project-Gutenberg books (10 books each in English, French, German, Spanish, Italian, Dutch, Chinese, Japanese), processed by Qwen2.5-32B:\n\n| Language | CorrDim (mean)   | CorrDim (median) |\n|---|----------|--------|\n| English  | 6.67    | 6.69  |\n| French   | 6.72    | 6.72     |\n| German   | 7.34       | 7.29           |\n| Spanish  | 6.10    | 6.20           |\n| Italian  | 6.67    | 6.69           |\n| Dutch    | 6.77    | 6.77           |\n| Chinese  | 7.61             | 6.90           |\n| Japanese | 6.98             | 7.05           |\n\nMost languages exhibit a CorrDim of around 6–7. Small variations likely reflect differences in genre and writing systems: our Spanish and English samples are primarily novels; the German corpus consists mostly of scholarly works (e.g., Hegel’s philosophical texts); and the Chinese and Japanese books employ traditional character sets that differ from the modern scripts on which the model was trained.\n\n2. Code: 30 files each from Python (CPython stdlib), Java (OpenJDK java.base), and C (Linux kernel):\n\n| Language | Mean CorrDim (std) |\n|----------|-------------|\n| Python   | 5.05 (0.73) |\n| Java     | 4.93 (0.89) |\n| C        | 4.94 (0.55) |\n\nCode consistently shows lower CorrDim than natural language, reflecting its higher structural regularity.\nWe will integrate these findings and discuss how degeneration phenomena differ between code, natural-language, and complex reasoning tasks.\n\n---\n> The threshold range needs to be set manually, and there is currently no adaptive algorithm.\n\nWe used the simple formula from Appendix A—dependent only on token sequence length and invariant to embedding‐distance scaling—for all experiments. We will clarify this procedure in the main text.\n\n---\n> Limitation 3. Authors should add discussion and mitigation on potential bias against low-resource languages with inherently repetitive morphology (e.g., language-conditioned normalisation).\n\nThank you for raising this important point. We agree that careful consideration is needed when evaluating languages with limited resources or highly repetitive morphological structures.\n\nWe would like to clarify that correlation dimension (CorrDim) is designed to capture the dynamical properties of the text generation process, rather than surface-level features such as vocabulary size or overt repetition. As such, it is less sensitive to superficial morphological differences and more focused on long-range dependencies embedded in the generation trajectory.\n\nTo empirically examine this, we conducted an experiment using three Japanese literary texts, each rendered in two different scripts:\n- Kanji + kana (vocabulary size > 2,000): the standard orthographic system combining ideographic and phonographic characters;\n- Kana only (vocabulary size < 200): a fully phonographic version using syllabary alone.\n\nThese versions are parallel in content—that is, they express the exact same meaning and structure, differing only in their script and vocabulary richness.\n\n| Book | Kanji & Kana (vocab > 2000) | Kana only (vocab < 200) |\n|------|----------|--------------|\n| Shisei | 6.72 | 6.65 |\n| Doko e | 7.47 | 7.72 |\n| Kesshouki | 6.51 | 6.62 |\n\nAs shown, the correlation dimensions remain stable across these drastically different surface representations. This suggests that CorrDim is indeed robust to changes in vocabulary size and script, and is more reflective of deeper structural properties.\n\nAdditionally, as demonstrated in our Lin–Tegmark experiment (Section 4.1), CorrDim varies significantly (from ≈1 to >20) depending on the strength of long-range dependencies, even when applied to simple binary sequences (e.g., 0/1 grammars). This further supports our claim that CorrDim responds primarily to temporal structure, rather than surface repetition.\n\nEven when CorrDim is used to detect degeneration phenomena like repetition, it does so by analyzing the internal trajectory of the LLM’s probability distributions—not by simply counting repeated tokens on the surface, like Rep-N or Self-BLEU scores in previous works.\n\nWe will include this analysis and discussion in the revised manuscript, along with a broader reflection on potential biases and promising directions for future investigation.\n\n---\nWe trust these additions and clarifications will address your concerns. Thank you again for your plentiful constructive feedback."}]

Author rebuttal:

Meta-review: This work establishes a measure of text complexity conditioned on a language model based on the self-similarity of the fractal structure of its next-word probability distributions. It establishes a common dimension shared by a wide range of human languages as well as more and less predictable forms of text, and introduces a range of other observations.

Reviewers generally commended this work for its novel and timely insights, grounded approach, and good writing. Reviewers mostly found the metric (itself not a contribution of this work) to be intuitive and informative in this context, and noted that it may have some practical applications, though the extent of those may be modest.

Given this, we recommend accepting the paper. Please update the work as discussed with the reviewers.
Decision: Accept (poster)