Title: Fractal Patterns May Illuminate the Success of Next-Token Prediction
Venue: NeurIPS 2024
Accepted: yes
Keywords: fractal patterns

large language models

Hurst exponent

self-similarity

Joseph exponent

perplexity

information theory
Abstract: We study the fractal structure of language, aiming to provide a precise formalism for quantifying properties that may have been previously suspected but not  formally shown. We establish that language is: (1) self-similar, exhibiting complexities at all levels of granularity, with no particular characteristic context length, and (2) long-range dependent (LRD), with a Hurst parameter of approximately 0.7.
Based on these findings, we argue that short-term patterns/dependencies in language, such as in paragraphs, mirror the patterns/dependencies over larger scopes, like entire documents. This may shed some light on how  next-token prediction can capture the structure of text across multiple levels of granularity, from words and clauses to broader contexts and intents. In addition, we carry out an extensive analysis across different domains and architectures, showing that fractal parameters are robust.
Finally, we demonstrate that the tiny variations  in fractal parameters seen across LLMs improve upon perplexity-based bits-per-byte (BPB) in predicting their downstream performance. We hope these findings offer a fresh perspective on language and the mechanisms underlying the success of LLMs.

Fractal Patterns May Illuminate the Success of Next-Token Prediction

Abstract

We study the fractal structure of language, aiming to provide a precise formalism for quantifying properties that may have been previously suspected but not formally shown. We establish that language is: (1) self-similar , exhibiting complexities at all levels of granularity, with no particular characteristic context length, and (2) long-range dependent (LRD), with a Hurst parameter of approximately H = 0 . 70 ± 0 . 09 . Based on these findings, we argue that short-term patterns/dependencies in language, such as in paragraphs, mirror the patterns/dependencies over larger scopes, like entire documents. This may shed some light on how next-token prediction can capture the structure of text across multiple levels of granularity, from words and clauses to broader contexts and intents. In addition, we carry out an extensive analysis across different domains and architectures, showing that fractal parameters are robust. Finally, we demonstrate that the tiny variations in fractal parameters seen across LLMs improve upon perplexity-based bits-per-byte (BPB) in predicting their downstream performance. We hope these findings offer a fresh perspective on language and the mechanisms underlying the success of LLMs.

1 Introduction

How does the training objective of predicting the next token in large language models (LLMs) yield remarkable capabilities? Consider, for instance, the two models: Gemini [5] and GPT4 [50]. These models have demonstrated capabilities that extend to quantitative reasoning, summarization, and even coding, which has led some researchers to ponder if there was more to intelligence than 'on-the-fly improvisation' [11]. While providing a satisfactory explanation is a difficult endeavor, a possible insight can be drawn from fractals and self-similarity. We elucidate the connection in this work.

Self-Similarity. Self-similar processes were introduced by Kolmogorov in 1940 [36]. The notion garnered considerable attention during the late 1960s, thanks to the extensive works of Mandelbrot and his peers [19]. Broadly speaking, an object is called 'self-similar' if it is invariant across scales, meaning its statistical or geometric properties stay consistent irrespective of the magnification applied to it (see Figure 1). Nature and geometry furnish us with many such patterns, such as coastlines, snowflakes, the Cantor set and the Kuch curve. Despite the distinction, self-similarity is often discussed in the context of 'fractals,' another term popularized by Mandelbrot in his seminal book The Fractal Geometry of Nature [45]. However, the two concepts are different [26]. See Section 2.

In language, in particular, there have been studies arguing for the presence of a self-similar structure. Nevertheless, due to computational constraints, it was not feasible to holistically model the joint probability distribution of language. As such, linguists often resorted to rudimentary approximations in their arguments, such as by substituting a word with its frequency or length [9], or by focusing on



























⋂]⌉˜



























⋂]⌉˜





the recurrence of a specific, predetermined word [49, 3]. These studies fall short of fully capturing the structure of language due to the simplifying assumptions they make, as discussed in Section 4.

Highlighting the self-similar nature of a process can have profound implications. For instance, conventional Poisson models for Ethernet traffic were shown to fail because traffic was self-similar [16, 39, 51, 69]. In such cases, recognizing and quantifying self-similarity had practical applications, such as in the design of buffers [40]. Similarly in language, we argue that self-similarity may offer a fresh perspective on the mechanisms underlying the success of LLMs. Consider the illustrative example shown in Figure 1, where the task is to predict the subsequent measurement in a time series, specifically predicting next tokens in a Wikipedia article (see Section 2 for details). The three plots in Figure 1 (left) represent different manifestations of the same process observed across three distinct time scales. Notably, we observe rich, self-similar details, such as burstiness, in all of them. A well-established approach for quantifying self-similarity is the Hölder exponent [66], which we denote by S . In language, we find it to be S = 0 . 59 ± 0 . 08 , confirming statistical self-similarity.

Why is this important? We hypothesize that since LLMs are trained to predict the future of a selfsimilar process, they develop proficiency in capturing patterns across multiple levels of granularity for two interconnected reasons. First, self-similarity implies that the patterns at the level of a paragraph are reflective of the patterns seen at the level of a whole text, which is reminiscent of the recursive structure of language [53]. Thus, recognizing short-term patterns can aide in learning broader contexts. Second, because language displays intricate patterns at all levels of granularity, it would not be enough to rely only on the immediate context of a sentence to predict the next token. Instead, the model needs to identify patterns at higher levels of granularity; e.g. follow the direction of the argument and the broader intent. It must balance between short- and long-term contexts. Willinger et al. [68] and Altmann et al. [3] argue for self-similarity in language due to this hierarchical nature.

Long-range dependence. However, self-similarity alone is not sufficient for a predictive model to exhibit anything resembling 'intelligent' behavior. In fact, some self-similar processes, despite their intricate details, remain entirely unpredictable. A quintessential example is the simple Brownian motion, which is a Wiener process with independent increments. Its discrete analog is B n = ∑ n i =1 ε i , where ε i ∼ N (0 , σ 2 ) . Despite possessing rich details at all granularities, a model trained to predict B n cannot learn anything useful from data since the process itself has independent increments.

Thus, for strong capabilities to emerge, the process must have some degree of predictability or dependence as well. One classical metric for quantifying predictability in a stochastic process is the Hurst parameter [31], developed by the hydrologist H. E. Hurst in 1951 while studying the Nile river. It is generally considered to be a robust metric [68], unlike the wavelet estimator [1] and the periodogram method [24] that can be sensitive to errors [54]. As discussed in Section 2.3, we find the Hurst parameter in language to be H = 0 . 70 ± 0 . 09 . For context, H only takes values in [0 , 1] . A value H > 0 . 5 implies predictability in the data, while H = 0 . 5 indicates random increments.

While it is compelling that our estimate of H in language lies nearly midway between predictability ( H = 1 ) and noise ( H = 0 . 5 ), a Hurst parameter of about 0 . 75 turns out to occur commonly in nature, including in river discharges, Ethernet traffic, temperatures, precipitation, and tree rings [16, 21, 8]. For agents that learn from data, such as LLMs, this value is also reminiscent of processing-based

theories of curiosity, which suggest that a sweet spot of complexity exists (not too simple, nor too unpredictable) that facilities or accelerates learning [34].

Importantly, predictability and self-similarity together imply long-range dependence (LRD). This follows from the definition of self-similarity, where the patterns at small scales mirror those at larger scales so, for example, the correlations established at micro levels are also pertinent at macro levels. LRD is arguably crucial for enhancing the functionality of predictive models because processes with only short-range dependence could be forecasted (somewhat trivially) with lookup tables that provide the likelihood of transitions over brief sequences. By contrast, this is not possible in LRD processes whose contexts extend indefinitely into the past.

Information Theoretic Complexity. To define fractal parameters, we follow recent works such as [28, 22, 41, 47, 25] in adopting an information-theoretic characterization of the complexity in language using minimal-length codes or surprise. This corresponds to an intrinsic, irreducible description of language and the minimum compute overhead to comprehend/decode it [22], which also correlates well with actual reading times [28, 41]. In this context, self-similarity means that the intrinsic complexity or surprise in language (measured in bits) cannot be smoothed out, even as we look into broader narratives. That is, surprising paragraphs will follow predictable paragraphs, in a manner that is statistically similar to how surprising sentences follow predictable sentences.

Analysis. How robust are these findings? To answer this question, we carry out an extensive empirical analysis across various model architectures and scales, ranging from 1B to over 500B parameters. We find that fractal parameters are quite robust to the choice of the architecture.

However, there exists tiny variations across LLMs. Interestingly, we demonstrate that from a practical standpoint, these differences help in predicting downstream performance in LLMs compared to using perplexity-based metrics alone, such as bits-per-byte (BPB). Specifically, we introduce a new metric and show that using it to predict downstream performance can increase the adjusted R 2 from approximately 0 . 65 when using solely BPB, to over 0 . 86 with the new metric 2 .

Statement of Contribution. In summary, we:

highlight how the fractal structure of language can offer a new perspective on the capabilities of LLMs, and provide a formalism to quantify properties, such as long-range dependence.

establish that language is self-similar and long-range dependent. We provide concrete estimates in language of the three parameters: the self-similarity (Hölder) exponent, the Hurst parameter, and the fractal dimension. We also estimate the related Joseph exponent.

carry out a comparative study across different model architectures and scales, and different domains, such as ArXiv and GitHub, demonstrating that fractal parameters are robust.

connect fractal patterns with learning. Notably, we show that a 'median' Hurst exponent improves upon perplexity-based bits-per-byte (BPB) in predicting downstream performance.

2 Fractal Structure of Language

2.1 Preliminaries

Suppose we have a discrete-time, stationary stochastic process ( x t ) t ∈ N , with E [ x t ] = 0 and E [ x 2 t ] = 1 . We will refer to ( x t ) t ∈ N as the increment process to distinguish it from the integral process ( X t ) t ∈ N defined by X t = ∑ t k =0 x k . While ( x t ) t ∈ N and ( X t ) t ∈ N are merely different representations of the same data, it is useful to keep both representations in mind. For example, self-similarity is typically studied in the context of integral processes whereas LRD is defined on increment processes.

In the literature, it is not uncommon to mistakenly equate parameters that are generally different. For example, the Hurst parameter H has had many definitions in the past that were not equivalent, and Mandelbrot himself cautioned against this [44]. The reason behind this is because different parameters can agree in the idealized fractional Brownian motion, leading some researchers to equate them in general [66]. We will keep the self-similarity exponent S and H separate in our discussion.

OpenWeb

Github

FreeLaw

PileCC

10-3

10-3

10-3

10-3

Ω 10-4

Ω 10-4

Ω10-4

Ω10-4

10-5

10-5

10-5

10-5

101

103

101

103

101

103

101

103

T

T

2

T

10-3

Wiki

10-3

PubMed

10-3

DMMath

10-3

ArXiv

Ω 10-4

Ω10-4

Q10-4

Q10-4

10-5

10-5

10-5

10-5

101

103

101

103

101

103

101

103

T

T

T

1

103

OpenWeb

103

Github

103

FreeLaw

103

PileCC

102

102

102

102

S

S

S

S

/

R

101

R

101

R

101

R

101

10°

10°

10°

10°

100

103

100

103

100

103

100

103

n

n

n

n

103

Wiki

103

PubMed

103

DMMath

103

ArXiv

102

102

102

102

S

S

S

S

一

R

101

R

101

R

101

R

101

10°

10°

100

10°

100

103

100

103

100

103

100

103

n

n

n

n

Experimental Setup. In order to establish self-similarity and LRD in language, we convert texts into sequences of bits using a large language model (LLM). Specifically, we use PaLM2L (Unicorn) [6] to calculate the probability of the next token w t conditioned on its entire prefix w [ t -1] = ( w 0 , w 1 , . . . , w t -1 ) . As discussed in Section 1, this captures its intrinsic, irreducible description [22]. By the chain rule [15], the corresponding number of bits assigned to w t is z t = -log p ( w t | w [ t -1] ) . Unlike in prior works, which rely on simplifications such as by substituting a word with its length [9] or by focusing on the recurrence of a single word [49, 3], we use the LLM to approximate the full joint distribution of language since LLMs are known to produce calibrated probability scores at the token level [33]. We carry out these calculations for prefixes of up to 2048 tokens ( ≈ 8 pages of text). With a suitable normalization, such as bits-per-byte (BPB), one obtains a standardized description of text, consistent across tokenizers. BPB is widely used as a tokenizer-agnostic metric to compare LM modeling performance, e.g. for The Pile [23].

Besides PaLM2, we also experiment and report on various model sizes of PaLM [12] and decoderonly T5 [55]. Namely, we report results for models: PaLM2 XXS (Gecko), XS (Otter), S (Bison), M, and L (Unicorn); PaLM 8B, 62B, 540B; and decoder-only T5.1.1 at Base (110M), Large (341M), XL (1.2B), and XXL (5B) sizes. For PaLM and PaLM2, we use the checkpoints pretrained in Chowdhery et al. [12] and Anil et al. [6]. All T5.1.1 decoder baselines, on the other hand, are trained with a casual language modeling objective for 262B tokens of C4 [55]. All experiments are executed on Tensor Processing Units (TPUs). More details on how we train T5.1.1 baselines are in Appendix A.

Once z t is computed for a document, we follow standard definitions in constructing the increment process ( x t ) t ∈ N by normalizing z t to have a zero-mean and unit variance. Intuitively, fractal parameters are intended to measure a fundamental property of the process (e.g. LRD) that should not be affected by scale, hence the normalization. The integral process ( X t ) t ∈ N is calculated based on ( x t ) t ∈ N , as described earlier and depicted in Figure 1 (top). Normalizing bits (to have zero mean and unit variance) models language as a random walk. It is a standard approach used extensively in the literature in various contexts, such as in DNA sequences [52, 57, 48, 35, 59].

10

3

10

2

0.0

0.5

1.0

1.5

S

OpenWebText2

Github

FreeLaw

Pile-CC

Wikipedia (en)

PubMed Central

DM Mathematics

ArXiv







 

 

⊔]√[√̂

⊔√˜˜⊕∐glyph[arrowvertexdbl]

∑]⌈˜∖

∨]⌋]√˜̂]∐({˜{}

√∑]√

⊕∐˜

For analysis, we use The Pile validation split [23], consisting of 22 subdomains such as Wikipedia and GitHub. We restrict analysis to sufficiently-long documents of length > 4 K tokens and use the first 2K tokens only, to sidestep potential effects of the finite length of documents and the model context. To mitigate noise, only domains with > 1 K documents are compared; we report results for them separately and their median. We use bootstrapping [17] to estimate the error margin.

Notation. We write f ( x ) ∼ x c if f ( x ) = x c L ( x ) for some function L that satisfies L ( tx ) /L ( x ) → 1 as x → ∞ for all t > 0 . Examples of slowly varying functions are constants L ( x ) = c and L ( x ) = log x . When f ( x ) ∼ x c , we abuse terminology slightly by referring to f ( x ) as a power law.

2.2 Self-similarity exponent - Scale invariance

An integral process is said to be self-similar if it exhibits statistical self-similarity. More precisely, ( X t ) t ∈ N is self-similar if ( X τt ) t ∈ N is distributionally equivalent to ( τ S X t ) t ∈ N for some exponent S . Thus, scaling of time is equivalent to an appropriate scaling of space. We will refer to τ as the granularity level and to the exponent S as the self-similarity or Hölder exponent [66]. Many time series in nature exhibit self-similar structures, such as human blood pressure and heart rate [27].

One approach for calculating S is as follows. Fix glyph[epsilon1] glyph[lessmuch] 1 and denote the τ -increments by ( X t + τ -X t ) t ∈ N . These would correspond, for instance, to the number of bits used for clauses, sentences, paragraphs and longer texts as τ increases. In terms of the increment process ( x t ) t ∈ N , this corresponds to aggregating increments into 'bursts'. Let p glyph[epsilon1] ( τ ) be the probability mass of the event {| X t + τ -X t | ≤ glyph[epsilon1] } t ∈ N . Then, S can be estimated by fitting a power law relation p glyph[epsilon1] ( τ ) ∼ τ -S [66]. Generally, S is robust to the choice of glyph[epsilon1] ∈ [10 -3 , 10 -2 ] as shown in Figure 4 (left) so we fix it to glyph[epsilon1] = 5 × 10 -3 .

Figure 2 plots the probability p glyph[epsilon1] ( τ ) against τ using PaLM2-L. We indeed observe a power law relation over at least two orders of magnitude; i.e. linear in a log-log scale, with a median self-similarity exponent of S = 0 . 59 ± 0 . 08 . Section 3 shows that the median S is robust to the choice of the LLM.

2.3 Hurst parameter - Long-range dependence

The Hurst parameter H ∈ [0 , 1] quantifies the degree of predictability or dependence over time [31]. It is calculated using the so-called rescaled-range (R/S) analysis. Let ( x t ) t ∈ N be an increment process. For each n ∈ N , write y t = x t -1 t ∑ t k =0 x k and Y t = ∑ t k =0 y t . The range and scale are defined, respectively, as R ( n ) = max t ≤ n Y t -min t ≤ n Y t and S ( n ) = σ ( { x k } k ≤ n ) , where σ is the standard deviation. Then, the Hurst parameter H is estimated by fitting a power law relation R ( n ) /S ( n ) ∼ n H . As stated earlier, for completely random processes, such as a simple Brownian motion, it can be shown that H = 1 / 2 . In addition, H > 1 / 2 implies dependence over time [16, 68, 8].

Writing ρ n = E [( x t + n x t ] for the autocovariance function of the increment process ( x t ) t ∈ N , the Hurst parameter satisfies H = 1 -β/ 2 when ρ n ∼ n -β as n →∞ [26, 16]. Since in self-similar processes, H > 1 / 2 implies long-range dependence (LRD), LRD is equivalent to the condition that the autocovariances are not summable. In terms of the integral process, it can be shown that [58]: lim n →∞ Var( X n ) n = 1 + 2 ∑ ∞ i =1 ρ i . Hence, if H < 1 / 2 , the auto-covariances are summable and

∑√̂⊕˜̂(˜{√√∐⌈

〈⊕(⊕∐√[˜⌉∐√]̂√

Var( X n ) grows, at most, linearly fast on n . On the other hand, if the process has LRD, Var( X n ) grows superlinearly on n . In particular, using the Euler-Maclaurin summation formula [7, 2], one obtains Var( X n ) ∼ n 2 H if H > 1 / 2 . Figure 3 plots the rescaled range R ( n ) /S ( n ) against n . We observe a power law relation with a median Hurst parameter of H = 0 . 70 ± 0 . 09 .

2.4 Fractal dimension - Complexity at all levels

Broadly speaking, the fractal dimension of an object describes its local complexity. For a geometric object Z , such as the Koch curve, let τ be a chosen scale (e.g. a short ruler for measuring lengths or a small square for areas). Let N ( τ ) be the minimum number of objects of scale τ that cover Z ; i.e. contain it entirely. Then, the fractal dimension of Z , also called its Hausdorff dimension, is: D = -lim τ → 0 { log N ( τ ) log τ } [54]. For example, a line has a fractal dimension 1 , in agreement with its topological dimension, because N ( τ ) = C/τ for some constant C > 0 .

By convention, an object is referred to as 'fractal' if D is different from its topological dimension. For example, the fractal dimension of the Koch curve is about 1.26 when its topological dimension is 1. Fractals explain some puzzling observations, such as why estimates of the length of the coast of Britain varied significantly from one study to another, because lengths in fractals are scale-sensitive. Mandelbrot estimated the fractal dimension of the coast of Britain to be 1.25 [43].

The definition above for the fractal dimension D applies to geometric shapes, but an analogous definition has been introduced for stochastic processes. Let ( x t ) t ∈ R be a stationary process with autocovariance ρ n . Then, its fractal dimension D is determined according to the local behavior of ρ n at the vicinity of n = 0 , by first normalizing ( x t ) t ∈ R to have a zero-mean and a unit variance, and modeling ρ n using a power law ρ n ∼ 1 -n α as n → 0 + , for α ∈ (0 , 2] . Then, the fractal dimension D ∈ [1 , 2] of ( x t ) t ∈ R is defined by D = 2 -α/ 2 [26]. It can be shown that D = 2 -S [26]. For language, this gives a median fractal dimension of D = 1 . 41 ± 0 . 08 .

2.5 Joseph effect - Burstiness

Finally, we examine another related parameter that is commonly studied in self-similar processes. The motivation behind it comes from the fact that in processes with LRD, one often observes burstiness as shown in Figure 1; i.e. clusters over time in which the process fully resides on one side of the mean, before switching to the other. This is quite unlike random noise, for instance, where measurements are evenly distributed on both sides of the mean. The effect is often referred to as the Joseph effect, named after the biblical story of the seven fat years and seven lean years [68, 46, 66].

A common way to quantify the Joseph effect for integral processes ( X t ) t ∈ N is as follows [66]. First, let σ τ be the standard deviation of the τ -increments X t + τ -X t . Then, fit a power law relation σ τ ∼ τ J . The exponent J here is called the Joseph exponent. In an idealized fractional Brownian motion, both J and the self-similarity exponent S coincide. Figure 5 provides the detailed empirical results. Overall, we find that J = 0 . 49 ± 0 . 08 .

3 Analysis

Comparative Analysis. Table 1 compares fractal parameters across different domains, such as ArXiv, Github and Wikipedia. In general, most domains share similar self-similarity and Hurst exponents with a few exceptions. The first notable exception is DM-Mathematics, which has a Hurst parameter of about 0.5, indicating a lack of LRD. Upon closer inspection, however, a value of H = 0 . 5

OpenWeb

Github

FreeLaw

PileCC

102

102

102

102

b 101

b101

b 101

b101

100

100

100

100

101

102

103

101

102

103

101

102

103

101

102

103

T

T

T

T

102

Wiki

102

PubMed

102

DMMath

102

ArXiv

b 101

b 101

b 101

b 101

100

100

100

100

101

102

103

101

102

103

101

102

103

101

102

103

T

T

T

T

is not surprising for DM-Mathematics because its documents consist of independent mathematical questions as shown in Figure 6. In Figure 4 (right), we plot the partial autocorrelation function for each of the 8 domains against time lag (context length). Indeed, we see that DM-Mathematics shows markedly less dependence compared to the other domains. The second notable observation is the relatively larger value of H = 0 . 79 in GitHub, indicating more structure in code. This is in agreement with earlier findings by Kokol and Podgorelec [35] who estimated LRD in computer languages to be greater than in natural language. In Table 2, we compare the three fractal parameters S , H and J using different families of LLM and different model sizes. Overall, we observe that the parameters are generally robust to the choice of the architecture.

Downstream Performance. By definition, fractal parameters are calculated on the sequence of negative log-probability scores after normalizing them to zero-mean and unit variance. Hence, they may offer an assessment of downstream performance that improves upon using a perplexity-based metric like bits-per-byte (BPB) alone. To test this hypothesis, we evaluate the 12 models in Table 2 on challenging downstream zero- and few-shot benchmarks focusing on language understanding and reasoning. We include results for 0-shot (0S) and 3-shot (3S) evaluation for BIG-Bench Hard tasks [63, 64] reporting both direct and chain-of-thought (CoT) prompting results following Chung et al. [13]. In addition we report 0-shot and 5-shot (5S) MMLU [30], and 8-shot (8S) GSM8K [14] with CoT. Raw accuracy is reported for all tasks. BBH and MMLU scores are averaged across all 21 tasks and 57 subjects, respectively. These benchmarks are quite diverse and include tasks such as logical deduction, arithmetic, translation error detection, disambiguation, as well as general knowledge (e.g. history, computer science, law, sports, movies, etc). All prompt templates for our evaluation are taken from Chung et al. [13], Longpre et al. [42], which we refer the reader to for more details. We prompt all models using a 2048 context length. See Table 8 of Appendix C for the full results.

The first (surprising) observation is that the median Hurst parameter is itself strongly correlated with the BPB scores with an absolute Pearson correlation coefficient of 0.83, even though the Hurst exponent is calculated after normalizing all token losses to zero-mean and unit variance! Informally,

this implies that second-order statistics on the sequence of token losses of a particular model can predict its mean! Self-similarity exponent, by contrast, has an absolute correlation of 0.23 with BPB.

Figure 7 displays downstream performance against both the median Hurst exponent and the median BPB score, where median values are calculated on the 8 domains in The Pile benchmark listed in Table 1. In general, both the BPB score and the median Hurst are good predictors of downstream performance. However, we observe that improvements in BPB alone without impacting the median Hurst exponent do not directly translate into improvements downstream. This is verified quantitatively in Table 3 (middle), which reports the adjusted R 2 values - the proportion of variance in each downstream metric that can be predicted using BPB, H , or by combining them together into H B = 1 / BPB+H , with BPB replaced with its reciprocal so that higher values are better. We observe that H B yields indeed a stronger predictor of downstream performance. Hence, while H and BPB are correlated, combining them yields a better predictor, so each of H and BPB conveys useful information not captured by the other metric. See Appendix C for similar analysis using the exponents S and J .

Context Length at Training Time ( Negative Result ). Finally, we present a negative result. Selfsimilarity and LRD point to an intriguing possibility: the importance of training the model with extensive contexts in order to capture the fractal-nature of language, which may elevate the model's capabilities regardless of the context length needed during inference. To test this hypothesis, we pretrain three decoder-only T5.1.1 models with 1B parameters on SlimPajama-627B [62] for up to 100B tokens using three context lengths: 2K, 4K and 8K, all observing the same number of tokens per batch. We use SlimPajama-627B instead of C4 because most documents in C4 are short ( ≈ 94% of them are < 2 K tokens in length). Refer to Appendix A for details. These models are, then, evaluated on the same downstream benchmarks listed in Figure 7. As shown in Table 3 (right) however, we do not observe any improvements in performance with context length in this particular setup.

4 Related Works and Directions for Future Research

The statistical attributes of human language have long piqued scholarly curiosity. One example is Zipf's law, which Shannon leveraged to estimate the entropy of English to be around 1 bit per letter [60], but his calculation did not consider second-order statistics. More recently, Eftekhari [18] proposed a refinement to Zipf's law, suggesting its application to letters rather than words. Another related result is Heap's law, which states that the number of unique words is a power law function

0.62

0.67

0.72

0.4

1.1

Median BPB

0S BBH Direct

0.62

0.67

0.72

0.4

1.1

Median BPB

0S MMLU

0.62

0.67

0.72

0.4

1.1

Median BPB

3S BBH Direct

0.62

0.67

0.72

0.4

1.1

Median BPB

3S BBH CoT

0.62

0.67

0.72

0.4

1.1

Median BPB

5S MMLU

0.62

0.67

0.72

0.4

1.1

Median BPB

8S GSM8K CoT

0.62

0.67

0.72

0.4

1.1

Median BPB

0S BBH+MMLU

0.62

0.67

0.72

0.4

1.1

Median BPB

FS BBH+MMLU+GSM8K

of the document's length [29]. However, both Zipf's and Heap's laws are invariant to the semantic ordering of text, so they do not capture important aspects, such as long-range dependence (LRD) [49].

In terms of self-similarity in language, the Menzerath-Altmann law stipulates a self-similar behavior in the following sense: when the size of a language construct increases, the size of its constituents decreases, and this happens at all scales [49, 4]. In Ausloos [9], the authors model texts as a time series by replacing a word with its length. After that, they study the fractal behavior of language. However, as mentioned in [22], replacing a word with its length is invalid because it is not translationindependent (i.e. one could map every word to an arbitrary token, including tokens of equal length). In our work, we model language as a series of bits calculated from conditional entropies, reflecting the intrinsic structure of the language itself, inspired by findings in linguistics such as [28, 22, 41]. The existence of self-similarity in language is attributed to its hierarchical nature [68, 3], such as duality of patterning [37].

In Najafi and Darooneh [49], the authors define a fractal dimension for each word. Informally, they examine the recurrence of a single, predetermined word as a binary series, similar to the approach used in Altmann et al. [3]. However, this only applies to individual words and cannot model higherlevel clauses. For instance, it does not distinguish between 'time' in the phrase 'once upon a time' and 'time' in 'space and time.' Kokol and Podgorelec [35] estimate LRD in natural language, and suggest that its LRD is close to that of pure noise! They conjecture this was due to their use of ASCII encoding. In computer languages, they observe LRD and suggest it is because they are formal.

Besides the above concerns in prior studies that examined the self-similar structure in language, another concern is that they sometimes give extremely large values of the fractal dimension, sometimes exceeding 10 [4]! Such values are difficult to interpret because the fractal dimension D should fall in D ∈ [1 , 2] for time series. We do not observe such issues in our analysis. In our case, D = 1 . 41 ± 0 . 08 .

Limitations and Future Research. Our analysis is currently limited to the English language so it may not apply to other languages that differ significantly. For instance, some languages such as Pirahã (spoken in the Amazon) do not have a recursive structure like most languages do [20]. We also do not model the semantic or lexical form of language. While our information-theoretic approach is well-founded and captures the intrinsic complexity of language, it does not account for the semantic nuances that contribute to meaning. Thirdly, self-similarity may explain why parameter sharing, such as in ALBERT [38], can be successful but exploiting self-similarity more directly in LLMs could lead to further optimizations. Exploring these aspects are promising directions for future research.

5 Concluding Remarks

In this work, we highlight intriguing insights into the underlying fractal structure of language and how it may be interconnected with the remarkable capabilities of LLMs. Our formalism quantifies properties of language that may have been suspected, but not previously formally shown. In particular, the need in LLMs to balance between short- and long-term contexts is reflected in the self-similar

structure of language, while long-range dependence is quantifiable using the Hurst parameter. For instance, the absence of LRD in DM-Mathematics is reflected in its Hurst parameter of H ≈ 0 . 5 . Interestingly, the estimated median Hurst value of H = 0 . 70 ± 0 . 09 in language reflects an intriguing balance between predictability and noise that is similar to many other phenomena, and combining both H with BPB together yields a stronger predictor of downstream performance. We carry out an extensive comparative analysis across different domains and model architectures, revealing that fractal parameters are generally robust. We hope that future research can further probe into these fractal properties, unearthing deeper understandings of the relation between intelligence and language.

References

Abry, P., Gonçalvés, P., and Flandrin, P. (1995). Wavelets, spectrum analysis and 1/f processes. Wavelets and statistics , pages 15-29. 2

Alabdulmohsin, I. M. (2018). Summability calculus: A comprehensive theory of fractional finite sums . Springer. 6

Altmann, E. G., Cristadoro, G., and Esposti, M. D. (2012). On the origin of long-range correlations in texts. Proceedings of the National Academy of Sciences , 109(29):11582-11587. 2, 4, 9

Andres, J. (2009). On de Saussure's principle of linearity and visualization of language structures. Glottotheory , 2(2):1-14. 9

Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Petrov, S., Johnson, M., Antonoglou, I., Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., et al. (2023a). Gemini: A family of highly capable multimodal models. arXiv:2312.11805v1 [cs.CL] . 1

Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., Chu, E., Clark, J. H., Shafey, L. E., Huang, Y., Meier-Hellstern, K., Mishra, G., Moreira, E., Omernick, M., Robinson, K., Ruder, S., Tay, Y., Xiao, K., Xu, Y., Zhang, Y., Abrego, G. H., Ahn, J., Austin, J., Barham, P., Botha, J., Bradbury, J., Brahma, S., Brooks, K., Catasta, M., Cheng, Y., Cherry, C., Choquette-Choo, C. A., Chowdhery, A., Crepy, C., Dave, S., Dehghani, M., Dev, S., Devlin, J., Díaz, M., Du, N., Dyer, E., Feinberg, V., Feng, F., Fienber, V., Freitag, M., Garcia, X., Gehrmann, S., Gonzalez, L., Gur-Ari, G., Hand, S., Hashemi, H., Hou, L., Howland, J., Hu, A., Hui, J., Hurwitz, J., Isard, M., Ittycheriah, A., Jagielski, M., Jia, W., Kenealy, K., Krikun, M., Kudugunta, S., Lan, C., Lee, K., Lee, B., Li, E., Li, M., Li, W., Li, Y ., Li, J., Lim, H., Lin, H., Liu, Z., Liu, F., Maggioni, M., Mahendru, A., Maynez, J., Misra, V., Moussalem, M., Nado, Z., Nham, J., Ni, E., Nystrom, A., Parrish, A., Pellat, M., Polacek, M., Polozov, A., Pope, R., Qiao, S., Reif, E., Richter, B., Riley, P., Ros, A. C., Roy, A., Saeta, B., Samuel, R., Shelby, R., Slone, A., Smilkov, D., So, D. R., Sohn, D., Tokumine, S., Valter, D., Vasudevan, V., Vodrahalli, K., Wang, X., Wang, P., Wang, Z., Wang, T., Wieting, J., Wu, Y., Xu, K., Xu, Y., Xue, L., Yin, P., Yu, J., Zhang, Q., Zheng, S., Zheng, C., Zhou, W., Zhou, D., Petrov, S., and Wu, Y. (2023b). PaLM 2 technical report. arXiv:2305.10403v3 [cs.CL] . 4, 17

Apostol, T. M. (1999). An elementary view of Euler's summation formula. The American Mathematical Monthly , 106(5):409-418. 6

Aref, S. (1998). Hurst phenomenon and fractal dimensions in long-term yield data. In Conference on Applied Statistics in Agriculture . 2, 5

Ausloos, M. (2012). Generalized Hurst exponent and multifractal function of original and translated texts mapped into frequency and length time series. Physical Review E , 86(3):031108. 1, 4, 9

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs. 14

Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y ., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T., and Zhang, Y . (2023). Sparks of artificial general intelligence: Early experiments with GPT-4. 1

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022). PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 . 4, 14, 17

Chung, H. W., Hou, Le and, L. S., Zoph, B., Tay, Y., Fedus, W., and et al. (2022). Scaling instruction-finetuned language models. arXiv:2210.11416v5 [cs.LG] . 7

Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv:2110.14168v2 [cs.LG] . 7

Cover, T. M. (1999). Elements of information theory . John Wiley & Sons. 4

Crovella, M. E. and Bestavros, A. (1995). Explaining world wide web traffic self-similarity. Technical report, Boston University Computer Science Department. 2, 5

Efron, B. and Tibshirani, R. J. (1994). An introduction to the bootstrap . CRC press. 5, 15

Eftekhari, A. (2006). Fractal geometry of texts: An initial application to the works of Shakespeare. Journal of Quantitative Linguistics , 13(2-3):177-193. 8

Embrechts, P. and Maejima, M. (2000). An introduction to the theory of self-similar stochastic processes. International journal of modern physics B , 14(12n13):1399-1420. 1

Everett, D. (2005). Cultural constraints on grammar and cognition in pirahã: Another look at the design features of human language. Current anthropology , 46(4):621-646. 9

Feller, W. (1951). The Asymptotic Distribution of the Range of Sums of Independent Random Variables. The Annals of Mathematical Statistics , 22(3):427 - 432. 2

Futrell, R. and Hahn, M. (2022). Information theory as a bridge between language function and language form. Frontiers in Communication , 7:657725. 3, 4, 9

Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. (2020). The Pile: An 800GB dataset of diverse text for language modeling. arXiv:2101.00027v1 [cs.CL] . 4, 5, 8

Geweke, J. and Porter-Hudak, S. (1983). The estimation and application of long memory time series models. Journal of time series analysis , 4(4):221-238. 2

Gibson, E., Futrell, R., Piantadosi, S. P., Dautriche, I., Mahowald, K., Bergen, L., and Levy, R. (2019). How efficiency shapes human language. Trends in cognitive sciences , 23(5):389-407. 3

Gneiting, T. and Schlather, M. (2004). Stochastic models that separate fractal dimension and the Hurst effect. SIAM Review , 46(2):269-282. 1, 5, 6

Goldberger, A. L., Amaral, L. A., Hausdorff, J. M., Ivanov, P. C., Peng, C.-K., and Stanley, H. E. (2002). Fractal dynamics in physiology: alterations with disease and aging. Proceedings of the national academy of sciences , 99(suppl_1):2466-2472. 5

Hale, J. (2001). A probabilistic earley parser as a psycholinguistic model. In Second meeting of the north american chapter of the association for computational linguistics . 3, 9

Heaps, H. S. (1978). Information retrieval, computational and theoretical aspects . Academic Press. 9

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 . 7

Hurst, H. E. (1951). Long-term storage capacity of reservoirs. Transactions of the American society of civil engineers , 116(1):770-799. 2, 5

Jouppi, N. P., Yoon, D. H., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., and Patterson, D. (2020). A domain-specific supercomputer for training deep neural networks. Communications of the ACM , 63(7):67-78. 14

Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., HatfieldDodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 . 4

Kidd, C. and Hayden, B. Y. (2015). The psychology and neuroscience of curiosity. Neuron , 88(3):449-460. 3

Kokol, P. and Podgorelec, V. (2000). Complexity and human writings. Complexity , 7:1-6. 4, 7, 9

Kolmogorov, A. N. (1940). Wienersche spiralen und einige andere interessante kurven in hilbertscen raum, cr (doklady). Acad. Sci. URSS (NS) , 26:115-118. 1

Ladd, D. R. (2012). What is duality of patterning, anyway? Language and Cognition , 4(4):261273. 9

Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 . 9

Leland, W. E., Taqqu, M. S., Willinger, W., and Wilson, D. V. (1994). On the self-similar nature of Ethernet traffic. IEEE/ACM Transactions on networking , 2(1):1-15. 2

Leland, W. E. and Wilson, D. V. (1991). High time-resolution measurement and analysis of LAN traffic: Implications for LAN interconnection. In IEEE INFCOM . 2

Levy, R. (2008). Expectation-based syntactic comprehension. Cognition , 106(3):1126-1177. 3, 9

Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., and Roberts, A. (2023). The flan collection: designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org. 7

Mandelbrot, B. (1967). How long is the coast of Britain? Statistical self-similarity and fractional dimension. science , 156(3775):636-638. 6

Mandelbrot, B. (2002). Gaussian self-affinity and fractals: globality, the earth, 1/f noise, and R/S . Springer Science and Business Media. 3

Mandelbrot, B. B. (1982). The fractal geometry of nature . WHfreeman New York. 1

Mandelbrot, B. B. and Wallis, J. R. (1968). Noah, Joseph, and operational hydrology. Water resources research , 4(5):909-918. 6

Mollica, F., Bacon, G., Zaslavsky, N., Xu, Y., Regier, T., and Kemp, C. (2021). The forms and meanings of grammatical markers support efficient communication. Proceedings of the National Academy of Sciences , 118(49):e2025993118. 3

Montemurro, M. A. and Pury, P. A. (2002). Long-range fractal correlations in literary corpora. Fractals , 10(04):451-461. 4

Najafi, E. and Darooneh, A. H. (2015). The fractal patterns of words in a text: a method for automatic keyword extraction. PloS one , 10(6):e0130617. 2, 4, 9

OpenAI (2023). GPT-4 technical report. arXiv:2303.08774v4 [cs.CL] . 1

Paxson, V. and Floyd, S. (1995). Wide area traffic: the failure of Poisson modeling. IEEE/ACM Transactions on networking , 3(3):226-244. 2

Peng, C.-K., Buldyrev, S. V., Goldberger, A. L., Havlin, S., Sciortino, F., Simons, M., and Stanley, H. E. (1992). Long-range correlations in nucleotide sequences. Nature , 356(6365):168170. 4

Perfors, A., Tenenbaum, J., Gibson, E., and Regier, T. (2010). How recursive is language? a bayesian exploration. Recursion and human language , pages 159-175. 2

Pilgrim, I. and Taylor, R. P. (2018). Fractal analysis of time-series data sets: Methods and challenges. In Ouadfeul, S.-A., editor, Fractal Analysis , chapter 2. IntechOpen, Rijeka. 2, 6

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910.10683v4 [cs.LG] . 4, 14

Roberts, A., Chung, H. W., Levskaya, A., Mishra, G., Bradbury, J., Andor, D., Narang, S., Lester, B., Gaffney, C., Mohiuddin, A., Hawthorne, C., Lewkowycz, A., Salcianu, A., van Zee, M., Austin, J., Goodman, S., Soares, L. B., Hu, H., Tsvyashchenko, S., Chowdhery, A., Bastings, J., Bulian, J., Garcia, X., Ni, J., Chen, A., Kenealy, K., Clark, J. H., Lee, S., Garrette, D., Lee-Thorp, J., Raffel, C., Shazeer, N., Ritter, M., Bosma, M., Passos, A., Maitin-Shepard, J., Fiedel, N., Omernick, M., Saeta, B., Sepassi, R., Spiridonov, A., Newlan, J., and Gesmundo, A. (2022). Scaling up models and data with t5x and seqio . 14

Roche, S., Bicout, D., Maciá, E., and Kats, E. (2003). Long range correlations in DNA: scaling properties and charge transfer efficiency. Physical review letters , 91(22):228101. 4

Samorodnitsky, G. (2006). Long memory and self-similar processes. In Annales de la Faculté des sciences de Toulouse: Mathématiques , volume 15, pages 107-123. 5

Schenkel, A., Zhang, J., and Zhang, Y.-C. (1993). Long range correlation in human writings. Fractals , 1(01):47-57. 4

Shannon, C. E. (1951). Prediction and entropy of printed English. Bell system technical journal , 30(1):50-64. 8

Shazeer, N. and Stern, M. (2018). Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning , pages 4596-4604. PMLR. 14

Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. (2023). SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. 8, 14

Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 . 7

Suzgun, M., Scales, N., Scharli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., and Wei, J. (2022). Challenging BIG-Bench tasks and whether chain-of-thought can solve them. arXiv:2210.09261v1 [cs.CL] . 7

Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. (2024). Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 . 17, 18

Watkins, N. (2019). Mandelbrot's stochastic time series models. Earth and Space Science , 6(11):2044-2056. 2, 3, 5, 6

Wikimedia (2023). Downloads. 18

Willinger, W., Taqqu, M. S., Leland, W. E., and Wilson, D. V. (1995). Self-similarity in highspeed packet traffic: analysis and modeling of Ethernet traffic measurements. Statistical science , pages 67-85. 2, 5, 6, 9

Willinger, W., Taqqu, M. S., Sherman, R., and Wilson, D. V. (1997). Self-similarity through high-variability: statistical analysis of Ethernet LAN traffic at the source level. IEEE/ACM Transactions on networking , 5(1):71-86. 2

A Experiment Details

All of our experiments are conducted in JAX/Flax [10] using the open source T5X framework [56].

T5 baselines in Table 2 and 3 are pretrained from scratch using the open source T5.1.1 decoder-only architecture from the T5X library. 3 . We pretrain using a causal language modeling objective over the C4 corpus with the default T5 vocabulary as per Raffel et al. [55]. Training is done for 500k steps with a sequence length of 1024 and batch size of 512, resulting in a total of 262B tokens seen during pretraining. We optimize our model with the Adafactor [61] optimizer with an inverse square root learning rate schedule, 1k warmup steps, and an initial learning rate of 1e-2. Models are trained using 256 TPUv5e chips [32].

T5 context length ablation experiments in Table 3 are trained with the same pretraining objective but over the SlimPajama-627B corpus [62] and using a modified version of the T5 vocabulary that preserves whitespace and introduces byte-fallback for out of vocabulary tokens. This is similar to Chowdhery et al. [12], but preserving the original T5 vocabulary. Models with sequence lengths 2048, 4096, 8192 are trained with batch sizes of 512, 256, and 128 respectively to preserve the number of tokens seen per batch and overall training steps. We train all models for 100k steps, using the same learning rate schedule described above. Hence, all models observe 100B tokens.

B Full Results

In this section, we provide the full list of parameters calculated for each combination of LLM and domain. We use bootstrapping [17] to estimate the error margin.

C Predicting Downstream Performance

Table 8 presents detailed downstream performance results, along with corresponding upstream metrics.

In Table 9, we repeat the same analysis in Section 3 using the adjusted R 2 coefficient, but with the self-similarity S and Joseph exponents J . Unlike in the median Hurst exponent, we do not observe

any improvement when combining perplexity scores with the self-similarity exponent S or the Joseph exponent J .

Please note that our results are not directly comparable to all previous published results for the same models; please cite the original results from [12, 6]. Here, we only aim for a fair comparison between models: only pretrained models without instruction tuning are used, we do not optimize any prompts for each model, and we evaluate all models using only a 2K sequence length.

D Gemma-2B Results

In this section, we present empirical results using the publicly released Gemma-2B checkpoint [65]. Results are shown in Figures 8 and 9.

Gemma-2B is much smaller than all of the models we have used previously. Yet, we generally observe similar conclusions. First, we have a self-similar structure with near perfect linear fits in a log-log plots. Second, we also observe power laws using the rescaled-range analysis, with a Hurst exponent of about 0.7 in most domains except DM Mathematics (smallest Hurst exponent of about 0.58) and GitHub (largest Hurst exponent of about 0.82), in general agreement to the rest of the models. Both the self-similarity and Hurst parameters are provided in Table 10.

E Comparison to n-Gram Models

Here, we report some early investigations that examine how fractal parameters change as one includes longer contexts during inference. The setup used in this experiment is slightly different from what we use throughout the rest of the paper. Specifically, we take the Wikipedia dataset

10 1

10 2

Scale

10

4

10

3

10

2

p

ArXiv

10 1

10 2

Scale

10

4

10

3

10

2

p

DM Mathematics

10 1

10 2

Scale

10

4

10

3

10

2

p

PubMed Central

10 1

10 2

Scale

10

4

10

3

10

2

p

Wikipedia (en)

10 1

10 2

Scale

10

4

10

3

10

2

p

Pile-CC

10 1

10 2

Scale

10

4

10

3

10

2

p

FreeLaw

10 1

10 2

Scale

10

4

10

3

10

2

p

Github

10 1

10 2

Scale

10

4

10

3

10

2

p

OpenWebText2

10 2

Scale

10 0

10 1

10 2

R

/

S

ArXiv

10 2

Scale

10 0

10 1

10 2

R

/

S

DM Mathematics

10 2

Scale

10 0

10 1

10 2

R

/

S

PubMed Central

10 2

Scale

10 0

10 1

10 2

R

/

S

Wikipedia (en)

10 2

Scale

10 0

10 1

10 2

R

/

S

Pile-CC

10 2

Scale

10 0

10 1

10 2

R

/

S

FreeLaw

10 2

Scale

10 0

10 1

10 2

R

/

S

Github

10 2

Scale

10 0

10 1

10 2

R

/

S

OpenWebText2

( wikipedia/20230601.en ) dataset [67]. We split each document of length >4K words along word boundaries and constrain the context length during inference in PaLM-8B to n words, for n ∈ { 1 , 16 , 32 , 64 , 128 , 256 , 512 , 1024 , 2048 } . Hence, the language model now resembles an n -gram model. We calculate probability scores and calculate fractal parameters accordingly. Figure 10 shows how both the self-similarity exponent S and the Hurst parameter H change as a function of n .

We observe that H increases monotonically with context length as expected, since it implies more predictability. However, even in a 1-gram model, H can be larger than 1/2 because the words themselves are not independent of each other, so the sequence of probability scores can still contain dependence over time in a 1-gram model.

10 0

10 1

10 2

10 3

Inference Context (in words)

0.50

0.55

0.60

0.65

S

10 0

10 1

10 2

10 3

Inference Context (in words)

0.60

0.65

0.70

0.75

0.80

H

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 4

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Guidelines: We follow well-established definitions throughout our analysis.

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We only release a portion of the code that can be used to calculate fractal parameters.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 2.1 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We use the bootstrap method to estimate error margins and report those in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work studies the fractal structure of language. We do not foresee any potential negative societal impact this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original creators/owners of all datasets and model checkpoints that we use in our analysis.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a detailed application of ideas from fractal geometry to natural language data, using language models to compute the relevant information-theoretic properties.  They find, in particular, tell-tale evidence of self-similarity (common structure across scales) and long-range dependencies.  A particularly interesting result, in my opinion, is that using some of the estimated fractal parameters can help predict LMs' downstream performance over and above what can be predicted from raw LM performance (bits-per-byte), suggesting that some of this performance can be explained by their discovery of this fractal structure.  While the paper can be a bit dense and contains a huge range both of theoretical background and experimental results that make it a little hard to read, I do believe that it pays off to understand and makes a nice contribution that will be of interest to many people in the field.\n\nSTRENGTHS:\n* Provides a thorough and detailed application of fractal geometry to natural language data.\n* Use of the new fractal metric helps predict downstream task performance more than just BPB alone.\n* Shows that the main findings are robust to the choice of LLM that is used for estimating information-theoretic properties of text.\n\nWEAKNESSES:\n* Could cite more literature about e.g. dependency lengths (https://doi.org/10.1073/pnas.1502134112) and other known properties of natural language.  For instance, \"duality of patterning\" (https://doi.org/10.1515/langcog-2012-0015) has been used to refer to the fact that morphemes->words and words->sentences exhibit similar structural properties, which is akin to self-similarity. \n* Although the paper is mostly self-contained, it can be very dense for readers not very familiar with the mathematics of fractals.\n\nQUESTIONS:\n* Lines 46-47: if the reader is not familiar with S, it's not clear what the value means and why it has the implications that it does.  Can you explain more at this point or move these remarks to a bit later?\n* Have you checked whether a power-law is genuinely the best fit to the data, versus other laws that often resemble them (e.g. linear in log-log)? See e.g. https://doi.org/10.1126/science.1216142\n* Can you say more about the use of LLMs to compute and represent the information-theoretic quantities in this context?  As a way of making sure that we are learning about language in particular and not just artifacts/properties of the models.\n\nLIMITATIONS:\nYes; I appreciated especially the mention of the results being English-only", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce a new perspective to that language is self-similar and predictability and self-similarity together imply long-range dependency, based on empirical analysis across different scales of LMs and information theoretic views. This new perspective may enable us to understand the strong capabilities of current causal LLMs. They studies three parameters, self-similarity (Hölder) exponent, the Hurst parameter, and the fractal dimension, as well as Joseph exponent. They also introduces a new metric that can more precisely approximately downstream performance than BPB.\n\nSTRENGTHS:\n- This paper provides some new interesting perspectives to advance our understanding of how LLMs acquire such strong capabilities after large-scale pre-training using simple autoregressive training objective. \n- They conduct large-scale analysis using LMs with different scale in three different model families. \n- They also empirically found that a median Hurst exponent can be a more reliable indicator of downstream performance than perplexity-based BPB. While perplexity has shown to strongly correlated with downstream performance, prior studies also show their limitations on predicting downstream performance, and this work may encourage future work to use this new metric instead.\n\nWEAKNESSES:\n- To my understanding, this work views language  as a  mere sequence of negative log probabilities while prior studies in linguistic / NLP often consider much more rich structures of languages and am not fully convinced the validity of analysis.\n- While the analysis includes three models, none of the models’ checkpoints aren’t publicly available (i.e., PaLM and PaLM2 and their newly trained T5-decoder only models) and followup work may not be able to reproduce the results. I’m curious why the authors didn’t test other models with openly available model checkpoints such as Llama 2, 3 / OLMo / Pythia. \n- Overall I found the paper is a bit hard to follow (e.g., Introduction discusses prior literature in depth before providing high-level overview or motivation, Experimental setups come write after preliminaries). Some restructures of sections or writing may improves readability of the paper.\n\nQUESTIONS:\n- Did authors try other models with publicly available model checkpoints so that people can reproduce the results? \n- Why did authors train T5-decoder model from scratch, instead of using T5 variant that went through next token prediction training (e.g., T5 LM adapt)?\n\nLIMITATIONS:\nThe authors provide limitation sections that include some of the limitations I was thinking about.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper draws connections between fractal patterns and language by evaluating properties such as self-similarity and long-range-dependency. Using a range of LLMs, they estimate the Holder exponent, Hurst parameter and fractal dimension for language from different domains, including web text, code, and math problems. They show that these parameters may have connections with LLM learning ability, demonstrating that using the median Hurst parameter can improve prediction of downstream model performance over only using bits-per-byte.\n\nSTRENGTHS:\n- The authors estimate fractal parameters across a range of datasets and LLMs, showing that they are fairly robust to choice of LLM and domain. For domains with significant deviation, such as DM-Mathematics, the authors are able to attribute this to the dataset's lack of long-range dependency. \n- The authors show that the Hurst parameter captures useful information for predicting downstream performance that is not captured by BPB alone.\n\nWEAKNESSES:\n- Even though fractal parameters were found to be consistent across LLMs, I think their stated contribution (this 'establish[es] that language is self-similar and long-range-dependent') is far too strong a claim. As the authors mention, their method ignores many important facets of language, such as semantic nuance, and is reliant on the current state of LLMs' ability to model language. \n- I am unclear how future work can build upon these insights about language, especially that 'exploiting self-similarity more directly' could lead to further LLM optimization.\n\nQUESTIONS:\nI don't follow the potential connection between self-similarity and the success of parameter sharing? (mentioned in Limitations section)\n\nTypos: \n- Line 131: such bits-per-byte (BPB)\n- Line 132: BPB is a widely used as a\n- Line 272: such as One example\n\nLIMITATIONS:\nYes, the authors discuss that their analysis is limited to English data and fails to capture the semantic meaning of language. However, as mentioned in Questions section, the possible connection between self-similarity and parameter-sharing is not obvious to me and could be further explained.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors try to reveal the existence of fractal structures to handle language in language modeling using recent language models based on the next token prediction. For that purpose, the authors rely on several aspects of fractal structures, which are self-similarity, long-range dependence, and information-theoretic complexity. The authors describe these aspects using metrics for typical phenomena under the assumption of fractal structure, which are self-similarity exponent, Hurst parameter, fragmental dimension, and Joseph effect. The experimental results show that the characteristics of modeling texts of ArXiv, Github, and Wikipedia data in The Pile validation split, whose tokens are longer than 4K, are along with their assumption. In the analysis of the downstream tasks, BBH, MMLU, and GSM8K also show that the task-solving performances are predictable from the language modeling performance based on their assumption, except for considering sequence lengths in training.\n\nSTRENGTHS:\n- Providing an assumption to unify characteristics of interpreting language and language modeling under the variously used approach, next token prediction may help solving various kinds of tasks related to natural languages.\n- The analysis is not only restricted to language modeling and the authors investigate the performance correlation between language modeling and downstream tasks based on their assumption.\n\nWEAKNESSES:\n- The target language is limited to English. Thus, the validity of expanding the insights in the paper to other languages is uncertain.\n- The assumed baseline in this analysis is a random sequence. The authors can use finite-state automatons (FSAs) or Context-free grammars (CFGs) to make sequences that are random but close to languages. Moreover, to focus on the success of recent pre-trained language models, you should have prepared N-gram language models as baselines.\n- How strongly the observation results fit the assumed distribution is not calculated mathematically, like model selection by Information Criteria such as Akaike's Information Criteria (AIC) and Bayesian information criteria (BIC).\n- There is a gap between the ability to mimic the characteristics of language and showing the required knowledge for questions. Thus, discussing the correlation between the performance of language modeling and downstream tasks is limited from this viewpoint. Hence, targeting generation tasks like summarization, story generation, and machine translation are more suitable for deepening the analysis.\n\nQUESTIONS:\n- In the analysis, the authors seem to only focus on the context length in training even though the inference of long sequential tokens is closer to the aspect of long distant dependencies between tokens in language. What is the main reason for this decision?\n\n[Comment] In presentation style, instead of showing the method name, showing the motivation or target phenomenon as the section or paragraph title, like \"Joseph effect\" -> \"Burstiness,\" may support the reader's understanding.\n\nLIMITATIONS:\nThe limitation does not include the gap between language modeling and solving downstream tasks.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}