Fractal Patterns May Illuminate the Success of Next-Token Prediction
Authors: Ibrahim Alabdulmohsin
Vinh Q. Tran
Mostafa Dehghani
Venue: NeurIPS 2024 poster (2024)
Keywords: fractal patterns
large language models
Hurst exponent
self-similarity
Joseph exponent
perplexity
information theory
Primary area: natural_language_processing
TL;DR: We study the fractal structure of language and show that fractal parameter can be used in predicting downstream performance better than perplexity-based metrics alone.

Abstract: We study the fractal structure of language, aiming to provide a precise formalism for quantifying properties that may have been previously suspected but not  formally shown. We establish that language is: (1) self-similar, exhibiting complexities at all levels of granularity, with no particular characteristic context length, and (2) long-range dependent (LRD), with a Hurst parameter of approximately 0.7.
Based on these findings, we argue that short-term patterns/dependencies in language, such as in paragraphs, mirror the patterns/dependencies over larger scopes, like entire documents. This may shed some light on how  next-token prediction can capture the structure of text across multiple levels of granularity, from words and clauses to broader contexts and intents. In addition, we carry out an extensive analysis across different domains and architectures, showing that fractal parameters are robust.
Finally, we demonstrate that the tiny variations  in fractal parameters seen across LLMs improve upon perplexity-based bits-per-byte (BPB) in predicting their downstream performance. We hope these findings offer a fresh perspective on language and the mechanisms underlying the success of LLMs.

Fractal Patterns May Illuminate the Success of Next-Token Prediction
Ibrahim Alabdulmohsin
Vinh Q. Tran Google Deepmind New York, USA
Mostafa Dehghani Google Deepmind Mountain View, USA dehghan @oopgal..@gggge...mom com
Google Deepmind Zürich, Switzerland bbomossinggoogle..o. com
vgtranggoogle.cm com
Abstract
We study the fractal structure of language, aiming to provide precise formalism
for quantifying properties that may have been previously suspected but not formally
shown. We establish that language is: (1) self-similar, exhibiting complexities at all levels of granularity, with no particular characteristic context length, and (2) long-range dependent (LRD), with a Hurst parameter of approximately H 0. 70 0.09. Based on these findings, we argue that short-term patterns/deepeddeciis in language, such as in paragraphs, mirror the patterns/dependencies over larger scopes, like entire documents. This may shed some light on how next-token prediction capture the structure of text across multiple levels of granularity, from words and clauses to broader contexts and intents. In addition, we carry out an extensive analysis across different domains and architectures, showing that fractal parameters are robust. Finally, we demonstrate that the tiny variations in fractal parameters seen across LLMs improve upon perplexity- based bits-per-by. (BPB) in predicting their downstream performance. We hope these findings offer a fresh perspective on language and the mechanisms underlying the success of LLMs.
Introduction
How does the training objective of predicting the next token in large language models (LLMs) yield remarkable capabilities? Consider, for instance, the two models: Gemini [5] and GPT4 [50]. These models have demonstrated capabilities that extend quantitative reasoning, summarization, and even coding, which has led some researchers to ponder there was more to intelligence than "on-the-fly improvisation" [1]]. While providing satisfactory explanation is difficult endeavor, a possible insight can be drawn from fractals and self-similarity. We elucidate the connection in this work.
Self-Similarity. Self-similar processes were introduced by Kolmogorov in 1940 [36]. The notion garnered considerable attention during the late 1960s, thanks to the extensive works of Mandelbrot and his peers [19]. Broadly speaking, object is called "self-similar" if it is invariant across scales, meaning its statistical or geometric properties stay consistent irrespective of the magnification applied it (see Figure 1). Nature and geometry furnish us with many such patterns, such coastlines, snowflakes, the Cantor set and the Kuch curve. Despite the distinction, self-similarity is often discussed in the context of "fractals," another term popularized by Mandelbrot in his seminal book The Fractal Geometry of Nature [45]. However, the two concepts are different [26]. See Section 2.
In language, in particular, there have been studies arguing for the presence of self-similar structure. Nevertheless, due to computational constraints, it was not feasible holistically model the joint probability distribution of language. As such, linguists often resorted to rudimentary approximations in their arguments, such as by substituting a word with its frequency or length [9], or by focusing on
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

20
2000
2000
4000
50
90
10o0
1500
2000
3000
1500
2000
30
70
2250
1375 Time
1500
1250
1375
1500
Time
Figure I: Manifestations of processes across different time scales. A region marked in red corresponds to the magnified plot beneath it. LEFT: The process exhibits self-similarity with rich details at all levels of granularity. It is an integral process (Xt)seN calculated from Wikipedia (see Section 2). RIGHT: Example of process that is not self-similar, looking smoother larger time scales.
the recurrence of a specific, predetermined word [49, These studies fall short of fully capturing the structure of language due to the simplifying assumptions they make, as discussed in Section 4.
Highlighting the self-similar nature of process have profound implications. For instance, conventional Poisson models for Ethernet traffic were shown to fail because traffic was self-similar [16, 39, 51, 69]. In such cases, recognizing and quantifying self-similarity had practical applications, such as in the design of buffers [40]. Similarly in language, we argue that self-similarity may offer fresh perspective
the mechanisms underlying the success of L.Mss. Consider the illustrative example shown in Figure 1, where the task is to predict the subsequent measurement in time series, specifically predicting next tokens in Wikipedia article (see Section 2 for details). The three plots in Figure (left) represent different manifestations of the same process observed across three distinct time scales. Notably, we observe rich, self-similar details, such as burstiness, in all of them. A well-established approach for quantifying self-similarig is the Hölder exponent [66], which we denote by S. In language, we find be S 0.59 = 0.08, confirming statistical self self- similariia
Why is this important? We hypothesize that since LLMs are trained to predict the future of a self- similar process, they develop proficiency in capturing patterns across multiple levels of granularity for two interconnected reasons. First, self-similarity implies that the patterns at the level of a paragraph are reflective of the patterns seen the level of whole text, which is reminiscent of the recursive structure of language [53]. Thus, recognizing short-term patterns can aide in learning broader contexts. Second, because language displays intricate patterns at all levels of granularity, would not be enough to rely only on the immediate context of sentence to predict the next token. Instead, the model needs to identify patterns higher levels of granularity; e.g. follow the direction of the argument and the broader intent. It must balance between short- and long-term contexts. Willinger al. [68] and Altmann al. [3] argue for self silmaaiiil in language due this hierarchical nature.
Long-range dependence. However, self-similarity alone not sufficient for a predictive model to exhibit anything resembling "intelligent" behavior. In fact, some self-similar processes, despite their intricate details, remain entirely unpredictable. A quintessential example is the simple Brownian motion, which is a Wiener process with independent increments. Its discrete analog is B, = > Ei. where E; ~ NN0 d²). Despite possessing rich details all gramularities, model trained to predict B, cannot learn anything useful from data since the process itself has independent increments.
Thus, for strong capabilities to emerge, the process must have some degree of predictability or dependence as well. One classical metric for quantifying predictability in a stochastic process is the Hurst parameter [31], developed by the hydrologist H. E. Hurst in 1951 while studying the Nile river. It is generally considered to be a robust metric [68], unlike the wavelet estimator [1] and the periodogram method [24] that can be sensitive to errors [54]. As discussed in Section 2.3, we find the Hurst parameter in language to be = 0.70 + 0.09. For context, H only takes values [o, 1]. A value H 0.5 implies predictability in the data, while H 0.5 indicates random increments.
While it is compelling that our estimate of H in language lies nearly midway between predictability (H = 1) and noise (H = 0.5), Hurst parameter of about 0.75 turns out to occur commonly in nature, including in river discharges, Ethernet traffic, temperatures, precipitation, and tree rings [16, 21, 8]. For agents that learn from data, such as LLMs, this value is also reminiscent of processing based

---

theories of curiosity, which suggest that a sweet spot of complexity exists (not too simple, nor too unpredictable) that facilities or accelerates learning [34].
Importantly, predictability and self-similarity together imply long-range dependence (LRD). This follows from the definition of self-similarity, where the patterns small scales mirror those at larger scales so, for example, the correlations established at micro levels are also pertinent at macro levels. LRD is arguably crucial for enhancing the functionality of predictive models because processes with only short-range dependence could be forecasted (somewhat trivially) with lookup tables that provide the likelihood of transitions over brief sequences. By contrast, this not possible in LRD processes whose contexts extend indefinitely into the past.
Information Theoretic Complexity. To define fractal parameters, we follow recent works such as [28, 22, 41, 47, 25] in adopting an information-theoress characterization of the complexity in language using minimal-length codes or surprise. This corresponds an intrinsic, irreducible description of language and the minimum compute overhead to comprehend/decode it [22], which also correlates well with actual reading times [28, 41]. this context, self-similarity means that the intrinsic complexity or surprise in language (measured in bits) cannot be smoothed out, even as we look into broader narratives. That is, surprising paragraphs will follow predictable paragraphs, in a manner that is statistically similar to how surprising sentences follow predictable sentences.
Analysis. How robust are these findings? To answer this question, we carry out an extensive empirical analysis across various model architectures and scales, ranging from IB to over 500B parameters. We find that fractal parameters are quite robust to the choice of the architecture.
However, there exists tiny variations across LLMs. Interestingly, we demonstrate that from a practical standpoint, these differences help in predicting downstream performance in LLMs compared to using verplexity-based metrics alone, such as bits-per-bytt (BPB). Specifically, we introduce a new metric and show that using predict downstream performance can increase the adjusted R from approximately 0.65 when using soiely BPB, to over 0.86 with the new metric?.
Statement of Contribution. In summary, we:
1. highlight how the fractal structure of language can offer a new perspective on the capabilities of LLMs,, and provide a formalism to quantify properties, such as long-range dependence.
2. establish that language is self-similar and long-range dependent. We provide concrete estimates language of the three parameters: the self-similarity (Hölder) exponent, the Hurst parameter, and the fractal dimension. We also estimate the related Joseph exponent.
3. carry out a comparative study across different model architectures and scales, and different domains, such ArXiv and GitHub, demonstrating that fractal parameters are robust.
4. connect fractal patterns with learning. Notably, show that "median" Hurst exponent improves upon perplexity-basee bits-per-byte (BPB) in predicting downstream performance.
Fractal Structure of Language
2.1 Preliminaries
Suppose we have a discrete-time, stationary stochastic process (II)EE,, with E[x,]= = 0 and E[ We will refer to (ar))en the increment process distinguish it from the infegral process (Xt).e defined by X, = Et Σi= Tk. While (zz) and (Xe)xeN are merely different representations of the same data, it is useful to keep both representations in mind. For example, self-similarity is typically studied in the context of integral processes whereas LRD is defined on increment processes.
In the literature, it is not uncommon to mistakenly equate parameters that are generally different. For example, the Hurst parameter H has had many definitions in the past that were not equivalent, and Mandelbrot himself cautioned against this [44]. The reason behind this is because different parameters can agree in the idealized fractional Brownian motion, leading some researchers to equate them in general [66]. We will keep the self-similarity exponent S and H separate in our discussion.
We release the code for calculating fractal parameters at: https: /t:g:://ttttttt com/ hamm/google-eeo research/ google- www.eeaccceeeeeeetteeeeeeeeeeeee
language
3

---

10"1
OpenWeb
10"1
GRhub
10"
FreeLaw
10"2
HleCC
0.10
0.10
0.10
a.10
10"2
10
10"2
10"2
10
101
10
10
10'
10'
10
RioMed
10"²
DMMath
10-3
AOll
10a5
10
10a
10
10
10a
10a
10
10'
Figure 2: Peak probability (T(T) (T) is plotted against the granularity level r (see Section 2.2). We observe power laws (T) T indicating self seff-similarrty, with median exponent of S 0.59 0.08.
107
OpenWeb
10²
Github
FreeLaw
10a
RlleC
107
102
10a
101
103
10
10a
10a
10a
10
10a
10P
10a
10"
10a
10"
10a
107
10a
10a
10a
PuoMed
DMMath
10a
55
102
102
10a
10a
10
10²
10a
10a
10a
10"
10
10a
10a
102
10a
10a
10a
Figure 3: Rescaled range R(n)/ (n) is plotted against the number normalized bits n. We observe power law R(n)/S(m) ~ n in all domains. When aggregating all datasets, H = 0.70 + 0.09.
Experimental Setup. In order to establish self-similarity and LRD in language, convert texts into sequences of bits using large language model (LLM). Specifically, we use PaLM2- the next token wa conditioned on its entire prefix W|r-1] Wgg Wi..... ... w)-1). As discussed in Section 1, this captures its intrinsic, irreducible description [22]. By the chain rule [15], the corresponding number of bits assigned to w, [pp1])] Unlike in prior works, which rely simplifications such as by sub- stituting word with its length [9] or by focusing on the recurrence of a single word [49, 3], we use the LLM to approximate the full joint distribution of language since LLMs are known to produce calibrated probability scores at the token level [33]. We carry out these calculations for prefixes of up to 2048 tokens (* 8 pages of text). With suitable normalization, such as bits bits-per-byte (BPB), one obtains a standardized description text, consistent across tokenizers. BPB widely used as a aenniizer-agostti metric to compare LM modeling performance, e.g. for The Pile [23].
(Unicorn) [6] calculate the probability
= - log p[:]] ((wp//p-1))...
Besides PaLM2, we also experiment and report on various model sizes of PaLM [12] and decoder- only T5 [55]. Namely, we report results for models: PaLM2 XXS (Gecko), XS (Otter), S (Bison), M, and L (Unicorn); PaL. 8B, 62B, 540B; and decoder-only 1.1 at Base (1 10M), Large (341M), (1.2B), and XXL (5B) sizes. For PaLM and P2LM2, we use the checkpoints pretrained in Chowdhery ct al. [12] and Anil et al. [6]. All T5.1.1 decoder baselines, on the other hand, are trained with a executed
casual language modeling objective for 262B tokens of C4 [55]. All experiments Tensor Processing Units (TPUs). More details on how we train T.. baselines are in Appendix A.
Once is computed for document, we follow standard definitions in constructing the increment process (2r) ( by normalizing 2; to have zero-mean and unit variance. Intuitively. fractal pa- rameters are intended to measure a fundamental property of the process (e.g. LRD) that should not be affected by scale, hence the normalization. The integral process (Xt)teN is calculated based on (ZN)) as described earlier and depicted in Figure I (top). Normalizing bits (to have zero mean and unit variance) models language as a random walk. is a standard approach used extensively in the literature in various contexts, such in DNA sequences [52, 57, 48, 35, 59].

---

1.5
TeetWebText2
Wikipedia
OpenWebText2
Github
PubMed Central
10-2
Github
FreeLaw
DM Mathematics
FreeLaw
1.0
Pil-CC
Arkiv
Pilc-CC
PACF
Wikipedia (en)
PubMed Central
DM Mathematics
ADXN
10-3
0.0
10-3
10
200
400
600
800
1000
Lag
Figure 4: LEFT: Estimates of the self-similarity exponent S are generally robust to the choice of c. RIGHT: The partial auto-correlation function calculated across domains. DM Mathematics has a much shorter dependence compared to the rest of the domains, in agreement with its Hurst parameter.
For analysis, we use The Pile validation split [23], consisting of 22 subdomains such as Wikipedia and GitHub. We restrict analysis to sufficiently-lng documents of length > 4K tokens and use the first 2K tokens only, sidestep potential effects of the finite length of documents and the model context. To mitigate noise, only domains with > 1K documents are compared; we report results for them separately and their median. We use bootstrapping [17] to estimate the error margin.
Notation. We write f(x) ~ xf if [(x) = x L(x) for some function that satisfies [(xx)/((x) + 1 as x oo for all t (xx) = logæ. When f(r) z², we abuse terminology slightly by referring to f(xx) as power law.
Examples of slowly varying functions are constants L(x) c and
2.2 Self-similarity exponent Scale invariance
An integral process is said to be self-similar if it exhibits statistical self-similarity. More precisely, (Xt)een is self-similar if (Xrt)KeN is distributionally equivalent to (r Xt)teN for some exponent S. Thus, scaling of time is equivalent to an appropriate scaling of space. We will refer to T as the granularity level and to the exponent as the self-similarity or Hölder exponent [66]. Many time series in nature exhibit self-similar structures, such as human blood pressure and heart rate [27].
One approach for calculating is follows. Fix 1 and denote the 7-increments by (X++r Xt)LEN. These would correspond, for instance, to the number of bits used for clauses, sentences, paragraphs and longer texts as increases. In terms of the increment process (1E)) this corresponds to aggregating increments into "bursts". Let pe (T) be the probability mass of the event [|x++r -X, <
d)eN Then, S can be estimated by fitting a power law relation (T(T) ~ - [66]. Generally, is robust to the choice of E [10-3, 10-2) as shown in Figure 4 (left) so we fix it to = 5 x 10-3
Figure 2 plots the probability (T) against using P2LM2-L. We indeed observe power law relation over at least two orders of magnitude; i.e. linear in a log-log scale, with a median self-similarity exponent of S 0.59 + 0.08. Section 3 shows that the median S is robust the choice of the LLM.
2.3 Hurst parameter - Long-range dependence
The Hurst parameter H E [0, 1] quantifies the degree of predictability or dependence over time [31]. It is calculated using the so so-called rescaled-range (R/S) analysis. Let (x)))E be an increment process. For each N, write y/ = 2, Ei - T& and Y, Et respectively, as R(n) = max Y min:sn Y, and S(n) = ((*))en)) where or is the standard deviation. Then, the Hurst parameter H is estimated by fitting a power law relation R(n)/S(n) ~ n As stated earlier, for completely random processes, such simple Brownian motion, it can be shown that = 1/2. In addition, > 1/2 implies dependence over time [16, 68, 8].
k= Yt. The range and scale are defined,
Writing P.. [[ for the autocovariance function of the increment process (It)TEN. the Hurst parameter satisfies H = - 3/2 when Pa. ~ n -8 as n -> co [26, 16]. Since in self-similar processes, H > 1/2 implies long-range dependence (LRD), LRD equivalent to the condition that the autocovariances are not summable. In terms of the integral process, it can be shown that [58]: Var(X,.)
lim
+ Σ Pi. Hence, if H < 1/2, the auto-covariances are summable and
5

---

OpenWeb
GitHub
FreeLaw
PáeCC
Wiki
PubMed
Math
ArXiv
S
0.53 + .05
0.60 + .05
0.61 + .05
0.56 + .03
0.62 + .02
0.60 + .07
0.42 + .03
0.70 + .03
0.68 1 .01
0.79 + .01
0.68
+: .00
0.70 4: .00
0.74 1: .01
0.65
1 .00
0.30 + .01
0.72 .01
0.46 .01
0.49 1: .00
0.49 di .00
0.50 .00
0.52 .00
0.44 1: .00
0.28 d: .00
0.49 .00
Table I: A comparison of the fractal parameters across different domains with > 1000 documents each in The Pile benchmark (see Section 2.1 for selection criteria). DM-Mathematics is markedly different because each document consists of questions, with no LRD.
Var(Xn) grows, at most, linearly fast on n. On the other hand, if the process has LRD, (ar(X,) grows superlinearly on In particular, using the Euler-Maclaurin summation formula [7, 2]. one obtains Var(Xn) ~ n2 if H 1/2. Figure plots the rescaled range R(n)/S(v)) against n. We observe a power law relation with median Hurst parameter of H = 0.70 + 0.09.
2.4 Fractal dimension - Complexity all levels
Broadly speaking, the fractal dimension of an object describes its local complexity. For a geometric object Z. such as the Koch curve, let be a chosen scale (c.g. a short ruler for measuring lengths or small square for areas). Let (r) be the minimum number of objects of scale 7 that cover Z; ie. contain entirely. Then, the fractal dimension of Z, also called its Hausdorff dimension, is:
[54]. For example, line has a fractal dimension 1, in agreement with its topological dimension, because (r) C/T for some constant C > 0.
limm {
log ((r)
log
By convention, an object is referred to as "fractal" if D is different from topological dimension. For example, the fractal dimension of the Koch curve is about 1.26 when its topological dimension is Fractals explain some puzzling observations, such as why estimates of the length of the coast of Britain varied significantly from one study to another, because lengths in fractals are scale-sensitive. Mandelbrot estimated the fractal dimension of the coast of Britain to be 1.25 [43].
The definition above for the fractal dimension D applies to geometric shapes, but an analogous definition has been introduced for stochastic processes. Let (Zt)SER be stationary process with autocovariance Pr. Then, its fractal dimension D is determined according to the local behavior of Pr. the vicinity of n 0, by first normalizing (Zr)teR to have a zero-mean and unit variance, and modeling On using power law Pn. ~ n as - 0+,, for a E (0, 2]. Then, the fractal dimension E [1, 2] LERR defined by = 2- a/2 [26]. can shown that = 2 [26]. For language, this gives median fractal dimension of D 1.41 + 0.08.
2.5 Joseph effect Burstines
Finally, examine another related parameter that is commonly studied in self-similar processes. The motivation behind it comes from the fact that in processes with LRD, one often observes burstiness as shown in Figure 1; i.e. clusters over time in which the process fully resides on one side of the mean, before switching to the other. This is quite unlike random noise, for instance, where measurements evenly distributed on both sides of the mean. The effect is often referred to as the Joseph effect, named after the biblical story of the seven fat years and seven lean years [68, 46, 66].
A common way to quantify the Joseph effect for integral processes (Xt)ten is follows [66]. First, let ø, be the standard deviation of the 7-increments X++- X,- Then, fit a power law relation a- ~ T The exponent J here is called the Joseph exponent. In an idealized fractional Brownian motion, both and the self-similarity exponent coincide. Figure 5 provides the detailed empirical results. Overall, we find that J = 0.49 + 0.08.
Analysis
Comparative Analysis. Table compares fractal parameters across different domains, such as ArXiv, Github and Wikipedia. In general, most domains share similar self-similarity and Hurst exponents with few exceptions. The first notable exception DMMMathematiess, which has a Hurst parameter of about 0.5, indicating lack of LRD. Upon closer inspection, however, value of H 0.5

---

10²
OpenWeb
10'
Github
10'
FreeLww
10
Hie3c
307
10'
10'
9
10
100
102
16
10
10
10
10'
10'
10'
10a
10'
10'
10"
102
10?
PubMed
DMMeth
10a
10²
10a
10
10
10a
10
10a
10'
10
10
10'
107
10'
Figure 5: The standard deviation of the r-increments X(+7 - X2 is plotted against the scale т. We, again, observe another power law relation a ~ T², with Joseph exponent J = 0.49 + 0.08.
TS-Decoder
PaI. 62B
PaLM2 S
110M
340M
1B
SB
8B
540B
XXS
XS
M
L
Self-similarity exponent S
.88 t.0
-80 06
.60 st..00 00
.88 4.08
.80 07
624..8
.84
.59
00
27..00
.56 05
.59
.60
Hurst exponent
...2.08
.64 00
07
682.07
.682.07
.66
66...7
-672.00
.68 09
682..0
Joseph expoment
444.00
444 06
44
474.06
488
A74
484
46 2
.49
Table 2: A comparison of the estimated median fractal parameters by various LLMs over the entire Pile validation split. Estimates are generally robust to the choice of the LLM, but the tiny variations median И reflect improvements the model quality. See Section 3.
is not surprising for DM-Mathematies because its documents consist of independent mathematical questions as shown Figure 6. In Figure (right), we plot the partial autocorrelation function for each of the 8 domains against time lag (context length). Indeed, we see that DM-Mathematics shows markedly less dependence compared to the other domains. The second notable observation the relatively larger value of H 0.79 in GitHub, indicating more structure in code. This is in agreement with earlier findings by Kokol and Podgorelec [35] who estimated LRD computer languages to be greater than in natural language. In Table 2, compare the three fractal parameters S, H and J using different families of LLM and different model sizes. Overall, we observe that the parameters generally robust to the choice of the architecture.
Downstream Performance. By definition, fractal parameters are calculated on the sequence of negative log-proo probbabbbyy scores after normalizing them zero-mean and unit variance. Hence, they may offer an assessment of downstream performance that improves upon using a perplexity- -bssd metric like bits-per-byte (BPB) alone. To test this hypothesis, we evaluate the 12 models in Table on challenging downstream zero- and few-shot benchmarks focusing language understanding and reasoning. We include results for 0-shot (OS) and 3-shot (3S) evaluation for BIG-Bench Hard tasks [63, 64] reporting both direct and chain-of-thought (Co'T) prompting results following Chung et al. [13]. In addition we report 0-shot and 5-shot (5S) MMLU [30], and 8-shot (8S) GSM8K [14] with COT. Raw accuracy is reported for all tasks. BBH and MMLU scores averaged across all 21 tasks and 57 subjects, respectively. These benchmarks are quite diverse and include tasks such as logical deduction, arithmetic, translation error detection, disambiguation, history, computer science, law, sports, movies, etc). All prompt templates for our evaluation are taken from Chung et al. [13], Longpre et al. [42]. which refer the reader for more details. We prompt all models using a 2048 context length. See Table 8 of Appendix for the full results.
well as general knowledge (e.g.
The first (surprising) observation is that the median Hurst parameter is itself strongly correlated with the BPB scores with an absolute Pearson correlation coefficient of 0.83, even though the Hurst exponent is calculated after normalizing all token losses to zero-mean and unit variance! Informally,

---

Document I: What is the square reot 211269 to the nearest integer? 460. What ie the aquare reot of 645374 the nearest integer? 803...
Document Suppose 5+1 - 35, -2-r S+1 15 - -70. Is r multiple of 4? True. Suppose 2+1
Dees 15 divida (-2)/1 118/(-B)? False...
Figure 6: Two examples of documents from the DM-Mathematics subset of The Pile benchmark [23]. Each document comprises of multiple independent questions. The lack of LRD in this data is reflected in its Hurst parameter of H 0.50 + 0.01
Benchmark
BPB
H
H
2K
4K
8K
BBH Direct
0,785
0.84]
0.883
1.81
1.68
1.76
DS MMLU BBH+MMLU
0.653
0.831
0.825
25.73
26.04
25.81
0.685
0.849
13.39
13.49
13.42
BBH Direct
0.767
0.895
0.926
21.35
24.76
23.14
BBH CoT
0.881
0.892
0,979
16.87
12.21
7.14
3S MMLU
0.660
0.853
0.832
26.57
26.69
27.07
GSM8K
0.867
1.74
FS BBI--MMLU-GSMEK
0.717
0.890
0.891
15.58
15.46
14.65
Table 3: MIDDLE three columns show the adjusted R²: the proportion of variation in downstream performance (row) predictable by a linear function of the input (column). Median Hurst (H) and (especially) the combined metric H predict downstream performance better than BPB alone. S and J do not give any improvement (see Appendix C). RIGHT: the downstream performance for three decoder-only 1..1.1. models pretrained on 100B tokens with 2K, 4K, 8K context lengths.
this implies that second-order statistics on the sequence of token losses of particular model can predict its mean! Self-similarity exponent, by contrast, has absolute correlation of 0.23 with BPB.
Figure displays downstream performance against both the median Hurst exponent and the median BPB score, where median values are calculated on the domains in The Pile benchmark listed in Table In general, both the BPB score and the median Hurst are good predictors of downstream performance. However, we observe that improvements in BPB alone without impacting the median Hurst exponent do not directly translate into improvements downstream. This is verified quantitatively in Table (middle), which reports the adjusted R² values the proportion of variance in each downstream metric that can be predicted using BPB, H, or by combining them together into H = 1/BPB +- H, with BPB replaced with its reciprocal so that higher values are better. We observe that HE yields indeed a stronger predictor of downstream performance. Hence, while and BPB are correlated, combining them yields better predictor, so each of H and BPB conveys useful information not captured by the other metric. See Appendix for similar analysis using the exponents S and
Context Length at Training Time (Negative Result). Finally, we present a negative result, Self- similarity and LRD point extensive contexts in order to capture the practal-nature of language, which may elevate the model's capabilities regardless of the context length needed during inference. To test this hypothesis, pretrain three decoder-only T5.1.1 1.1 models with 1B parameters on SlimPajama-622 100B tokens using three context lengths: 2K, 4K and 8K, all observing the same number of tokens per batch. We use SlimPajama-6275 instead of C4 because most documents in C4 are short (Rs 94% of them 2K tokens in length). Refer to Appendix A for details. These models are, then, evaluated on the same downstream benchmarks listed in Figure 7. As shown in Table 3 (right) however, we do not observe any improvements in performance with context length in this particular setup.
intriguing possibility: the importance of training the model with
[62] for up to
4 Related Works and Directions for Future Research
The statistical attributes of human language have long piqued scholarly curiosity. One example Zipf's law, which Shannon leveraged to estimate the entropy English to be around 1 bit per letter [60], but his calculation did not consider second-order statistics. More recently, Eftekhari [18] proposed refinement to Zipf's law, suggesting its application to letters rather than words. Another related result is Heap's 's law, which states that the number of unique words is power law function

---

68H
05 MMLUU
35 BBH
nrs
Media DPB
DPB
Median DPB
0
0
0.4
0.4
0.62
0.67
0.67
0.72
0.62
0.67
0.72
Median Hurst
Median Hurst
Mediaan Hurst
Median Hurst
1.1
sts MMLU
1.1
Bs CSMMK Coll
1.1
os BUBI+KNMLL
1.1
FS B01H+MMMLUGGSHI
Mediaa BPB
Median BPB
Median
0.62
0,67
0.72
0.62
0.67
0.72
0.62
0.67
0.72
0.62
0.67
0.72
Median Hurst
Median Hurst
Median Hurst
Median Hurst
Figure 7: Downstream metric, indicated by bubble size where larger is better, is plotted vs. the median Hurst and the median BPB for all 12 language models.
of the document's 's length [29]. However, both Zipf's s and Heap's laws are invariant to the semantic ordering of text, so they do not capture important aspects, such as long-range dependence (LRD) [49].
In terms of self-similarity in language, the Benzzerath-Almann law stipulates self-similar behavior in the following sense: when the size of a language construct increases, the size of its constituents decreases, and this happens at all scales [49, 4]. In Ausloos [9], the authors model texts as a time series by replacing a word with its length. After that, they study the fractal behavior of language. However, as mentioned in [22], replacing a word with its length is invalid because it is not translation- independent (i.e. one could map every word to an arbitrary token, including tokens of equal length). In our work, we model language as series of bits calculated from conditional entropies, reflecting the intrinsic structure of the language itself, inspired by findings in linguistics such as [28, 22, 41]. The existence of self-similarity in language is attributed to its hierarchical nature [68, 3], such as duality of patterning [37].
In Najafi and Darooneh [49], the authors define a fractal dimension for each word. Informally, they examine the recurrence of a single, predetermined word binary series, similar the approach used in Altmann et al. [3]. However, this only applies to individual words and cannot model higher- level clauses. For instance, it does distinguish between "time" in the phrase "once upon a time" and "time" in "space and time." Kokol and Podgorelec [35] estimate LRD in natural language, and suggest that its LRD is close that of pure noise! They conjecture this was due to their use of ASCII encoding. In computer languages, they observe LRD and suggest it is because they are formal.
Besides the above concerns in prior studies that examined the self-similar structure in language, another concern is that they sometimes give extremely large values of the fractal dimension, sometimes exceeding 10 [4]! Such values difficult interpret because the fractal dimension D should fall in D E [1, 2] for time series, We do not observe such issues in our analysis. In our case, D = 1.41 0.08.
Limitations and Future Research. Our analysis is currently limited to the English language so it may not apply to other languages that differ significantly. For instance, some languages such as Pirahã (spoken in the Amazon) do not have a recursive structure like most languages do [20]. We also do not model the semantic lexical form of language. While well-founded and captures the intrinsic complexity of language, it does not account for the semantic nuances that contribute to meaning. Thirdly, self- self-similarity may explain why parameter sharing, such as in ALBERT [38], can be successful but exploiting self-similarity more directly in LLMs could lead to further optimizations. Exploring these aspects are promising directions for future research.
information-theoreti approach is
Concluding Remarks
In this work, we highlight intriguing insights into the underlying fractal structure of language and how it may be interconnected with the remarkable capabilities of LLMss. Our formalism quantifies properties of language that may have been suspected, but not previously formally shown. In particular, the need in LLMs to balance between short- and long-term contexts is reflected in the self-similar

---

structure of language, while long-range dependence is quantifiable using the Hurst parameter. For instance, the absence of LRD in DM-Mathematics is reflected in its Hurst parameter of H 0.5. Interestingly, the estimated median Hurst value of H = 0.70 + 0.09 in language reflects an intriguing balance between predictability and noise that is similar to many other phenomena, and combining both H with BPB together yields stronger predictor of downstream performance. We carry out an extensive comparative analysis across different domains and model architectures, revealing that fractal parameters are generally robust. We hope that future research can further probe into these fractal properties, unearthing deeper understandings the relation between intelligence and language.
Acknowledgement
The authors would like to thank Justin Gilmer and Olivier Bousquet for their feedback earlier drafts of this manuscript, and both Google Deepmind and Google Research teams at large for the insightful discussions and providing supportive research environment.
References
[1] Abry, P., Gonçalvés, P., and Flandrin, P. (1995). Wavelets, spectrum analysis and 1/f processes. Wavelets and statistics, pages 15-29.
[2] Alabdulmohsin, I. (2018). Summability calculus: A comprehensive theory of fractional finite sums. Springer. 6
[3] Altmann, E. G., Cristadoro, G., and Esposti, M. D. (2012). On the origin of long-range corre- lations in texts. Proceedings of the National Academy of Sciences, 109(29): 1582-11587. 2, 4,
[4] Andres, J. (2009). On de Saussure's principle of linearity and visualization of language structures. Glottotheory, 1-2)
[5] Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk. J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Petrov, S., Johnson, M., Antonoglou, I. Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., et al. (2023a). Gemini: A family of highly capable multimodal models. 2rXi::2312.110055
[cs. [cs.CL].
[6] Anil, R., Dai, A. M.. Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., Chu, E., Clark, J. H., Shafey, L. E., Huang, Y., Meicr-Hellstern, K., Mishra, G., Moreira, E., Omernick, M., Robinson, K., Ruder, S., Tay, Y., Xiao, K., Xu, Y., Zhang, Y., Abrego, G. H., Ahn, J., Austin, J., Barham, P., Botha, J., Bradbury, J., Brahma, S., Brookso K., Catasta, M., Cheng, Y., Cherry, C., Choquette- Choou, A., Chowdhery, A., Crepy, C., Dave, S., Dehghani, M., Dev, S., Devlin, J., Díaz, M., Du, N., Dyer, E., Feinberg, V., Feng, F., Fienber, V., Freitag, M., Garcia, X., Gehrmann, S., Gonzalez, I.., Gur-Ari, G., Hand, S., Hashemi, H., Hou, L., Howland, J., Hu, A., Hui, J., Hurwitz, J., Isard, M., Itycheriah, A., Jagielski, M., Jia, W., Kenealy, K., Krikun, M., Kudugunta, S., Lan, C., Lee, K., Lee, B., Li. E.. Li. M., Li, W., Li, Y., Li. J., Lim, H., Lin, H., Liu, Z., Liu, F., Maggioni, M., Mahendru, A., Maynez, J.. Misra, V., Moussalem, M., Nado, Z., Nham, J., Ni, E., Nystrom, A., Parrish, A., Pellat, M., Polacek, M., Polozov, A., Pope. R., Qiao, S., Reif, E., Richter, B., Riley, P., Ros, A. C., Roy, A., Saeta, B., Samuel, R., Shelby, R., Slone, A., Smilkov, D., So. D. R., Sohn, D., Tokumine, S., Valter, D., Vasudevan, V., Vodrahalli, K., Wang, X., Wang, P., Wang, Z., Wang. T., Wieting, J., Wu, Y., Xu, K., Xu, Y., Xue, L., Yin, P., Yu, J., Zhang, Q., Zheng, S., Zheng, C., Zhou, W., Zhou, D., Petrov, S., and Wu, Y. (2023b).
PaI.M 2 technical report. arXiv: 10403v3 CL.) 4, 17
[7] Apostol, T. M. (1999). An elementary view of Euler's summation formula. The American Mathematical Monthly, 106(5):409418.
[8] Aref, S. (1998). Hurst phenomenon and fractal dimensions long-term yield data. In Conference on Applied Statistics in Agriculture. 2,5
[9] Ausloos, M. (2012). Generalized Hurst exponent and multifractal function of original and translated texts mapped into frequency and length time series. Physical Review E, 86(3):031 108. 1, 4,
10

---

[10] Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman- Milne, S., and Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs. 14
[11] Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. ., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T.,, and Zhang, (2023). Sparks of artificial general intelligence: Early experiments with GPT-4.
[12] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung,
H. W., Sutton, C., Gehrmann, S., et al. (2022). PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:220..03311. 4, 14, 17
[13] Chung, H. W. Hou, Le and, L. S., Zoph, B., Tay, Y., Fedus, W., and et al. (2022). Scaling mnstruction-finetune language models. arXiv:2210. 11466v5 [cs.LG].
[14] Cobbe, K., Kosaraju, V., Bavarian, M.. Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv:21 10. 1612 [cs.LG).
[15] Cover, T. M. (1999). Elements of information theory. John Wiley & Sons.
[16] Crovella, M. E. and Bestavros, A. (1995). Explaining world wide web traffic self-similarity. Technical report, Boston University Computer Science Department. 2,5
[17] Efron, B. and Tibshirani, R. J. (1994). An introduction the bootstrap. CRC press. 5, 15
[18) Eftekhari, A. (2006). Fractal geometry of texts: An initial application the works of Shake- speare. Journal of Quantitative Linguistics, 13(2-3):177-193.
[19] Embrechts, P. and Maejima, M. (2000). An introduction to the theory of self-similar stochastic processes. International journal of modern physics B, 14(12n13):1399- 1420. 1
[20] Everett, D. (2005). Cultural constraints on grammar and cognition in pirahä: Another look at the design features of human language. Current anthropology, 46(4):621-64) 9
[21] Feller, W. (1951). The Asymptotic Distribution of the Range of Sums of Independent Random Variables. The Annals of Mathematical Statistics, 22(3):427 432. 2
[22] Futrell, R. and Hahn, M. (2022). Information theory as bridge between language function and language form. Frontiers in Communicationn 7:657725. 3, 4,
[23] Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang. He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. (2020). The Pile: An 800GB dataset of diverse text for language modeling.
[cs.CL]. 5,
[24] Geweke, and Norter-Hudak, S. (1983). The estimation and application of long memory time series models. Journal of time series analysis, 4(4):221
[25] Gibson, E., Futrell, R., Piantadosi, S. P., Dautriche, I., Mahowald, K., Bergen, L., and Levy, R. (2019). How efficiency shapes human language. Trends in cognitive sciences, 23(5):389 3
[26] Gneiting, T. and Schlather, M. (2004). Stochastic models that separate fractal dimension and the Hurst effeet.. SIAM Review, 46(2):269 5,
[27] Goldberger, A. L., Amaral, L. A., Hausdorff, M., Ivanov, P. C.,, Peng, C.-K., and Stanley, H. E. (2002). Fractal dynamics in physiology: alterations with disease and aging. Proceedings of the national academy of sciences, (4:2ppp_)):2466-24722
[28] Hale, J. (2001), A probabilistic earley parser as a psycholinguistic model. In Second meeting of the north american chapter of the association for computational linguistics. 3,9
[29] Heaps, H. S. (1978). Information retrieval, computational and theoretical aspects. Academic Press.
[30] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, (2020). Measuring massive multitask language understanding, arXiv preprint 20X9v:2009.02: 7
11

---

[31] Hurst, H. E. (1951). Long-term storage capacity of reservoirs. Transactions of the American society of civil engineers, 116(1):770-799, 2.5
[32] Jouppi, N. P., Yoon, D. H., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., and Patterson, D. (2020). A domain-specinf supercomputer for training deep neural networks. Communications of the ACM, 63(7):67-78. 14
[33] Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield- Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.0522)
[34] Kidd, C. and Hayden, B. Y. (2015). The psychology and neuroscience of curiosity. Neuron, 88(3):449-460)
[35] Kokol, P. and Podgorelec, (2000). Complexity and human writings. Complexity, 7:1-6. 4, 7, 9
[36] Kolmogorov, A. N. (1940). Wienersche spiralen und einige andere interessante kurven in hilbertscen raum, (doklady). Acad. Sci. URSS (NS), 26:1 16:115-118 118.
[37] Ladd, D. R. (2012). What is duality patterning, anyway? Language and Cognition, 4(4):261- 273. 9
[38] Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). AL- BERT: A lite BERT for self-supervised learning of language representations. arXiv preprint arXiv: 1oXiv:1909.//900 11942.
[39] Leland, W. E., Taqqu, M. S., Willinger, W., and Wilson, D. V. (1994). On the self-similar nature of Ethernet traffic. IEEE/ACM Transactions on networking. 2(1):1-15.
[40] Leland, W. E. and Wilson, D. V. (1991). High time-resolution measurement and analysis of LAN traffie: Implications for LAN interconnection. In IEEE INFCOM.
[41] Levy, R. (2008). Expectation- baaeeedttoo--baeddddeee syntactic comprehension. Cognition, 106(3):1126-117) 3,
9
[42] Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., and Roberts, A. (2023). The flan collection: designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning. ICML'23. JMLR.org. 7
[43] Mandelbrot, B. (1967). How long the coast of Britain? Statistical self-similarity and fractional dimension. science, 156(3775):636-6 638. 6
[44] Mandelbrot, B. (2002). Gaussian self-affinity and fractals: globality, the earth, 1/f noise, and R/S. Springer Science and Business Media. 3
[45] Mandelbrot, B. (1982). The fractal geometry of nature. WH freeman New York.
[46] Mandelbrot, B. B. and Wallis, J. R. (1968). Noah, Joseph, and operational hydrology. Water resources research, 4(5):909-918.
[47] Mollica, F., Bacon, G., Zaslavsky, N., Xu, Y., Regier, T., and Kemp. C. (2021). The forms and meanings of grammatical markers support efficient communication. Proceedings of the National Academy of Sciences, 18849):c2025993111
[48] Montemurro, M. A. and Pury, P. A. (2002). Long-range fractal correlations in literary corpora. Fractals, 10(04):451-40 4
[49] Najafi, E. and Darooneh, A. H. (2015). The fractal patterns of words in a text: method for automatic keyword extraction. PloS one, 10(6):c010017. 2, 4,
[50] OpenAl (2023). GPT-4 technical report. axXiv:2303.08774w [cs. Css.
12

---

[51] Paxson, V. and Floyd. S. (1995). Wide area traffic: the failure of Poisson modeling, JEEE/ACM Transactions on networking, 3(3):226-244 24..
[52] Peng, C.-K., Buldyrev, S. V., Goldberger, A. I.., Havlin, S., Sciortino, F., Simons, M., and Stanley, H. E. (1992). Long-range correlations in nucleotide sequences. Nature, 356(6365):168- 170.
[53] Perfors, A.. Tenenbaum, J., Gibson, E., and Regier, T. (2010). How recursive is language? a bayesian exploration. Recursion and human language, pages 159-175. 2
[54] Pilgrim, I. and Taylor, R. P. (2018). Fractal analysis of time-series data sets: Methods and challenges. In Ouadfeul, S.-A., editor, Fractal Analysis, chapter IntechOpen, Rijeka. 2, 6
[55] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910. 1068344 [cs.LG]. 4, 14
[56] Roberts, A., Chung, H. W., Levskaya, A., Mishra, G., Bradbury, J., Andor, D., Narang, S., Lester, B., Gaffney, C., Mohiuddin, A., Hawthorne, C., Lewkowycz, A., Salcianu, A., van Zee, M., Austin, J., Goodman, S., Soares, L. B., Hu, H., Tsvyashchenko, S., Chowdhery, A., Bastings, J., Bulian, J., Garcia, X., Ni, J., Chen, A., Kenealy, K., Clark, J. H., Lee, S., Garrette, D., Lee-Thorp, J., Raffel, C., Shazeer, N., Ritter, M., Bosma, M., Passos, A., Maitin-Shepard,, J., Fiedel, N., Omernick, M., Sacta, B., Sepassi, R., Spiridonov, A., Newlan, J., and Gesmundo, A. (2022). Scaling up models and data with t5x and seqio. 14
[57] Roche, S., Bicout, D., Maciá, E.,, and Kats, H. (2003). Long range correlations in DNA: scaling properties and charge transfer efficiency. Physical review letters, 91(22):228101.
[58] Samorodnitsky, G. (2006). Long memory and self-similar processes. In Annales de la Faculté des sciences de Toulouse: Mathématiques, volume 15, pages 107-123. 5
[59] Schenkel, A., Zhang, J., and Zhang. Y.-C. (1993). Long range correlation human writings. Fractals, 1(01):47-57. 4
[60] Shannon, C. E. (1951). Prediction and entropy of printed English. Bell system technical journal, 30(1):50-64. 8
[61] Shazeer, N. and Stern, M. (2018). Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 1596-4604 PMLR. 14
[62] Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestmess, J., and Dey, N. (2023). SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. 8, 14
[63] Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., al. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0460 7
[64] Suzgun, M., Scales, N., Scharli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., and Wei, J. (2022). Challenging BIG-Bench tasks and whether [cs.CL].
chain-of-thought can solve them. arXiv:2210.092611
[65] Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre. L., Rivière, M.. Kale, M. S., Love, J., et al. (2024), Gemma: Open models based on gemini research and technology. arXiv preprint arXivv2403.0888
17, 18
[66] Watkins, N. (2019). Mandelbrot's 's stochastic time series models. Earth and Space Science, 6(11):2044-200 2056. 2, 5,
[67] Wikimedia (2023). Downloads. 18
[68] Willinger, W., Taqqu, M. S., Leland, W. E., and Wilson, D. V. (1995). Self-similarity in high- speed packet traffic: analysis and modeling of Ethernet traffic measurements. Statistical science, pages 67-85. 6,
[69] Willinger, W., Taqqu, M. S., Sherman, R., and Wilson, D. V. (1997). Self-similarity through high-variabiliity statistical analysis of Ethernet LAN traffic at the source level. JEFF/ACM Transactions on networking, 5(1):71-86. 2
13

---

A Experiment Details
All of our experiments are conducted in JAX/Flax [10] using the open source T5X framework [56].
T3 baselines in Table 2 and 3 are pretrained from scratch using the open source T5.1. 1. I decoder-only architecture from the T5X library.". We pretrain using causal language modeling objective over the C4 corpus with the default TS vocabulary as per Raffel et al. [55]. Training is done for 500k steps with sequence length of 1024 and batch size of 512, resulting in total of 262B tokens pretraining. We optimize our model with the Adafactor [61] optimizer with an inverse square root learning rate schedule, 1K warmup steps, and initial learning rate of 1e-2. Models trained using 256 TPUv5e chips [32].
during
T5 context length ablation experiments in Table 3 are trained with the same pretraining objective but over the SlimPajama-6 B-27BB corpus [62] and using a modified version of the T5 vocabulary that preserves whitespace and introduces byte-fallback for out of vocabulary tokens. This is similar to Chowdhery et al. [12], but preserving the original T5 vocabulary. Models with sequence lengths 2048, 4096, 8192 trained with batch sizes of 512, 256, and 128 respectively preserve the number of tokens seen per batch and overall training steps. We train all models for 100k steps, using the same learning rate schedule described above. Hence, all models observe 100B tokens.
'nttps://github. http://gggghhuhoo.o.eeeeo http:///ww.tptooo//////////////a models
14

---

B Full Results
In this section, provide the full list of parameters calculated for each combination of LLM and domain. We use bootstrapping [17] to estimate the error margin.
Model
OpenWebTex12 Github
FreeLaw
Pile-CC
Wikipedia
PubMed
Mathematics
ArXiv
15-Decoder1110
2,89
1.82
2.45
2,88
2.80
2.36
2.28
2.70
75-Decader.3al
15-Decode-1B
2.38
1.37
1.91
2,41
2.29
1.88
2,00
2.19
15-Devoder5BB
2.19
1.22
1.73
2,25
1.73
1.11
2.01
PaLMI-8B
2.26
0.79
1.66
2,36
2.08
1.89
1.40
2.08
PaLMI-62B
2.02
PaLM1 540B
1.88
0.54
1.33
2,01
1.58
1.57
1.25
1.68
PaLM2-XXS
2.37
0.87
1.77
2,46
2.17
1,96
1.38
1.96
PaLM2-XX XS
2.12
0.73
1.53
2,22
1.92
1.72
1.27
1.72
Pal M2.
1.95
0.60
1.37
2,06
1.71
1.57
1.19
1.55
PaLM2-M
1.88
0,56
1.31
1.99
1.59
1.51
1.12
1.48
Table Log-perplexity (NLL) scores evaluated on the first 2048 tokens, after trimming the first 100 tokens, of documents belonging to each of the shown domains. Only documents with a minimum length of 4K tokens are used.
OpenWehText2
FreeLaw
Pile-CC
Wikipedia
PuhMed
Mathematics
ArXiv
Decoder: 110M
0.58 1
0.67 1
0.51
0.54 1
0.59 1
0.59 i
0.51 1
0.58
0.04
0.03
0.02
0.07
0.04
0.03
0.04
0.05
15-Decodes-340M
0.52 +
0.59 +
0.63 +
0.58 +
0.61 +
0.61 +
0.48 +
0.61 =
0.03
T5. Decoder. 1B
0.54 1
0.66 1
0.61 +
0.57 1
0.59 1:
0.60 1
0.50 1
0.63 1
0.01
0.06
0.02
T5 Decoder 5B
0.51
0.70 +
0.60
0.58 +
0.58 1:
0.57
0.45 1
0.67
0.04
0.04
0.04
0.02
0.03
0.02
0.02
0.05
PaLM1-88 8B
0.56 +
0.67 +
0.63 +
0.58 +
0.55 +
0.62 +
0.50 1
0.68 +
0.03
PaLM1-62B
0.49
0.65 +
0.63
0.57 =
0.63
0.61
0,48
0.68 =
0.03
0.09
0.09
0003
0.05
0,04
0.05
0.03
PaLMI-540B
0.51 +
0.68 +
0.64
0.58 +
0,67 +
0.64
0,48 +
0.65
0.04
0.09
0.05
0014
0.03
0.08
0.03
0.04
PaL-2-XXS
0.53 +
0.02
0.61 +
0.05
0.58 +
0.01
0.00 +
F000
0.57 +
0.05
0.61 +
0,03
0.52
0.02
2000
PaLM2-XS
0.54
0.57 +
0.58
9.50 =
0,60 +
0.37
0.45 +
0.73 =
0.04
0.06
0.03
0000
0.04
0.06
0.02
0.06
Pal M2.S
0.55
0.55 +
0.59
0.541
0,65
0.58
0.49
0.61
0.02
0.15
0.02
0.08
0.04
0.05
0.04
0.03
PaLM2-M
0.70
0.56
0.46
0.02
0.06
0.04
0.05
0.03
0.04
0.04
0.05
PaLM2-L
0.53
0.60 +
0.61
0.56 +
0.62
0.60
0.42
=
0.05
0.05
0.05
0.03
0.02
0.07
0.03
0.03
Table 5: Self-similarity exponent evaluated on the first 2048 tokens, after trimming the first 100
tokens, of documents belonging to each of the shown domains. Only documents with a minimum length of 4K tokens used.
15

---

Model
OpenWebTea22 Github
FreeLaw
Pil-CC
Wikapedia
PuhMed
Mathematics
ArXiv
15-Decode- 110M
0.63 +
0.82 +
0.62 +
0.67 =
0,62 +
0.00 t00
0.65
0.51 +
0.68 =
0.00
0.01
0.01
0,01
0.00
0.01
0.01
15. 340M
0.63
0.82 +
0.62
0.67 F
6,80 1
0.64
0.54 1
0.67
0.01
0.01
0.00
0.00
1000 T
0.01
0.00
0.01
1
0.67
0.64 +
0.54 T
0.01
0.01
0.01
00000
0.01
0.00
0.00
0.00
T5-Deodde-SB
0.63 =
0.01
0.82 =
0.00
0.62 +
0.01
0,07
0000
0.62
1.00
0.64
0.01
0.51
0.00
0.00
PaLMI-8B
0.65
0.01
1.00
0.60
0.00
000
1000
0,60 =
0000
0.65
0.01
0,51 #
0000
0.00
1.00
PaLMI-540B
62L
0.67 1
0.00
0.01
0,80 1
0.79 +
0000
00.00
0.67 1
0.00
0.01
0.69 F
6890 F
1000
000
000
1.1
0.00
100
0.65
0.00
000
0.30 1
0.57 1
0000
0000
01
0000000000000
0,30
0.70 =:
0.70
100
0.00
PaLM2-XXS
0.01
1000
110
100
+F
0,00
F 390
000
0,68 1
0.00
0.65
1000
0558 F
00000
1.1
10.00
1011
PaLM2-XS
0.01
088 =
1000
0.60
0.01
F000 =
1000
0,67
0.00
0.65
0.00
0.56 +
1000
0.71
1,00
PaLM2-S
PaLM2-M
0.67
0.67 +
0.01
0880 #
0.80 +
0.67 +
0.66
0.01
0.70 =
0.69 #
0.000
0.70
0,68
0.63 +
0.63 =
0.01
0.52 F
0.54
00000
0.70
0,00
077
0.00
=
PaI.M-I.
0.68 +
0.79 +
0.68 +
0.70
0.74 +
0.65 +
0.50 +
0.72
0.01
0.01
0.00
0.00
0.00
0.01
0.01
Table Hurst exponent evaluated on first 2048 tokens, after trimming the first 100 tokens, of documents belonging to each of the shown domains. Only documents with minimum length of 4K tokens are used.
Model
OpenWehTexz2 Githah
Freel aw
Pile-CC
Wikipedia
PuhMed
Mathematics
ArXiv
15 Decoder 110M
0.44
0.53 +
0.42
0.49 +
0.45 +
0.43
0.33 +:
0.45 :
0.01
0.00
0.00
0.01
0.00
0.00
0.00
0.00
15-Decoder-34000
0.02
0.00
0.00
0.00
0.01
0.00
0.00
0.00
75-Decoder-IP
0.43 +
+ 0,00
0.43
0.49
+
0.42
0.33 +
0.01
0.01
0,01
0.01
0.00
0.00
0,01
15-Decoder55B
0.43
0.53 +
0.44
0.49 +
0.45 +
0.42
0.34 +
0.45
0.01
0.00
0.00
0.01
0.00
0.00
0.00
0.00
PaLMI-8B
0.45
0.51 +
0.49 +
0.44
0.34 +
0.00
0.00
0.00
1000
0.01
0.01
0.00
0.01
PaLM1-62B
0.45
0.00
0.50 +
0.01
0.47
0.00
0,490 =
1000
0.19
0.00
0.44
0.00
0.33 +
00000 0000
0.48 =
1,00
Ma-5M-
0.46 +
0.01
0.49 +
0.47
0.00
0.50 +
1000
0.50
0.00
0.44
0.00
0.33 F
1.00
0..4 3:48 =
0.00
M2-XXXX
0.50 +
0.45
0.50 F
+
0.45 +
0,34 1 f
0,.0
0.01
0.00
0.00
1000
0.00
0.00
0.000
0.00
PaLM2-XS
0.45
0.50 =
0.16
0,470
0.18
0.44
0.33
0.19
0.01
0.01
0.01
0.00
0.00
0.01
0.00
PaLM2-S
0.15 +
0.49 +
0.47
0.50 =
0.50
0.44
0.31 +
0.19
0.00
PaLM2.M
0.45 1:
0.49 1:
0.48 1
0.50 1
0.50
0.44
0.29 1.
0.49 ::
0.01
0.01
0.00
PaLM2-L
0.46
0.19 =
0.49
0.50 =
0.52
0.44
0.28 +
0.49 =
0.01
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 7: Joseph exponent J evaluated on the first 2048 tokens, after trimming the first 100 tokens, of documents belonging to each of the shown domains, Only documents with minimum length of 4K tokens are used.
C Predicting Downstream Performance
Table presents detailed downstream performance results, along with corresponding upstream metrics.
In Table 9, we repeat the same analysis Section 3 using the adjusted R² coefficient, but with the self-similarity S and Joseph exponents J. Unlike in the median Hurst exponent, do not observe
16

---

any improvement when combining perplexity scores with the self self-similarity exponent S or the Joseph exponent J.
Model
BPB
08 Direct
0S CoT
DS MMLU
35
35
5S MMLU
RS GSM8K CoT
OS +MMLU
FS
BBH +MMLU +GSM8K
Direct
CoT
T5. Decoder. 110M
1.11
0.83
0.11
25.65
21.36
5.69
25.62
0.91
13.06
13.35
75-Decader-3400
15-Decoder-1B
0.92
1.29
0.14
25.99
24.26
13.19
24.82
1.14
13.35
14,90
15-Decoder5B8
0.85
0.48
24.41
24.76
18.05
25.63
2.20
12.86
16.41
PaLMI-8B
0.78
6.46
1.21
23.53
32.18
27.60
24.56
5.16
13.68
19.87
PaLMI-62B
41.32
PaLM1 -40BB
0.66
23.26
4.72
67.78
52.44
56.02
70.50
56.79
40.89
60.51
PaLM2-XXS
0.81
8.99
0.13
25.26
30,71
26.08
24.72
2.96
14.91
18.69
PaLM2-XS XS
0.73
16.68
0.95
49.69
38.28
37.64
47.42
22.14
29.25
35.84
PaLM2-S
4.24
50.88
68.12
56.16
PaLM2-M
0.65
21.32
5.70
69.62
52.49
56.04
69.33
59.21
41.57
60.94
78.64
Table 8: Full downstream few-shot evaluation results compared upstream BPB. Here, BPB is computed over The Pile validation split using the first 2048 tokens of every document, All evaluation results are reported as raw (un-normalized) accuracy.
Please note that our results are not directly comparable to all previous published results for the same models; please cite the original results from [12, 6J. Here, only aim for a fair comparison between models: only pretrained models without instruction tuning are used, we do not optimize any prompts for each model, and we evaluate all models using only 2K sequence length.
BPB
S
J
BPB+S
BPB+J
OS BBH Direct MMLU BBH+MMLU
0.785
-0.060
0,673
0.761
0.794
0.653
-0.067
0.426
0.614
0.614
0.685
-0.065
0.472
0.650
0.651
-0.030
0.744
3S BBH CoT
0.881
-0.026
0.678
0,870
0.879
MMLU
0.660
-0.044
0.421
0.624
0.622
GSMSK COT
0.654
-0.037
0.427
0.619
0.616
FS BBH MMLU+GSMMK
0.717
-0.036
0.489
0.687
0.686
Table 9: Adjusted R², which measures the proportion of variation in downstream performance (row) that is predictable from the given input(s) (column) using trained linear regressor. Unlike in the median Hurst exponent, we do not observe any improvement when combining BPB scores with the self-similarity exponent or the Joseph exponent J.
D Gemma-2B Results
In this section, we present empirical results using the publicly released Gemma-2B checkpoint [65]. Results are shown in Figures 8 and
Gemma-22 2B is much smaller than all of the models we have used previously. Yet, we generally observe similar conclusions. First, we have self-similar structure with near perfect linear fits in log-log plots. Second, we also observe power laws using the rescaled-range analysis, with Hurst exponent about 0.7 in most domains except DM Mathematics (smallest Hurst exponent of about 0.58) and GitHub (largest Hurst exponent of about 0.82), in general agreement to the rest of the models. Both the similaiiyy and Hurst parameters are provided in Table 10.
E Comparison to n-Gram Models
Here, we report some early investigations that examine how fractal parameters change as one includes longer contexts during inference. The setup used this experiment slightly different from what throughout the rest of the paper. Specifically, we take the Wikipedia dataset
17

---

ArXiv
DM Mathematics
PubMed Central
Wikipedia (en)
10
10-2
10-2
10
-10->
1.10
10
10-4
10-1
10-4
10a
10a
10:
10"
10:
10'
10a
10"
Scale
Scale
Scale
Scale
PIle-CC
FreeLaw
10-1
Github
10
Ope2Web
1.10">
5.10"1
-010
10
10
10
10
10a
10a
10
10a
10:
10
10
10a
Scale
Scale
Scale
Scale
Figure 8: Here, follow similar setup to Figure 2, but using the publicly released Gemma-2B checkpoint [65], where the y-axis is the peak probability. We continue to observe linear fits in log-log scale over least two orders of magnitude, thus confirming a self-similar structure.
10a
ArKiv
10a
Mathematics
Pub Med Central
10a
Wikipedia
2 10
2 10
10
10a
10a
10a
10"
109
102
10a
Scale
Scale
Pile-CC
FreeLaw
Github
OpenWebText2
10a
10a
10a
10a
3 10
10a
1oa
10a
10a
10a
10a
102
10a
10t
10a
Scale
Scale
Scale
Scale
Figure 9: Here, we follow similar setup to Figure 3, but using the publicly released Gemma-2B checkpoint [65], where the y-axis the escaled-range (R/S). We continue to observe linear fits in log-log scale over at least two orders of magnitude.
(xikippddia/22230011 en) dataset [67]. We split each document of length >4K words along word boundaries and constrain the context length during inference in Pal. n E [1,16, 32, 64, 128, 256, 512, 1024, 2048). Hence, the language model now resembles an n-gram model. We calculate probability scores and calculate fractal parameters accordingly. Figure 10 shows how both the self-similarity exponent S and the Hurst parameter H change as a function of n.
words, for
We observe that H increases monotonically with context length expected, since it implies more predictability. However, even in a 1-gram model, H can be larger than 1/2 because the words themselves not independent of each other, the sequence of probability scores still contain dependence over time 1-gram model.
18

---

OpenWebTex12
Gitbub
FreeLaw
Pil--CC
Wikipedia
PubMed
Mathematics
ArXiv
0.002
0.63 0.011
0.00
0.58 0.00
0.57 0.02
0.57 0.00
0.00
0.61 0.002
0.63
0.01
0.83
0.01
0.67
0.01
0.68
0.01
0.65
0.01
0.64
0.01
0.58
0.01
0.69 0.01
Table 10: Self-similarity exponent (S) and the Hurst parameter (H) when using Gemma-2B. These values compare well to the obtained using the other models.
0.65
0.80
0.75
0.60
5
I 0.70
0.55
0.50
0.60
10"
10
10?
10
10
101
10?
10
Inference Context (in words)
Inference Context (in words)
Figure 10: S and H plotted for different constructions of bits, we vary the prefix length during inference.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Section 4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
19

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Guidelines: We follow well eelebbstall definitions throughout our analysis.
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
20

---

reproducible, they While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
We only release a portion of the code that can be used to calculate fractal parameters.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
(elllic/guiee//ooe
wwwpammssssoooolioioooooooooeooe Policyy))y) icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips.co. wwtpccccccppppececccccccccceelle wwwpc//ggge////eee
gtwdee/oooeeuuumiiiiii/c/ CoodeSubiiieeeeeeeee cotbbmisssinn iesionPooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
See Section 2.1 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
21

---

The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We use the bootstrap method to estimate error margins and report those in the
paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http//eurip...//
Answer: [Ye]]
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
22

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work studies the fractal structure of language. We do not foresee any potential negative societal impact this work.
Guidelines:
The answer NA means that there societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deg.oyeelo of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
23

---

Answer:
[Yes]
Justification: We cite the original creators/owners of all datasets and model checkpoints that we in our analysis.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

---

Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly berween institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "zB5Qk9XY9U", "reviewer": "Reviewer_UXZb", "summary": "In this paper, the authors try to reveal the existence of fractal structures to handle language in language modeling using recent language models based on the next token prediction. For that purpose, the authors rely on several aspects of fractal structures, which are self-similarity, long-range dependence, and information-theoretic complexity. The authors describe these aspects using metrics for typical phenomena under the assumption of fractal structure, which are self-similarity exponent, Hurst parameter, fragmental dimension, and Joseph effect. The experimental results show that the characteristics of modeling texts of ArXiv, Github, and Wikipedia data in The Pile validation split, whose tokens are longer than 4K, are along with their assumption. In the analysis of the downstream tasks, BBH, MMLU, and GSM8K also show that the task-solving performances are predictable from the language modeling performance based on their assumption, except for considering sequence lengths in training.", "questions": "- In the analysis, the authors seem to only focus on the context length in training even though the inference of long sequential tokens is closer to the aspect of long distant dependencies between tokens in language. What is the main reason for this decision?\n\n[Comment] In presentation style, instead of showing the method name, showing the motivation or target phenomenon as the section or paragraph title, like \"Joseph effect\" -> \"Burstiness,\" may support the reader's understanding.", "limitations": "The limitation does not include the gap between language modeling and solving downstream tasks.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Providing an assumption to unify characteristics of interpreting language and language modeling under the variously used approach, next token prediction may help solving various kinds of tasks related to natural languages.\n- The analysis is not only restricted to language modeling and the authors investigate the performance correlation between language modeling and downstream tasks based on their assumption.", "weaknesses": "- The target language is limited to English. Thus, the validity of expanding the insights in the paper to other languages is uncertain.\n- The assumed baseline in this analysis is a random sequence. The authors can use finite-state automatons (FSAs) or Context-free grammars (CFGs) to make sequences that are random but close to languages. Moreover, to focus on the success of recent pre-trained language models, you should have prepared N-gram language models as baselines.\n- How strongly the observation results fit the assumed distribution is not calculated mathematically, like model selection by Information Criteria such as Akaike's Information Criteria (AIC) and Bayesian information criteria (BIC).\n- There is a gap between the ability to mimic the characteristics of language and showing the required knowledge for questions. Thus, discussing the correlation between the performance of language modeling and downstream tasks is limited from this viewpoint. Hence, targeting generation tasks like summarization, story generation, and machine translation are more suitable for deepening the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the careful and insightful feedback. We hope that our rebuttal below addresses all of the reviewer’s questions, are happy to provide more details, and look forward to the reviewer’s response to our rebuttal.\n\n**The baseline**\n\nThe baseline we use in Figure 1 is used for illustration purposes only, mainly to show how a self-similar process looks like. We do not use random sequences in our analysis. While we appreciate the suggestion of using n-gram models, our work focuses on the self-similar structure of natural language. Note in particular that the output of an n-gram model, by definition, does *not* have long-range dependence (LRD). So, conducting the same analysis on the output of n-gram models would not yield meaningful insights into the fractal structure of language, as n-gram models cannot generate long-range dependencies.\n\n**Empirical fit**\n\nThank you for raising this point. We do demonstrate how well the power law fits the actual distribution in Figures 2 and 3. We observe a perfect linear fit in a log-log plot over at least two orders of magnitude. In addition, since the power law is of the form $\\beta n^{-c}$, there are only two parameters to estimate ($\\beta$ and $c$) so AIC is also quite low, given the near perfect fit and the small number of parameters.\n\n**Evaluation**\n\nWe use three popular and quite diverse benchmarks (BBH, MMLU, GSM8K) and we also try various prompting strategies (e.g. direct and chain-of-thought with 0, 3, 5, 8 shots). These benchmarks are quite diverse and include tasks such as logical deduction, multi-step arithmetic, disambiguation, as well as general knowledge (e.g. history, computer science, law, sports, movies, and dates, etc). In line with your suggestions, they also include translation-related tasks, such as Translation Error Detection in BBH. So, they cover a broad spectrum of tasks. We hope this addresses your concern about the evaluation datasets, and we will clarify this in the revised version of the paper.\n\n**Context length**\n\nThe main reason for including the discussion about the context length at training time in our analysis is because our findings suggest one intriguing possibility: that language models might benefit from being trained on long contexts even if short contexts are used at inference time. However, we did not find any evidence yet to support this. We chose to include this in the analysis section because we believe that mentioning negative results would still be very valuable to our community.\n\n**Presentation** \n\nThank you for the great suggestion about the presentation style. We will definitely do that in the revised version of the paper. \n\n\n**Follow-up**\n\nWe are grateful again for your detailed and constructive feedback. If we have satisfactorily answered your questions, we hope you would consider revising your scores. Otherwise, please let us know if there are any other questions or concerns, so we can respond to them during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ckYaCCuI6B", "reviewer": "Reviewer_iLZg", "summary": "The paper draws connections between fractal patterns and language by evaluating properties such as self-similarity and long-range-dependency. Using a range of LLMs, they estimate the Holder exponent, Hurst parameter and fractal dimension for language from different domains, including web text, code, and math problems. They show that these parameters may have connections with LLM learning ability, demonstrating that using the median Hurst parameter can improve prediction of downstream model performance over only using bits-per-byte.", "questions": "I don't follow the potential connection between self-similarity and the success of parameter sharing? (mentioned in Limitations section)\n\nTypos: \n- Line 131: such bits-per-byte (BPB)\n- Line 132: BPB is a widely used as a\n- Line 272: such as One example", "limitations": "Yes, the authors discuss that their analysis is limited to English data and fails to capture the semantic meaning of language. However, as mentioned in Questions section, the possible connection between self-similarity and parameter-sharing is not obvious to me and could be further explained.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors estimate fractal parameters across a range of datasets and LLMs, showing that they are fairly robust to choice of LLM and domain. For domains with significant deviation, such as DM-Mathematics, the authors are able to attribute this to the dataset's lack of long-range dependency. \n- The authors show that the Hurst parameter captures useful information for predicting downstream performance that is not captured by BPB alone.", "weaknesses": "- Even though fractal parameters were found to be consistent across LLMs, I think their stated contribution (this 'establish[es] that language is self-similar and long-range-dependent') is far too strong a claim. As the authors mention, their method ignores many important facets of language, such as semantic nuance, and is reliant on the current state of LLMs' ability to model language. \n- I am unclear how future work can build upon these insights about language, especially that 'exploiting self-similarity more directly' could lead to further LLM optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the careful and insightful feedback. We especially appreciate the positive assessment of the thoroughness, originality, and importance of our work. We hope that our rebuttal below addresses all of the reviewer’s questions, are happy to provide more details, and look forward to the reviewer’s response to our rebuttal.\n\n**Information-theoretic complexity**\n\nThank you for raising this point. Please see our general comment above on why we believe an information-theoretic complexity is meaningful for our analysis. We will clarify this more in the revised version of the paper.\n\n**Future work and parameter sharing**\n\nWe are currently exploring several directions for future work. For example, do texts generated by LLMs have a self-similar structure? What is the impact of RLHF on the self-similar structure of generated texts? Can we develop techniques during training to encourage models to mimic the self-similarity we see in language? \n\nOn the architecture side, self-similarity in language means that linguistic patterns repeat at different scales. This mirrors how parameter sharing works: the same set of parameters (attention mechanisms) is applied across different positions in a sequence, assuming similar operations are needed to understand the language at different levels. This is what we mean by saying that the success of parameter sharing techniques, such as ALBERT, may be partially explained by self-similarity. We will explain this more in the revised version of the paper. We have some preliminary experiments, for instance, that show that parameter-sharing is more effective for language than in vision (presumably because language is self-similar). But, it is unclear yet what the best way to exploit self-similarity in the architecture is. We believe these are exciting areas for future research.\n\n**Typos**\n\nThank you for pointing them out. We will fix them in the revised version of the paper. \n\n**Follow-up**\n\nWe are grateful again for your detailed and constructive feedback. If we have satisfactorily answered your questions, we hope you would consider revising your scores. Otherwise, please let us know if there are any other questions or concerns, so we can respond to them during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tA7pwVH49z", "reviewer": "Reviewer_cohu", "summary": "This paper introduce a new perspective to that language is self-similar and predictability and self-similarity together imply long-range dependency, based on empirical analysis across different scales of LMs and information theoretic views. This new perspective may enable us to understand the strong capabilities of current causal LLMs. They studies three parameters, self-similarity (Hölder) exponent, the Hurst parameter, and the fractal dimension, as well as Joseph exponent. They also introduces a new metric that can more precisely approximately downstream performance than BPB.", "questions": "- Did authors try other models with publicly available model checkpoints so that people can reproduce the results? \n- Why did authors train T5-decoder model from scratch, instead of using T5 variant that went through next token prediction training (e.g., T5 LM adapt)?", "limitations": "The authors provide limitation sections that include some of the limitations I was thinking about.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper provides some new interesting perspectives to advance our understanding of how LLMs acquire such strong capabilities after large-scale pre-training using simple autoregressive training objective. \n- They conduct large-scale analysis using LMs with different scale in three different model families. \n- They also empirically found that a median Hurst exponent can be a more reliable indicator of downstream performance than perplexity-based BPB. While perplexity has shown to strongly correlated with downstream performance, prior studies also show their limitations on predicting downstream performance, and this work may encourage future work to use this new metric instead.", "weaknesses": "- To my understanding, this work views language  as a  mere sequence of negative log probabilities while prior studies in linguistic / NLP often consider much more rich structures of languages and am not fully convinced the validity of analysis.\n- While the analysis includes three models, none of the models’ checkpoints aren’t publicly available (i.e., PaLM and PaLM2 and their newly trained T5-decoder only models) and followup work may not be able to reproduce the results. I’m curious why the authors didn’t test other models with openly available model checkpoints such as Llama 2, 3 / OLMo / Pythia. \n- Overall I found the paper is a bit hard to follow (e.g., Introduction discusses prior literature in depth before providing high-level overview or motivation, Experimental setups come write after preliminaries). Some restructures of sections or writing may improves readability of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the careful and insightful feedback. We especially appreciate the positive assessment of the thoroughness, originality, and importance of our work. We hope that our rebuttal below addresses all of the reviewer’s questions, are happy to provide more details, and look forward to the reviewer’s response to our rebuttal.\n\n**Information-theoretic complexity**\n\nThank you for raising this point. Please see our general comment above on why we believe an information-theoretic complexity is meaningful for our analysis. We will clarify this more in the revised version of the paper.\n\n\n\n**Reproducibility**\n\nThis is an important point and we thank the reviewer for raising it. Our response is threefold. \n\nFirst, we have run new experiments using the released gemma-2b checkpoint (https://huggingface.co/google/gemma-2b) to compare it with the models used in the paper. Due to the short rebuttal time, we were able to use the smallest gemma model and a subset of the Pile validation split only. But, as discussed in the general comment above, we observe a good agreement overall. Gemma 2B is much smaller than all of the models we have used in our paper. Yet, we generally observe similar conclusions. Please see our general response above for details. Based on these findings, we do believe our results will continue to hold using publicly available checkpoints. We plan to continue running the analysis on the full validation data split of the Pile and include those results in the supplementary materials of the paper.\n\nSecond, the reason we included T5 trained from scratch is to ensure that our findings are reproducible. We believe that by training a model from scratch instead of using released checkpoints, we are more confident in our findings. We have provided all of our training details for reproducibility. We hope this answers your question. \n\nThird, to ensure reproducibility and to encourage the community to explore these fractal properties in language, we also release a code for calculating all fractal params. \n\n**Clarity of writing**\n\nWe will add more clarity in the revised version of the paper. If there are any specific suggestions for restructuring or clarifying some points, please let us know and we would gladly incorporate them into the paper. \n\n**Follow-up**\n\nWe are grateful again for your detailed and constructive feedback. If we have satisfactorily answered your questions, we hope you would consider revising your scores. Otherwise, please let us know if there are any other questions or concerns, so we can respond to them during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hAbG7fKEaS", "reviewer": "Reviewer_ND2C", "summary": "This paper provides a detailed application of ideas from fractal geometry to natural language data, using language models to compute the relevant information-theoretic properties.  They find, in particular, tell-tale evidence of self-similarity (common structure across scales) and long-range dependencies.  A particularly interesting result, in my opinion, is that using some of the estimated fractal parameters can help predict LMs' downstream performance over and above what can be predicted from raw LM performance (bits-per-byte), suggesting that some of this performance can be explained by their discovery of this fractal structure.  While the paper can be a bit dense and contains a huge range both of theoretical background and experimental results that make it a little hard to read, I do believe that it pays off to understand and makes a nice contribution that will be of interest to many people in the field.", "questions": "* Lines 46-47: if the reader is not familiar with S, it's not clear what the value means and why it has the implications that it does.  Can you explain more at this point or move these remarks to a bit later?\n* Have you checked whether a power-law is genuinely the best fit to the data, versus other laws that often resemble them (e.g. linear in log-log)? See e.g. https://doi.org/10.1126/science.1216142\n* Can you say more about the use of LLMs to compute and represent the information-theoretic quantities in this context?  As a way of making sure that we are learning about language in particular and not just artifacts/properties of the models.", "limitations": "Yes; I appreciated especially the mention of the results being English-only", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* Provides a thorough and detailed application of fractal geometry to natural language data.\n* Use of the new fractal metric helps predict downstream task performance more than just BPB alone.\n* Shows that the main findings are robust to the choice of LLM that is used for estimating information-theoretic properties of text.", "weaknesses": "* Could cite more literature about e.g. dependency lengths (https://doi.org/10.1073/pnas.1502134112) and other known properties of natural language.  For instance, \"duality of patterning\" (https://doi.org/10.1515/langcog-2012-0015) has been used to refer to the fact that morphemes->words and words->sentences exhibit similar structural properties, which is akin to self-similarity. \n* Although the paper is mostly self-contained, it can be very dense for readers not very familiar with the mathematics of fractals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the careful and insightful feedback. We especially appreciate the positive assessment of the thoroughness, originality, and importance of our work. We hope that our rebuttal below addresses all of the reviewer’s questions, are happy to provide more details, and look forward to the reviewer’s response to our rebuttal.\n\n**Additional References**\n\nThank you for the great suggestions. We will add those references along with a brief discussion in the related works section. \n\n**Clarity of writing**\n\nWe will add more clarity in the revised version of the paper. If there are any specific suggestions for restructuring or clarifying some points, please let us know and we would gladly incorporate them into the paper. \nWe will add more explanation about S and what it means prior to the discussion in Lines 46-47. Informally, a smaller value of S indicates a slower decay in the auto-correlation function and more fractal structure.\n\n**Power law fit**\n\nThis is an important point and we appreciate the suggested reference. In our experiments, we observe a near perfect linear fit on a log-log plot over more than two orders of magnitude (Figures 2 and 3), in agreement with what (Stumpf and Porter, 2012) argued for. (Stumpf and Porter, 2012) wrote: “As a rule of thumb, a candidate power law should exhibit an approximately linear relationship on a log-log plot over at least two orders of magnitude in both the x and y axes.” This is exactly what we observe. We will clarify this point in the revised version of the paper. \n\n**Information-theoretic complexity**\n\nThank you for raising this point. Please see our general comment above on why we believe an information-theoretic complexity is meaningful for our analysis. We will clarify this more in the revised version of the paper.\n\n**Follow-up**\n\nWe are grateful again for your detailed and constructive feedback. If we have satisfactorily answered your questions, we hope you would consider revising your scores. Otherwise, please let us know if there are any other questions or concerns, so we can respond to them during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their insightful and constructive feedback. We appreciate the positive feedback on the thoroughness, originality, and importance of our work. 

**Presentation**

We will incorporate the reviewers’ suggestions in the revised version of the paper. This includes providing missing references, clarifying the definition of S, and discussing the breadth of tasks covered in our evaluations, which include many tasks such as logical deduction, multi-step arithmetic, disambiguation, translation error detection, and general knowledge all prompted using various strategies (e.g. direct and chain-of-thought with either 0, 3, 5, or 8 shots). We believe our evaluations are quite thorough and diverse, and will clarify this more in the revised version of the paper.

**Public Checkpoints**

Please note that we have also applied the same methodology during the rebuttal period using the released gemma-2b checkpoint (https://huggingface.co/google/gemma-2b), in response to reviewer cohu's comments. Gemma 2B is much smaller than all of the models we have used in our paper. Yet, we generally observe similar conclusions. First, we have a self-similar structure with near perfect linear fits in a log-log plots. Second, we also observe power laws using the rescaled-range analysis, with a Hurst exponent of about 0.7 in most domains except DM Mathematics (smallest Hurst exponent of about 0.58) and GitHub (largest Hurst exponent of about 0.82), in general agreement to the rest of the models. Please see the attached PDF file for the figures generated using Gemma 2B. These results are based on a subset of the Pile validation split due to the short time constraint (for the rebuttal), so we plan to run the analysis on the full validation data and include those results in the supplementary materials of the paper. 

**Information-theoretic complexity**

We discuss in Lines 84-91 why we believe this information-theoretic complexity is meaningful for our analysis. It corresponds to an intrinsic, irreducible description of language and the minimum compute overhead to comprehend/decode it. One experimental evidence in psychology for why this characterization is natural for language comes from reading time measurements, which turned out to correlate well with information-theoretic complexity. In addition, fractal behavior has a clear interpretation in this context: e.g. surprising paragraphs will follow predictable paragraphs, in a manner that is statistically similar to how surprising sentences follow predictable sentences. Please see the discussion and references in Lines 84-91. We will clarify this more in the revised version of the paper.

While our work focuses on this specific aspect of language modeling, we recognize the potential for future research to explore more language structures, and we believe our work is a first step in the direction of exploring the relationship between language modeling and fractals. In addition, our findings regarding the self-similarity of "surprise" in language and the connection between Hurst parameter and downstream performance both are robust findings (as we show in the paper) and can be quite valuable on their own right.

**Follow-up**

If we have satisfactorily answered your questions, we hope you would consider revising your scores. Otherwise, please let us know if there are any other questions or concerns, so we can respond to them during the discussion period.

Meta-review: In this paper, the authors investigate the potential presence of fractal structures in language modeling, particularly within the framework of recent models that predict the next token. To support this idea, they focus on key properties of fractal structures: self-similarity, long-range dependence, and information-theoretic complexity. These properties are quantitatively described using metrics commonly associated with fractal phenomena, including the self-similarity exponent, Hurst parameter, fractal dimension, and the Joseph effect. Experimental results indicate that text modeling characteristics from ArXiv, GitHub, and Wikipedia datasets within The Pile's validation split, for sequences longer than 4K tokens, align with the authors' assumptions. Additionally, analyses of downstream tasks such as BBH, MMLU, and GSM8K reveal that task performance can generally be predicted from language modeling performance under these assumptions, except when sequence lengths during training are considered.

Overall, I find the paper to be interesting and the contribution solid. The weaknesses raised by the reviewers do not justify major corrections and were overally properly addressed by the authors. Yet, I do encourage the authors to seriously take their promise to make specific changes to the paper, as some reviewer comments have the potential to improve the paper significantly (e.g. comments on the linguistic properties of the baseline).
Decision: Accept (poster)