Fractal Patterns May Illuminate the Success of Next-Token Prediction

Authors: ibrahim  alabdulmohsin
vinh q tran
mostafa  dehghani
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 22:18:55

Abstract: We study the fractal structure of language, aiming to provide a precise formalism for quantifying properties that may have been previously suspected but not  formally shown. We establish that language is: (1) self-similar, exhibiting complexities at all levels of granularity, with no particular characteristic context length, and (2) long-range dependent (LRD), with a Hurst parameter of approximately 0.7.
Based on these findings, we argue that short-term patterns/dependencies in language, such as in paragraphs, mirror the patterns/dependencies over larger scopes, like entire documents. This may shed some light on how  next-token prediction can capture the structure of text across multiple levels of granularity, from words and clauses to broader contexts and intents. In addition, we carry out an extensive analysis across different domains and architectures, showing that fractal parameters are robust.
Finally, we demonstrate that the tiny variations  in fractal parameters seen across LLMs improve upon perplexity-based bits-per-byte (BPB) in predicting their downstream performance. We hope these findings offer a fresh perspective on language and the mechanisms underlying the success of LLMs.

Main paper: 
 introduction: 
 How does the training objective of predicting the next token in large language models (LLMs) yield remarkable capabilities? Consider, for instance, the two models: Gemini [5] and GPT4 [50] . These models have demonstrated capabilities that extend to quantitative reasoning, summarization, and even coding, which has led some researchers to ponder if there was more to intelligence than "on-the-fly improvisation" [11] . While providing a satisfactory explanation is a difficult endeavor, a possible insight can be drawn from fractals and self-similarity. We elucidate the connection in this work.Self-Similarity. Self-similar processes were introduced by Kolmogorov in 1940 [36] . The notion garnered considerable attention during the late 1960s, thanks to the extensive works of Mandelbrot and his peers [19] . Broadly speaking, an object is called "self-similar" if it is invariant across scales, meaning its statistical or geometric properties stay consistent irrespective of the magnification applied to it (see Figure 1 ). Nature and geometry furnish us with many such patterns, such as coastlines, snowflakes, the Cantor set and the Kuch curve. Despite the distinction, self-similarity is often discussed in the context of "fractals," another term popularized by Mandelbrot in his seminal book The Fractal Geometry of Nature [45] . However, the two concepts are different [26] . See Section 2.In language, in particular, there have been studies arguing for the presence of a self-similar structure. Nevertheless, due to computational constraints, it was not feasible to holistically model the joint probability distribution of language. As such, linguists often resorted to rudimentary approximations in their arguments, such as by substituting a word with its frequency or length [9] , or by focusing on Figure 1 : Manifestations of processes across different time scales. A region marked in red corresponds to the magnified plot beneath it. LEFT: The process exhibits self-similarity with rich details at all levels of granularity. It is an integral process (X t ) t∈N calculated from Wikipedia (see Section 2) . RIGHT: Example of a process that is not self-similar, looking smoother at larger time scales. the recurrence of a specific, predetermined word [49, 3] . These studies fall short of fully capturing the structure of language due to the simplifying assumptions they make, as discussed in Section 4.Highlighting the self-similar nature of a process can have profound implications. For instance, conventional Poisson models for Ethernet traffic were shown to fail because traffic was self-similar [16, 39, 51, 69] . In such cases, recognizing and quantifying self-similarity had practical applications, such as in the design of buffers [40] . Similarly in language, we argue that self-similarity may offer a fresh perspective on the mechanisms underlying the success of LLMs. Consider the illustrative example shown in Figure 1 , where the task is to predict the subsequent measurement in a time series, specifically predicting next tokens in a Wikipedia article (see Section 2 for details). The three plots in Figure 1 (left) represent different manifestations of the same process observed across three distinct time scales. Notably, we observe rich, self-similar details, such as burstiness, in all of them. A well-established approach for quantifying self-similarity is the Hölder exponent [66] , which we denote by S. In language, we find it to be S = 0.59 ± 0.08, confirming statistical self-similarity.Why is this important? We hypothesize that since LLMs are trained to predict the future of a selfsimilar process, they develop proficiency in capturing patterns across multiple levels of granularity for two interconnected reasons. First, self-similarity implies that the patterns at the level of a paragraph are reflective of the patterns seen at the level of a whole text, which is reminiscent of the recursive structure of language [53] . Thus, recognizing short-term patterns can aide in learning broader contexts. Second, because language displays intricate patterns at all levels of granularity, it would not be enough to rely only on the immediate context of a sentence to predict the next token. Instead, the model needs to identify patterns at higher levels of granularity; e.g. follow the direction of the argument and the broader intent. It must balance between short-and long-term contexts. Willinger et al. [68] and Altmann et al. [3] argue for self-similarity in language due to this hierarchical nature.Long-range dependence. However, self-similarity alone is not sufficient for a predictive model to exhibit anything resembling "intelligent" behavior. In fact, some self-similar processes, despite their intricate details, remain entirely unpredictable. A quintessential example is the simple Brownian motion, which is a Wiener process with independent increments. Its discrete analog is B n = n i=1 ε i , where ε i ∼ N (0, σ 2 ). Despite possessing rich details at all granularities, a model trained to predict B n cannot learn anything useful from data since the process itself has independent increments. Thus, for strong capabilities to emerge, the process must have some degree of predictability or dependence as well. One classical metric for quantifying predictability in a stochastic process is the Hurst parameter [31] , developed by the hydrologist H. E. Hurst in 1951 while studying the Nile river. It is generally considered to be a robust metric [68] , unlike the wavelet estimator [1] and the periodogram method [24] that can be sensitive to errors [54] . As discussed in Section 2.3, we find the Hurst parameter in language to be H = 0.70 ± 0.09. For context, H only takes values in [0, 1]. A value H > 0.5 implies predictability in the data, while H = 0.5 indicates random increments.While it is compelling that our estimate of H in language lies nearly midway between predictability (H = 1) and noise (H = 0.5), a Hurst parameter of about 0.75 turns out to occur commonly in nature, including in river discharges, Ethernet traffic, temperatures, precipitation, and tree rings [16, 21, 8] . For agents that learn from data, such as LLMs, this value is also reminiscent of processing-based theories of curiosity, which suggest that a sweet spot of complexity exists (not too simple, nor too unpredictable) that facilities or accelerates learning [34] .Importantly, predictability and self-similarity together imply long-range dependence (LRD). This follows from the definition of self-similarity, where the patterns at small scales mirror those at larger scales so, for example, the correlations established at micro levels are also pertinent at macro levels. LRD is arguably crucial for enhancing the functionality of predictive models because processes with only short-range dependence could be forecasted (somewhat trivially) with lookup tables that provide the likelihood of transitions over brief sequences. By contrast, this is not possible in LRD processes whose contexts extend indefinitely into the past.Information Theoretic Complexity. To define fractal parameters, we follow recent works such as [28, 22, 41, 47, 25] in adopting an information-theoretic characterization of the complexity in language using minimal-length codes or surprise. This corresponds to an intrinsic, irreducible description of language and the minimum compute overhead to comprehend/decode it [22] , which also correlates well with actual reading times [28, 41] . In this context, self-similarity means that the intrinsic complexity or surprise in language (measured in bits) cannot be smoothed out, even as we look into broader narratives. That is, surprising paragraphs will follow predictable paragraphs, in a manner that is statistically similar to how surprising sentences follow predictable sentences.Analysis. How robust are these findings? To answer this question, we carry out an extensive empirical analysis across various model architectures and scales, ranging from 1B to over 500B parameters. We find that fractal parameters are quite robust to the choice of the architecture.However, there exists tiny variations across LLMs. Interestingly, we demonstrate that from a practical standpoint, these differences help in predicting downstream performance in LLMs compared to using perplexity-based metrics alone, such as bits-per-byte (BPB). Specifically, we introduce a new metric and show that using it to predict downstream performance can increase the adjusted R2 from approximately 0.65 when using solely BPB, to over 0.86 with the new metric 2 .In summary, we:1. highlight how the fractal structure of language can offer a new perspective on the capabilities of LLMs, and provide a formalism to quantify properties, such as long-range dependence. 2. establish that language is self-similar and long-range dependent. We provide concrete estimates in language of the three parameters: the self-similarity (Hölder) exponent, the Hurst parameter, and the fractal dimension. We also estimate the related Joseph exponent. 3. carry out a comparative study across different model architectures and scales, and different domains, such as ArXiv and GitHub, demonstrating that fractal parameters are robust. 4. connect fractal patterns with learning. Notably, we show that a "median" Hurst exponent improves upon perplexity-based bits-per-byte (BPB) in predicting downstream performance.2 Fractal Structure of Language 
preliminaries: 
 Suppose we have a discrete-time, stationary stochastic process (x t ) t∈N , with E[x t ] = 0 and E[x 2 t ] = 1. We will refer to (x t ) t∈N as the increment process to distinguish it from the integral process (X t ) t∈N defined by X t = t k=0 x k . While (x t ) t∈N and (X t ) t∈N are merely different representations of the same data, it is useful to keep both representations in mind. For example, self-similarity is typically studied in the context of integral processes whereas LRD is defined on increment processes.In the literature, it is not uncommon to mistakenly equate parameters that are generally different. For example, the Hurst parameter H has had many definitions in the past that were not equivalent, and Mandelbrot himself cautioned against this [44] . The reason behind this is because different parameters can agree in the idealized fractional Brownian motion, leading some researchers to equate them in general [66] . We will keep the self-similarity exponent S and H separate in our discussion. Experimental Setup. In order to establish self-similarity and LRD in language, we convert texts into sequences of bits using a large language model (LLM). Specifically, we use PaLM2-L (Unicorn) [6] to calculate the probability of the next token w t conditioned on its entire prefix w [t-1] = (w 0 , w 1 , . . . , w t-1 ). As discussed in Section 1, this captures its intrinsic, irreducible description [22] . By the chain rule [15] , the corresponding number of bits assigned to w t is z t = -log p(w t |w [t-1] ). Unlike in prior works, which rely on simplifications such as by substituting a word with its length [9] or by focusing on the recurrence of a single word [49, 3] , we use the LLM to approximate the full joint distribution of language since LLMs are known to produce calibrated probability scores at the token level [33] . We carry out these calculations for prefixes of up to 2048 tokens (≈ 8 pages of text). With a suitable normalization, such as bits-per-byte (BPB), one obtains a standardized description of text, consistent across tokenizers. BPB is widely used as a tokenizer-agnostic metric to compare LM modeling performance, e.g. for The Pile [23] .Besides PaLM2, we also experiment and report on various model sizes of PaLM [12] and decoderonly T5 [55] . Namely, we report results for models: PaLM2 XXS (Gecko), XS (Otter), S (Bison), M, and L (Unicorn); PaLM 8B, 62B, 540B; and decoder-only T5.1.1 at Base (110M), Large (341M), XL (1.2B), and XXL (5B) sizes. For PaLM and PaLM2, we use the checkpoints pretrained in Chowdhery et al. [12] and Anil et al. [6] . All T5.1.1 decoder baselines, on the other hand, are trained with a casual language modeling objective for 262B tokens of C4 [55] . All experiments are executed on Tensor Processing Units (TPUs). More details on how we train T5.1.1 baselines are in Appendix A. Once z t is computed for a document, we follow standard definitions in constructing the increment process (x t ) t∈N by normalizing z t to have a zero-mean and unit variance. Intuitively, fractal parameters are intended to measure a fundamental property of the process (e.g. LRD) that should not be affected by scale, hence the normalization. The integral process (X t ) t∈N is calculated based on (x t ) t∈N , as described earlier and depicted in Figure 1 (top). Normalizing bits (to have zero mean and unit variance) models language as a random walk. It is a standard approach used extensively in the literature in various contexts, such as in DNA sequences [52, 57, 48, 35, 59 ]. For analysis, we use The Pile validation split [23] , consisting of 22 subdomains such as Wikipedia and GitHub. We restrict analysis to sufficiently-long documents of length > 4K tokens and use the first 2K tokens only, to sidestep potential effects of the finite length of documents and the model context. To mitigate noise, only domains with > 1K documents are compared; we report results for them separately and their median. We use bootstrapping [17] to estimate the error margin.Notation. We write f (x) ∼ x c if f (x) = x c L(x) for some function L that satisfies L(tx)/L(x) → 1 as x → ∞ for all t > 0. Examples of slowly varying functions are constants L(x) = c and L(x) = log x. When f (x) ∼ x c , we abuse terminology slightly by referring to f (x) as a power law.An integral process is said to be self-similar if it exhibits statistical self-similarity. More precisely, (X t ) t∈N is self-similar if (X τ t ) t∈N is distributionally equivalent to (τ S X t ) t∈N for some exponent S. Thus, scaling of time is equivalent to an appropriate scaling of space. We will refer to τ as the granularity level and to the exponent S as the self-similarity or Hölder exponent [66] . Many time series in nature exhibit self-similar structures, such as human blood pressure and heart rate [27] .One approach for calculating S is as follows. Fix 1 and denote the τ -increments by (X t+τ -X t ) t∈N . These would correspond, for instance, to the number of bits used for clauses, sentences, paragraphs and longer texts as τ increases. In terms of the increment process (x t ) t∈N , this corresponds to aggregating increments into "bursts". Let p (τ ) be the probability mass of the event {|X t+τ -X t | ≤ } t∈N . Then, S can be estimated by fitting a power law relation p (τ ) ∼ τ -S [66] . Generally, S is robust to the choice of ∈ [10 -3 , 10 -2 ] as shown in Figure 4 (left) so we fix it to = 5 × 10 -3 . Figure 2 plots the probability p (τ ) against τ using PaLM2-L. We indeed observe a power law relation over at least two orders of magnitude; i.e. linear in a log-log scale, with a median self-similarity exponent of S = 0.59 ± 0.08. Section 3 shows that the median S is robust to the choice of the LLM.The Hurst parameter H ∈ [0, 1] quantifies the degree of predictability or dependence over time [31] . It is calculated using the so-called rescaled-range (R/S) analysis. Let (x t ) t∈N be an increment process. For each n ∈ N, writey t = x t -1 t t k=0x k and Y t = t k=0 y t . The range and scale are defined, respectively, as R(n) = max t≤n Y t -min t≤n Y t and S(n) = σ ({x k } k≤n ), where σ is the standard deviation. Then, the Hurst parameter H is estimated by fitting a power law relation R(n)/S(n) ∼ n H . As stated earlier, for completely random processes, such as a simple Brownian motion, it can be shown that H = 1/2. In addition, H > 1/2 implies dependence over time [16, 68, 8] .Writing ρ n = E[(x t+n x t ] for the autocovariance function of the increment process (x t ) t∈N , the Hurst parameter satisfies H = 1 -β/2 when ρ n ∼ n -β as n → ∞ [26, 16] . Since in self-similar processes, H > 1/2 implies long-range dependence (LRD), LRD is equivalent to the condition that the autocovariances are not summable. In terms of the integral process, it can be shown that [58] : Var(X n ) grows, at most, linearly fast on n. On the other hand, if the process has LRD, Var(X n ) grows superlinearly on n. In particular, using the Euler-Maclaurin summation formula [7, 2] , one obtainslim n→∞ Var(Xn) n = 1 + 2 ∞ i=1 ρ i . Hence, if H < 1/2,Var(X n ) ∼ n 2H if H > 1/2. Figure 3 plots the rescaled range R(n)/S(n) against n.We observe a power law relation with a median Hurst parameter of H = 0.70 ± 0.09.Broadly speaking, the fractal dimension of an object describes its local complexity. For a geometric object Z, such as the Koch curve, let τ be a chosen scale (e.g. a short ruler for measuring lengths or a small square for areas). Let N (τ ) be the minimum number of objects of scale τ that cover Z; i.e. contain it entirely. Then, the fractal dimension of Z, also called its Hausdorff dimension, is:D = -lim τ →0log N (τ ) log τ [54] . For example, a line has a fractal dimension 1, in agreement with its topological dimension, because N (τ ) = C/τ for some constant C > 0.By convention, an object is referred to as "fractal" if D is different from its topological dimension. For example, the fractal dimension of the Koch curve is about 1.26 when its topological dimension is 1. Fractals explain some puzzling observations, such as why estimates of the length of the coast of Britain varied significantly from one study to another, because lengths in fractals are scale-sensitive. Mandelbrot estimated the fractal dimension of the coast of Britain to be 1.25 [43] .The definition above for the fractal dimension D applies to geometric shapes, but an analogous definition has been introduced for stochastic processes. Let (x t ) t∈R be a stationary process with autocovariance ρ n . Then, its fractal dimension D is determined according to the local behavior of ρ n at the vicinity of n = 0, by first normalizing (x t ) t∈R to have a zero-mean and a unit variance, and modeling ρ n using a power law ρ n ∼ 1 -n α as n → 0 + , for α ∈ (0, 2]. Then, the fractal dimension D ∈ [1, 2] of (x t ) t∈R is defined by D = 2 -α/2 [26] . It can be shown that D = 2 -S [26] . For language, this gives a median fractal dimension of D = 1.41 ± 0.08.Finally, we examine another related parameter that is commonly studied in self-similar processes. The motivation behind it comes from the fact that in processes with LRD, one often observes burstiness as shown in Figure 1 ; i.e. clusters over time in which the process fully resides on one side of the mean, before switching to the other. This is quite unlike random noise, for instance, where measurements are evenly distributed on both sides of the mean. The effect is often referred to as the Joseph effect, named after the biblical story of the seven fat years and seven lean years [68, 46, 66] .A common way to quantify the Joseph effect for integral processes (X t ) t∈N is as follows [66] . First, let σ τ be the standard deviation of the τ -increments X t+τ -X t . Then, fit a power law relation σ τ ∼ τ J . The exponent J here is called the Joseph exponent. In an idealized fractional Brownian motion, both J and the self-similarity exponent S coincide. Figure 5 provides the detailed empirical results. Overall, we find that J = 0.49 ± 0.08.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Section 4Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
analysis: 
 Comparative Analysis. Table 1 compares fractal parameters across different domains, such as ArXiv, Github and Wikipedia. In general, most domains share similar self-similarity and Hurst exponents with a few exceptions. The first notable exception is DM-Mathematics, which has a Hurst parameter of about 0.5, indicating a lack of LRD. Upon closer inspection, however, a value of H = 0.5Figure 5 : The standard deviation σ of the τ -increments X t+τ -X t is plotted against the scale τ . We, again, observe another power law relation σ ∼ τ J , with a Joseph exponent J = 0.49 ± 0.08. is not surprising for DM-Mathematics because its documents consist of independent mathematical questions as shown in Figure 6 . In Figure 4 (right), we plot the partial autocorrelation function for each of the 8 domains against time lag (context length). Indeed, we see that DM-Mathematics shows markedly less dependence compared to the other domains. The second notable observation is the relatively larger value of H = 0.79 in GitHub, indicating more structure in code. This is in agreement with earlier findings by Kokol and Podgorelec [35] who estimated LRD in computer languages to be greater than in natural language. In Table 2 , we compare the three fractal parameters S, H and J using different families of LLM and different model sizes. Overall, we observe that the parameters are generally robust to the choice of the architecture.Downstream Performance. By definition, fractal parameters are calculated on the sequence of negative log-probability scores after normalizing them to zero-mean and unit variance. Hence, they may offer an assessment of downstream performance that improves upon using a perplexity-based metric like bits-per-byte (BPB) alone. To test this hypothesis, we evaluate the 12 models in Table 2 on challenging downstream zero-and few-shot benchmarks focusing on language understanding and reasoning. We include results for 0-shot (0S) and 3-shot (3S) evaluation for BIG-Bench Hard tasks [63, 64] reporting both direct and chain-of-thought (CoT) prompting results following Chung et al. [13] . In addition we report 0-shot and 5-shot (5S) MMLU [30] , and 8-shot (8S) GSM8K [14] with CoT. Raw accuracy is reported for all tasks. BBH and MMLU scores are averaged across all 21 tasks and 57 subjects, respectively. These benchmarks are quite diverse and include tasks such as logical deduction, arithmetic, translation error detection, disambiguation, as well as general knowledge (e.g. history, computer science, law, sports, movies, etc). All prompt templates for our evaluation are taken from Chung et al. [13] , Longpre et al. [42] , which we refer the reader to for more details. We prompt all models using a 2048 context length. See Table 8 of Appendix C for the full results.The first (surprising) observation is that the median Hurst parameter is itself strongly correlated with the BPB scores with an absolute Pearson correlation coefficient of 0.83, even though the Hurst exponent is calculated after normalizing all token losses to zero-mean and unit variance! Informally, this implies that second-order statistics on the sequence of token losses of a particular model can predict its mean! Self-similarity exponent, by contrast, has an absolute correlation of 0.23 with BPB.Figure 7 displays downstream performance against both the median Hurst exponent and the median BPB score, where median values are calculated on the 8 domains in The Pile benchmark listed in Table 1 . In general, both the BPB score and the median Hurst are good predictors of downstream performance. However, we observe that improvements in BPB alone without impacting the median Hurst exponent do not directly translate into improvements downstream. This is verified quantitatively in Table 3 (middle), which reports the adjusted R 2 values -the proportion of variance in each downstream metric that can be predicted using BPB, H, or by combining them together into H B = 1/BPB + H, with BPB replaced with its reciprocal so that higher values are better. We observe that H B yields indeed a stronger predictor of downstream performance. Hence, while H and BPB are correlated, combining them yields a better predictor, so each of H and BPB conveys useful information not captured by the other metric. See Appendix C for similar analysis using the exponents S and J.Context Length at Training Time (Negative Result). Finally, we present a negative result. Selfsimilarity and LRD point to an intriguing possibility: the importance of training the model with extensive contexts in order to capture the fractal-nature of language, which may elevate the model's capabilities regardless of the context length needed during inference. To test this hypothesis, we pretrain three decoder-only T5.1.1 models with 1B parameters on SlimPajama-627B [62] for up to 100B tokens using three context lengths: 2K, 4K and 8K, all observing the same number of tokens per batch. We use SlimPajama-627B instead of C4 because most documents in C4 are short (≈ 94% of them are < 2K tokens in length). Refer to Appendix A for details. These models are, then, evaluated on the same downstream benchmarks listed in Figure 7 . As shown in Table 3 (right) however, we do not observe any improvements in performance with context length in this particular setup.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Guidelines: We follow well-established definitions throughout our analysis.• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
related works and directions for future research: 
 The statistical attributes of human language have long piqued scholarly curiosity. One example is Zipf's law, which Shannon leveraged to estimate the entropy of English to be around 1 bit per letter [60] , but his calculation did not consider second-order statistics. More recently, Eftekhari [18] proposed a refinement to Zipf's law, suggesting its application to letters rather than words. Another related result is Heap's law, which states that the number of unique words is a power law function of the document's length [29] . However, both Zipf's and Heap's laws are invariant to the semantic ordering of text, so they do not capture important aspects, such as long-range dependence (LRD) [49] .In terms of self-similarity in language, the Menzerath-Altmann law stipulates a self-similar behavior in the following sense: when the size of a language construct increases, the size of its constituents decreases, and this happens at all scales [49, 4] . In Ausloos [9] , the authors model texts as a time series by replacing a word with its length. After that, they study the fractal behavior of language. However, as mentioned in [22] , replacing a word with its length is invalid because it is not translationindependent (i.e. one could map every word to an arbitrary token, including tokens of equal length). In our work, we model language as a series of bits calculated from conditional entropies, reflecting the intrinsic structure of the language itself, inspired by findings in linguistics such as [28, 22, 41] . The existence of self-similarity in language is attributed to its hierarchical nature [68, 3] , such as duality of patterning [37] .In Najafi and Darooneh [49] , the authors define a fractal dimension for each word. Informally, they examine the recurrence of a single, predetermined word as a binary series, similar to the approach used in Altmann et al. [3] . However, this only applies to individual words and cannot model higherlevel clauses. For instance, it does not distinguish between "time" in the phrase "once upon a time" and "time" in "space and time." Kokol and Podgorelec [35] estimate LRD in natural language, and suggest that its LRD is close to that of pure noise! They conjecture this was due to their use of ASCII encoding. In computer languages, they observe LRD and suggest it is because they are formal.Besides the above concerns in prior studies that examined the self-similar structure in language, another concern is that they sometimes give extremely large values of the fractal dimension, sometimes exceeding 10 [4] ! Such values are difficult to interpret because the fractal dimension D should fall in D ∈ [1, 2] for time series. We do not observe such issues in our analysis. In our case, D = 1.41±0.08.Limitations and Future Research. Our analysis is currently limited to the English language so it may not apply to other languages that differ significantly. For instance, some languages such as Pirahã (spoken in the Amazon) do not have a recursive structure like most languages do [20] . We also do not model the semantic or lexical form of language. While our information-theoretic approach is well-founded and captures the intrinsic complexity of language, it does not account for the semantic nuances that contribute to meaning. Thirdly, self-similarity may explain why parameter sharing, such as in ALBERT [38] , can be successful but exploiting self-similarity more directly in LLMs could lead to further optimizations. Exploring these aspects are promising directions for future research.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• The full details can be provided either with the code, in appendix, or as supplemental material. 
concluding remarks: 
 In this work, we highlight intriguing insights into the underlying fractal structure of language and how it may be interconnected with the remarkable capabilities of LLMs. Our formalism quantifies properties of language that may have been suspected, but not previously formally shown. In particular, the need in LLMs to balance between short-and long-term contexts is reflected in the self-similar structure of language, while long-range dependence is quantifiable using the Hurst parameter. For instance, the absence of LRD in DM-Mathematics is reflected in its Hurst parameter of H ≈ 0.5. Interestingly, the estimated median Hurst value of H = 0.70 ± 0.09 in language reflects an intriguing balance between predictability and noise that is similar to many other phenomena, and combining both H with BPB together yields a stronger predictor of downstream performance. We carry out an extensive comparative analysis across different domains and model architectures, revealing that fractal parameters are generally robust. We hope that future research can further probe into these fractal properties, unearthing deeper understandings of the relation between intelligence and language.All of our experiments are conducted in JAX/Flax [10] using the open source T5X framework [56] .T5 baselines in Table 2 and 3 are pretrained from scratch using the open source T5.1.1 decoder-only architecture from the T5X library. 3 . We pretrain using a causal language modeling objective over the C4 corpus with the default T5 vocabulary as per Raffel et al. [55] . Training is done for 500k steps with a sequence length of 1024 and batch size of 512, resulting in a total of 262B tokens seen during pretraining. We optimize our model with the Adafactor [61] optimizer with an inverse square root learning rate schedule, 1k warmup steps, and an initial learning rate of 1e-2. Models are trained using 256 TPUv5e chips [32] .T5 context length ablation experiments in Table 3 are trained with the same pretraining objective but over the SlimPajama-627B corpus [62] and using a modified version of the T5 vocabulary that preserves whitespace and introduces byte-fallback for out of vocabulary tokens. This is similar to Chowdhery et al. [12] , but preserving the original T5 vocabulary. Models with sequence lengths 2048, 4096, 8192 are trained with batch sizes of 512, 256, and 128 respectively to preserve the number of tokens seen per batch and overall training steps. We train all models for 100k steps, using the same learning rate schedule described above. Hence, all models observe 100B tokens.In this section, we provide the full list of parameters calculated for each combination of LLM and domain. We use bootstrapping [17] to estimate the error margin. Please note that our results are not directly comparable to all previous published results for the same models; please cite the original results from [12, 6] . Here, we only aim for a fair comparison between models: only pretrained models without instruction tuning are used, we do not optimize any prompts for each model, and we evaluate all models using only a 2K sequence length. Table 9 : Adjusted R 2 , which measures the proportion of variation in downstream performance (row) that is predictable from the given input(s) (column) using a trained linear regressor. Unlike in the median Hurst exponent, we do not observe any improvement when combining BPB scores with the self-similarity exponent S or the Joseph exponent J.In this section, we present empirical results using the publicly released Gemma-2B checkpoint [65] .Results are shown in Figures 8 and 9 .Gemma-2B is much smaller than all of the models we have used previously. Yet, we generally observe similar conclusions. First, we have a self-similar structure with near perfect linear fits in a log-log plots. Second, we also observe power laws using the rescaled-range analysis, with a Hurst exponent of about 0.7 in most domains except DM Mathematics (smallest Hurst exponent of about 0.58) and GitHub (largest Hurst exponent of about 0.82), in general agreement to the rest of the models. Both the self-similarity and Hurst parameters are provided in Table 10 .Here, we report some early investigations that examine how fractal parameters change as one includes longer contexts during inference. The setup used in this experiment is slightly different from what we use throughout the rest of the paper. Specifically, we take the Wikipedia dataset Figure 8 : Here, we follow a similar setup to Figure 2 , but using the publicly released Gemma-2B checkpoint [65] , where the y-axis is the peak probability. We continue to observe linear fits in a log-log scale over at least two orders of magnitude, thus confirming a self-similar structure. Figure 9 : Here, we follow a similar setup to Figure 3 , but using the publicly released Gemma-2B checkpoint [65] , where the y-axis is the rescaled-range (R/S). We continue to observe linear fits in a log-log scale over at least two orders of magnitude.(wikipedia/20230601.en) dataset [67] . We split each document of length >4K words along word boundaries and constrain the context length during inference in PaLM-8B to n words, for n ∈ {1, 16, 32, 64, 128, 256, 512, 1024, 2048}. Hence, the language model now resembles an n-gram model. We calculate probability scores and calculate fractal parameters accordingly. Figure 10 shows how both the self-similarity exponent S and the Hurst parameter H change as a function of n.We observe that H increases monotonically with context length as expected, since it implies more predictability. However, even in a 1-gram model, H can be larger than 1/2 because the words themselves are not independent of each other, so the sequence of probability scores can still contain dependence over time in a 1-gram model. Table 10 : Self-similarity exponent (S) and the Hurst parameter (H) when using Gemma-2B. These values compare well to the ones obtained using the other models. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We use the bootstrap method to estimate error margins and report those in the paper.Guidelines:• The answer NA means that the paper does not include experiments. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our work studies the fractal structure of language. We do not foresee any potential negative societal impact this work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original creators/owners of all datasets and model checkpoints that we use in our analysis. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.We release the code for calculating fractal parameters at: https://github.com/google-research/ google-research/tree/master/fractals_languagehttps://github.com/google-research/t5x/tree/main/t5x/examples/decoder_only/ models 
 
 Appendix:  


Decision: In this paper, the authors investigate the potential presence of fractal structures in language modeling, particularly within the framework of recent models that predict the next token. To support this idea, they focus on key properties of fractal structures: self-similarity, long-range dependence, and information-theoretic complexity. These properties are quantitatively described using metrics commonly associated with fractal phenomena, including the self-similarity exponent, Hurst parameter, fractal dimension, and the Joseph effect. Experimental results indicate that text modeling characteristics from ArXiv, GitHub, and Wikipedia datasets within The Pile's validation split, for sequences longer than 4K tokens, align with the authors' assumptions. Additionally, analyses of downstream tasks such as BBH, MMLU, and GSM8K reveal that task performance can generally be predicted from language modeling performance under these assumptions, except when sequence lengths during training are considered.

Overall, I find the paper to be interesting and the contribution solid. The weaknesses raised by the reviewers do not justify major corrections and were overally properly addressed by the authors. Yet, I do encourage the authors to seriously take their promise to make specific changes to the paper, as some reviewer comments have the potential to improve the paper significantly (e.g. comments on the linguistic properties of the baseline).

Reviews: review: paper_summary: paper_summary: In this paper, the authors try to reveal the existence of fractal structures to handle language in language modeling using recent language models based on the next token prediction. For that purpose, the authors rely on several aspects of fractal structures, which are self-similarity, long-range dependence, and information-theoretic complexity. The authors describe these aspects using metrics for typical phenomena under the assumption of fractal structure, which are self-similarity exponent, Hurst parameter, fragmental dimension, and Joseph effect. The experimental results show that the characteristics of modeling texts of ArXiv, Github, and Wikipedia data in The Pile validation split, whose tokens are longer than 4K, are along with their assumption. In the analysis of the downstream tasks, BBH, MMLU, and GSM8K also show that the task-solving performances are predictable from the language modeling performance based on their assumption, except for considering sequence lengths in training.
strength_weakness: strength_weakness: - The target language is limited to English. Thus, the validity of expanding the insights in the paper to other languages is uncertain.
- The assumed baseline in this analysis is a random sequence. The authors can use finite-state automatons (FSAs) or Context-free grammars (CFGs) to make sequences that are random but close to languages. Moreover, to focus on the success of recent pre-trained language models, you should have prepared N-gram language models as baselines.
- How strongly the observation results fit the assumed distribution is not calculated mathematically, like model selection by Information Criteria such as Akaike's Information Criteria (AIC) and Bayesian information criteria (BIC).
- There is a gap between the ability to mimic the characteristics of language and showing the required knowledge for questions. Thus, discussing the correlation between the performance of language modeling and downstream tasks is limited from this viewpoint. Hence, targeting generation tasks like summarization, story generation, and machine translation are more suitable for deepening the analysis.
questions: questions: - In the analysis, the authors seem to only focus on the context length in training even though the inference of long sequential tokens is closer to the aspect of long distant dependencies between tokens in language. What is the main reason for this decision?

[Comment] In presentation style, instead of showing the method name, showing the motivation or target phenomenon as the section or paragraph title, like "Joseph effect" -> "Burstiness," may support the reader's understanding.
limitations: limitations: The limitation does not include the gap between language modeling and solving downstream tasks.
paper_summary: paper_summary: The paper draws connections between fractal patterns and language by evaluating properties such as self-similarity and long-range-dependency. Using a range of LLMs, they estimate the Holder exponent, Hurst parameter and fractal dimension for language from different domains, including web text, code, and math problems. They show that these parameters may have connections with LLM learning ability, demonstrating that using the median Hurst parameter can improve prediction of downstream model performance over only using bits-per-byte.
strength_weakness: strength_weakness: - Even though fractal parameters were found to be consistent across LLMs, I think their stated contribution (this 'establish[es] that language is self-similar and long-range-dependent') is far too strong a claim. As the authors mention, their method ignores many important facets of language, such as semantic nuance, and is reliant on the current state of LLMs' ability to model language. 
- I am unclear how future work can build upon these insights about language, especially that 'exploiting self-similarity more directly' could lead to further LLM optimization.
questions: questions: I don't follow the potential connection between self-similarity and the success of parameter sharing? (mentioned in Limitations section)

Typos: 
- Line 131: such bits-per-byte (BPB)
- Line 132: BPB is a widely used as a
- Line 272: such as One example
limitations: limitations: Yes, the authors discuss that their analysis is limited to English data and fails to capture the semantic meaning of language. However, as mentioned in Questions section, the possible connection between self-similarity and parameter-sharing is not obvious to me and could be further explained.
paper_summary: paper_summary: This paper introduce a new perspective to that language is self-similar and predictability and self-similarity together imply long-range dependency, based on empirical analysis across different scales of LMs and information theoretic views. This new perspective may enable us to understand the strong capabilities of current causal LLMs. They studies three parameters, self-similarity (Hölder) exponent, the Hurst parameter, and the fractal dimension, as well as Joseph exponent. They also introduces a new metric that can more precisely approximately downstream performance than BPB.
strength_weakness: strength_weakness: - To my understanding, this work views language  as a  mere sequence of negative log probabilities while prior studies in linguistic / NLP often consider much more rich structures of languages and am not fully convinced the validity of analysis.
- While the analysis includes three models, none of the models’ checkpoints aren’t publicly available (i.e., PaLM and PaLM2 and their newly trained T5-decoder only models) and followup work may not be able to reproduce the results. I’m curious why the authors didn’t test other models with openly available model checkpoints such as Llama 2, 3 / OLMo / Pythia. 
- Overall I found the paper is a bit hard to follow (e.g., Introduction discusses prior literature in depth before providing high-level overview or motivation, Experimental setups come write after preliminaries). Some restructures of sections or writing may improves readability of the paper.
questions: questions: - Did authors try other models with publicly available model checkpoints so that people can reproduce the results? 
- Why did authors train T5-decoder model from scratch, instead of using T5 variant that went through next token prediction training (e.g., T5 LM adapt)?
limitations: limitations: The authors provide limitation sections that include some of the limitations I was thinking about.
paper_summary: paper_summary: This paper provides a detailed application of ideas from fractal geometry to natural language data, using language models to compute the relevant information-theoretic properties.  They find, in particular, tell-tale evidence of self-similarity (common structure across scales) and long-range dependencies.  A particularly interesting result, in my opinion, is that using some of the estimated fractal parameters can help predict LMs' downstream performance over and above what can be predicted from raw LM performance (bits-per-byte), suggesting that some of this performance can be explained by their discovery of this fractal structure.  While the paper can be a bit dense and contains a huge range both of theoretical background and experimental results that make it a little hard to read, I do believe that it pays off to understand and makes a nice contribution that will be of interest to many people in the field.
strength_weakness: strength_weakness: * Could cite more literature about e.g. dependency lengths (https://doi.org/10.1073/pnas.1502134112) and other known properties of natural language.  For instance, "duality of patterning" (https://doi.org/10.1515/langcog-2012-0015) has been used to refer to the fact that morphemes->words and words->sentences exhibit similar structural properties, which is akin to self-similarity. 
* Although the paper is mostly self-contained, it can be very dense for readers not very familiar with the mathematics of fractals.
questions: questions: * Lines 46-47: if the reader is not familiar with S, it's not clear what the value means and why it has the implications that it does.  Can you explain more at this point or move these remarks to a bit later?
* Have you checked whether a power-law is genuinely the best fit to the data, versus other laws that often resemble them (e.g. linear in log-log)? See e.g. https://doi.org/10.1126/science.1216142
* Can you say more about the use of LLMs to compute and represent the information-theoretic quantities in this context?  As a way of making sure that we are learning about language in particular and not just artifacts/properties of the models.
limitations: limitations: Yes; I appreciated especially the mention of the results being English-only

Comments: title: Thank you
I believe the authors
Thank you
Some concerns are resolved
Thank you for your response. I'll keep my score.
Follow up
Follow up
Follow up
Follow up
comment: Dear reviewer, 

Thank you very much for the suggestions and the continued engagement.

- We indeed expect that the Hurst exponent would be close to 0.5 for unigram/bigram models. We will add those to the figure as well.

- As requested, we will add separate plots in the appendix for downstream performance plotted against the median Hurst parameter. 

Sincerely
Thank you!  And apologies for my delayed response.  I am happy with the response and your engagement with the other reviews as well.  I would like to hear more about "why we believe an information-theoretic complexity is meaningful", and the use of LLMs for that measurement, but it's by no means a blocker for me, and I would like to see the paper accepted.
Thanks for emphasizing your contributions: after reading through the material provided during rebuttal (particularly your additional n-gram experiments for reviewer UXZb), I agree with the points you make and I will raise my score to 5 and change contribution to 2 (fair). However, I'd appreciate if you could clarify one thing about the Palm-8B n-gram experiment: for your smallest prefix-length setting (16-gram?), you observe a Hurst exponent of ~68.5. I do not have a strong intuition for sensible values, but it seems that this is still quite high? Especially because as you mention, we would not expect n-gram models, where n is small, to generate outputs with LRD. Would it be correct to assume that for a very small n-gram model (unigram/bigram/4-gram), we would expect Hurst parameter close to 0.5, and if so, is this what we observe? 

Aside from this, I also have a suggestion for clarity: for the revised version of the paper, I would appreciate if you could provide alternative versions of Figure 7 for the Appendix. I'd like separate plots where downstream performance (currently represented by bubble size) is plotted against median BFP/Hurst/Hb. This would make it easier to qualitatively evaluate the relationship between these values.
Thank you so much for your immediate response! 

>In this experiment, we limit the context length of PaLM-8B during inference to n tokens (so that it becomes an n-gram model) and use it to generate the sequence. Then, we calculate the fractal parameters  and  and plot those against . As you can see from the figures,  indeed increase as  increases before it converges to some definite value, which makes sense because a larger  indicates more LRD.

Yes, restricting the context length and varying its size is essentially the same as the n-gram language model. Thus, including such experimental results can be what I expected. Since this conversation is conducted on OpenReview and I believe the authors will keep their promise, I'll raise my score to 7. Note that to see the drastic changes between the successful recent large language models and count-based traditional models, I still recommend you use count-based n-gram language models or PCFGs as your baseline. Of course, it's not mandatory.
Dear reviewer 

Thank you for engaging with us and for clarifying your question about n-gram models. We have done an early experiment as a sanity check (to verify that H indeed captures LRD) but did not include it in the paper because it was only done on a single model (PaLM-8B) on a single domain (Wikipedia). You can find the results here: https://ibb.co/MhDh9CW and https://ibb.co/Btq94Kz. 
 
In this experiment, we limit the context length of PaLM-8B *during inference* to n tokens (so that it becomes an n-gram model) and use it to generate the sequence. Then, we calculate the fractal parameters $S$ and $H$ and plot those against $n$. As you can see from the figures, $H$ indeed increase as $n$ increases before it converges to some definite value, which makes sense because a larger $H$ indicates more LRD. 

If this is the experiment you are looking for, we promise to include a more detailed version of the experiment (using more models and data domains) in the supplementary materials. 

Thank you
Thank you for engaging with us. We would appreciate it if you let us know of any remaining concerns so we can respond to them.

With regard to the "poor" contribution rating, we would like to respectfully emphasize the novelty and technical quality of our work.

- We believe our work is *quite original*: we are not aware of any prior work that has attempted to study the fractal nature of language using LLMs. We offer a quantitative evidence for the self-similar and long-range dependent nature of language.

- We demonstrate an *interesting* connection between the Hurst parameter and downstream performance of LLMs. This is by no means obvious or could have been expected in advance. 

- We have conducted an *extensive* empirical analysis across various domains and architectures, showcasing the robustness of our findings. This includes downstream evaluations on popular benchmarks that cover many tasks (please see our response above), prompted using various strategies (e.g. direct and chain-of-thought with either 0, 3, 5, or 8 shots). 

- Our work is *self-contained*: we provide the necessary mathematical background to explain how to calculate all fractal parameters, and we have strived to make it easy to read and follow.

- We include all details necessary to reproduce the results in the supplementary materials, in addition to releasing the code for calculating fractal parameters. We are also including results using gemma-2b as requested by reviewer cohu.

We agree that our work has some limitations, as one cannot answer all questions in a single paper (e.g. we focus on the English language alone). But, we respectfully disagree that the contribution of this work be considered "poor". We would appreciate it if you let us know of any remaining concerns so we can respond to them during the remainder of the discussion period.

Sincerely
I appreciate your detailed response. These are my replies to your rebuttals:

>**The baseline**

> The baseline we use in Figure 1 is used for illustration purposes only, mainly to show how a self-similar process looks like. We do not use random sequences in our analysis. While we appreciate the suggestion of using n-gram models, our work focuses on the self-similar structure of natural language. Note in particular that the output of an n-gram model, by definition, does not have long-range dependence (LRD). So, conducting the same analysis on the output of n-gram models would not yield meaningful insights into the fractal structure of language, as n-gram models cannot generate long-range dependencies.

Yes, your paper does not have an obvious baseline in terms of model comparison. However, to consider how the distribution observed on language models satisfies the characteristics of fractal patterns, you need to compare language models with other simple sequence generation methods like n-gram or probabilistic context-free grammars (PCFGs). Currently, the discussion is only based on the observed absolute values of generated texts by language models. Thus, readers find it difficult to estimate how the observed characteristics fit the proposed theory about fractal patterns.

>**Empirical fit**

> Thank you for raising this point. We do demonstrate how well the power law fits the actual distribution in Figures 2 and 3. We observe a perfect linear fit in a log-log plot over at least two orders of magnitude. In addition, since the power law is of the form , there are only two parameters to estimate ( and ) so AIC is also quite low, given the near perfect fit and the small number of parameters.

Thank you for clearing that point. I also think that the current result fits your assumption well. However, without relative comparison, readers cannot exclude the possibility that fractal is common in sequences that are not restricted to languages. Therefore, similar to my first question. Preparing a simple sequence generation model like n-gram or PCFGs can support the faithfulness of your claim.

>**Evaluation**

> We use three popular and quite diverse benchmarks (BBH, MMLU, GSM8K) and we also try various prompting strategies (e.g. direct and chain-of-thought with 0, 3, 5, 8 shots). These benchmarks are quite diverse and include tasks such as logical deduction, multi-step arithmetic, disambiguation, as well as general knowledge (e.g. history, computer science, law, sports, movies, and dates, etc). In line with your suggestions, they also include translation-related tasks, such as Translation Error Detection in BBH. So, they cover a broad spectrum of tasks. We hope this addresses your concern about the evaluation datasets, and we will clarify this in the revised version of the paper.

I appreciate your detailed explanation of the BBH benchmarks. This explanation helps me understand that BBH includes translation-related tasks, and my concern about that part is now resolved. 

>**Context length**

> The main reason for including the discussion about the context length at training time in our analysis is because our findings suggest one intriguing possibility: that language models might benefit from being trained on long contexts even if short contexts are used at inference time. However, we did not find any evidence yet to support this. We chose to include this in the analysis section because we believe that mentioning negative results would still be very valuable to our community.

The assumption itself is natural. Since the part is not so close to your main subject of the paper, I understand the stance.

>**Presentation**

> Thank you for the great suggestion about the presentation style. We will definitely do that in the revised version of the paper.

I appreciate your generous attitude. I hope many readers will easily understand the paper with the update.

>**Follow-up**

> We are grateful again for your detailed and constructive feedback. If we have satisfactorily answered your questions, we hope you would consider revising your scores. Otherwise, please let us know if there are any other questions or concerns, so we can respond to them during the discussion period.

Since the response actually solves some concerns, I'll raise my score to 5. In addition, if you can make a promise to me about the following requirement, I'll raise the score to 7:

Requirement: The analysis of relative differences between the sequences generated by language models and those by n-gram or PCFG-based models is included in the Appendix. By showing the result, your claim is grounded on the objective evidence. It changes the reliability of your assumption. Since the change is mainly conducted in the Appendix part, I think this is not included in the major revision. Needless to say, you have no need to follow the requirements during the rebuttal period.
Thanks for your response and clarifications. I will maintain my current score.
Thank you for your response. The new results based on gemma-2b look good. I'll keep my score. My original concern about reproducibility is mostly based on the initial evaluations are based on either closed models or the authors' checkpoints that are not publicly released. Adding gemma 2B mostly addressed this concern.
Dear reviewer,

Thank you again for the detailed and constructive comments.

As the discussion period is about to end, we would like to ensure we've addressed all of your questions and concerns. If you feel we have satisfactorily responded, please let us know. If you have any further questions, we are happy to address them before the discussion period closes.

Sincerely
Dear reviewer,

Thank you again for the detailed and constructive comments.

As the discussion period is about to end, we would like to ensure we've addressed all of your questions and concerns. If you feel we have satisfactorily responded, please let us know. If you have any further questions, we are happy to address them before the discussion period closes.

Sincerely
Dear reviewer,

Thank you again for the detailed and constructive comments.

As the discussion period is about to end, we would like to ensure we've addressed all of your questions and concerns. If you feel we have satisfactorily responded, please let us know. Otherwise, please let us know your remaining concerns so we can address them before the discussion period closes.

Sincerely
Dear reviewer,

Thank you again for the detailed and constructive comments.

As the discussion period is about to end, we would like to ensure we've addressed all of your questions and concerns. If you feel we have satisfactorily responded, please let us know. Otherwise, please let us know your remaining concerns so we can address them before the discussion period closes.

Sincerely