Intelligence at the Edge of Chaos

## Abstract

Abstract We explore the emergence of intelligent behavior in artificial systems by investigating how the complexity of rule-based systems influences the capabilities of models trained to predict these rules. Our study focuses on elementary cellular automata (ECA), simple yet powerful one-dimensional systems that generate behaviors ranging from trivial to highly complex. By training distinct Large Language Models (LLMs) on different ECAs, we evaluated the relationship between the complexity of the rules’ behavior and the intelligence exhibited by the LLMs, as reflected in their performance on downstream tasks. Our findings reveal that rules with higher complexity lead to models exhibiting greater intelligence, as demonstrated by their performance on reasoning and chess move prediction tasks. Both uniform and periodic systems, and often also highly chaotic systems, resulted in poorer downstream performance, highlighting a sweet spot of complexity conducive to intelligence. We conjecture that intelligence arises from the ability to predict complexity and that creating intelligence may require only exposure to complexity.

## 1 Introduction

The emergence and nature of intelligence within computational systems have long been subjects of fascination and rigorous study in the fields of artificial intelligence (AI) and theoretical computation. Traditional AI methodologies predominantly involve training models on high-quality datasets inherently imbued with human intelligence—such as natural language corpora, expert-annotated datasets, or data reflecting human cognitive processes (Coleman et al., ). This approach operates under the assumption that creating intelligent behavior necessitates exposure to intelligent data sources. In contrast, this paper explores an alternative hypothesis: that intelligence can emerge from modeling simple systems as long as they exhibit complex behaviors, even when the process that generates the data lacks inherent intelligence.

To investigate this hypothesis, we utilize Stephen Wolfram’s elementary cellular automata (ECA) as our experimental framework. ECAs are one-dimensional, binary-state, discrete computational systems defined by 256 possible 8-bit rules. They generate a diverse spectrum of behaviors ranging from simple, repetitive patterns to highly complex and chaotic structures (Wolfram, ). Despite their simple rule-based definitions, certain ECAs produce patterns of significant complexity, making them ideal for examining the relationship between intelligence and complexity.

Our methodology involves training separate instances of the GPT-2 language model (Radford et al., ) on datasets generated by individual ECAs. The models are tasked with predicting future states of the automata. Following this pretraining phase, we evaluate the models’ “intelligence” by quantifying performance on downstream logical reasoning and chess move prediction tasks.

This paper presents an extensive study exploring the relationship between system complexity and the emergence of intelligence in large language models (LLMs). We discover a positive correlation between the complexity of the ECA rules and the downstream performance of models trained on them, highlighting the role of complexity in fostering intelligent behavior. Surprisingly, we find that models can learn complex solutions when trained on simple rules. Our results point to an optimal complexity level, or “edge of chaos”, conducive to intelligence, where the system is structured yet challenging to predict. These findings enhance our understanding of intelligence in artificial systems and provide a framework for future research focused on the importance of complexity in developing these systems.

## 2 Background

### 2.1 Elementary Cellular Automata

Cellular automata (CAs) are computational models of complex systems, consisting of a grid of cells that evolve over time based on simple rules. They were first introduced by John von Neumann in the 1940s (von Neumann, ). CAs have been widely used to simulate various physical, biological, and computational systems due to their simplicity and ability to produce complex behavior.

Elementary Cellular Automata (ECAs) (Wolfram & Mallinckrodt, ) are a type of one-dimensional cellular automaton where each cell has a binary state, and its next state is determined by a simple rule that depends only on the current state of the cell and its two immediate neighbors. There are 256 possible ECA rules, 88 of which are unique after accounting for symmetries (Castillo-Ramirez & Magaña-Chavez, ). Notable examples include Rule 110, which has been proven to be Turing complete (Cook, ), and Rule 90, which generates the fractal-like Sierpinski triangle. These rules are categorized into four classes based on their behavior when initialized with random conditions: Class I, which evolves to a homogeneous state; Class II, which forms simple periodic structures; Class III, which produces chaotic and aperiodic patterns; and Class IV, which exhibits complex structures (Castillo-Ramirez & Magaña-Chavez, ).

ECAs are valuable computational models used to explore complex systems and emergent behaviors arising from simple rules. Their usefulness lies in their simplicity and the variety of patterns they can produce, making them ideal for studying pattern formation in computation (Meunier, ), physics (Banerjee & Dalui, ), and mathematical biology (Rasolonjanahary & Vasiev, ). Additionally, ECAs have been utilized in cryptography as a basis of certain security frameworks (Corona-Bermúdez et al., ) and in computer science education (Staubitz et al., ) to illustrate concepts in algorithms and computational theory. Their ability to model intricate systems with minimal computational resources has made ECAs a popular tool across scientific disciplines.

### 2.2 Large Language Models

Large language models (LLMs) are advanced artificial intelligence systems designed to understand and generate human-like text based on vast datasets (Brown et al., ). By leveraging deep learning techniques, these models analyze patterns in language to perform tasks such as translation, summarization, and conversational dialogue (Devlin et al., ). Notable examples like OpenAI’s GPT-4 have demonstrated remarkable capabilities in producing coherent and contextually relevant responses across a wide range of topics (Achiam et al., ). The development of LLMs represents a significant advancement in natural language processing, opening up new possibilities for applications in education, research, and industry (Katz et al., ).

Incorporating synthetic data into the training of large language models (LLMs) has been shown to significantly boost their performance. Synthetic data generation enables the creation of large, diverse, and high-quality datasets that may not be readily available in the real world (Anaby-Tavor et al., ). By augmenting training data with synthetic examples, models can learn from a broader range of linguistic patterns and rare occurrences, improving their ability to generalize to unseen scenarios (Edunov, ). As a result, LLMs trained with synthetic data exhibit enhanced robustness, accuracy, and adaptability across various downstream tasks (Feng et al., ).

### 2.3 Complexity Measures

Various complexity measures have been proposed to assess the behavior of dynamical systems. In this work, we employ the following measures:

- 1.
Lempel-Ziv Complexity assesses the compressibility of a sequence by counting the number of unique substrings in the sequence (Lempel & Ziv, ).
- 2.
Compression Complexity quantifies how effectively a sequence can be compressed using a data compression algorithm such as Zlib ().
- 3.
Lyapunov Exponent gauges a system’s sensitivity to initial conditions. Higher Lyapunov exponents indicate that small variations in initial states result in rapidly diverging outcomes. We adopt the method proposed by Wolf () for computing this metric.
- 4.
Krylov Complexity evaluates how information propagates in a system’s Hilbert space, measuring how quickly an operator spans larger regions of the state space over time (Parker et al., ).
- 5.
Wolfram Classification categorizes ECA rules into four categories based on behavior and complexity (see Section 2.1).

While these measures are correlated with one another, each measures different aspects of complexity. For most analyses, we focus on Lempel-Ziv Complexity and the Wolfram Classification. Additional complexity measures are shown for performance on downstream tasks (Section  5.1).

## 3 Methodology

In this study, we systematically investigate the relationship between system complexity and emergent intelligence. This section outlines our methodology, including the steps for data generation and model pretraining. An overview of the training process and task evaluations is provided in Figure 1.

Figure: Figure 1: Our framework for investigating the link between complexity and intelligence. We pretrain Large Language Models (LLMs) on Elementary Cellular Automata (ECAs) from different complexity classes using next-token prediction, then evaluate them on downstream reasoning and chess move prediction tasks. We use various measures to analyze the complexity of ECA rules, and quantify the relationship between complexity and downstream performance.
Refer to caption: extracted/5911414/figures/complexity_fig1_v5.png

### 3.1 Data Generation

To train our models, we simulate a selection of ECA rules. Each simulation generates a sequence of binary vectors, where each vector represents the system’s state at a specific time step. For each sample, we begin with a randomly initialized vector as the automaton’s initial state. The system is then evolved over 1000 time steps by repeatedly applying the chosen ECA rule. This process produces a sequence of binary vectors that capture the evolving dynamics of the ECA over time.

To increase the diversity of the training data, we extract random spatiotemporal windows from the full sequences. Specifically, we sample subsequences by selecting random windows of 60 time steps and 100 spatial dimensions from the binary vectors. This method exposes the model to a variety of contexts and state configurations, enhancing its ability to learn the dynamics of the ECA rules and generalize to new sequences. Each training sequence represents a randomly selected segment in space and time of the automaton’s evolution. We train models predicting either 1 or 5 steps in the future, to vary the difficulty of the task.

### 3.2 Training Procedure for GPT-2 Models

We utilized a modified GPT-2 architecture (Radford et al., ) adapted for binary input and output data, enabling it to perform next-token prediction on sequences of binary states. Instead of using a traditional token embedding layer followed by a softmax over a vocabulary, we replaced the token embeddings with a linear projection layer that directly maps binary vectors into the model’s embedding space. The GPT-2 model processes these embeddings to capture temporal dependencies and patterns within the sequences. At the output, we apply a linear projection layer to map the model’s hidden states back to the data dimensionality, generating the prediction for the next state of binary variables at each time step. This adaptation allows the GPT-2 model to handle binary data directly and perform next-token prediction without relying on a predefined vocabulary. This also makes the model deterministic, in line with the deterministic nature of ECAs.

### 3.3 Pretraining Setup

Each model was pretrained on next-token prediction using data generated from a single ECA rule for up to 10,000 epochs. Early stopping based on validation loss was utilized to prevent overfitting and conserve computational resources. The training data were organized into batches of 64 sequences, each comprising 60 time steps and 100 spatial dimensions.

We employed the Adam optimizer with an initial learning rate $\eta=2\times 10^{-6}$and a weight decay of 0.01. A learning rate scheduler with a linear warm-up over the first 10% of the total steps was implemented to stabilize the initial stages of training and improve convergence rates. After the warm-up phase, we applied cosine annealing to gradually decay the learning rate over the remaining training steps. Gradient accumulation was used to handle larger effective batch sizes within the constraints of GPU memory, allowing us to simulate larger batch sizes by accumulating gradients over multiple mini-batches. To prevent exploding gradients, we applied gradient clipping with a maximum norm of 1.0.

## 4 Experiments

To evaluate the emergent intelligence of models trained on cellular automata, we conducted experiments on three downstream tasks: one easy and one hard reasoning task inspired by the Abstraction and Reasoning Corpus (ARC) (Chollet, ), and a challenging chess move prediction task (Ruoss et al., ). These tasks were designed to quantify the models’ abilities in reasoning, abstraction, and long-term prediction, thereby assessing the level of intelligence encoded during pretraining on ECA rules of varying complexities. We freeze the layers of the pretrained GPT-2 models and train only the input and output projection layers for downstream tasks to ensure that performance differences reflect the inherent capabilities of the models.

Our central hypothesis is that models pretrained on complex rules will exhibit superior performance on downstream tasks compared to those pretrained on simple rules. The inclusion of both easy and hard tasks allows us to observe different performance trends and better understand the relationship between pretraining complexity, task difficulty, and emergent intelligence. The chess move prediction task, in particular, serves as an excellent system to test reasoning due to its inherent complexity and requirement for strategic thinking.

### 4.1 Downstream Task: Reasoning

We developed a downstream task inspired by the ARC (Chollet, ) to evaluate models’ problem-solving and reasoning abilities. Our approach utilizes sequence completion problems that require the model to infer transformation rules from provided examples and apply them to novel scenarios.

The data consists of a fixed number of shapes on a grid. At each time step, any of the following transformations can be applied to each shape: changing the color, rotating the shape by $90^{\circ}$, or shifting the shapes by one position. We designed two versions of the reasoning task, differing in complexity based on the number and type of transformations applied.

#### Easy

In the easy task, we only use $3\times 3$ squares that are fixed in position and orientation. The only possible transformation is a color change, which we perform in a predetermined order.

#### Hard

The hard task involves more complex patterns where all transformations are applied simultaneously. We used four distinct base shapes, each represented by a $5\times 5$ matrix. This results in complex sequences where shapes change color, orientation, and position over time. The combination of these transformations requires the model to reason over multiple simultaneous changes to accurately predict the next pattern in the sequence.

#### Data Preparation and Training on the Downstream Tasks

For the easy reasoning task, models were trained for 1,000 epochs with a learning rate of $1\times 10^{-4}$. For the hard reasoning task, due to its increased complexity and difficulty, we post trained for 10,000 epochs with a learning rate of $1\times 10^{-5}$. Early stopping based on validation loss was again used to prevent overfitting.

### 4.2 Downstream Task: Chess Move Prediction

For the chess experiment, we evaluated the capability of the different ECA-pretrained models to predict next moves in chess games represented using Standard Algebraic Notation (SAN) (). We use chess games from the Lichess Elite database (), focusing on games played between January and April 2016 by Grandmasters with ratings of 2200 and above. Each game was represented as a sequence of SAN moves. We split this collection into training, validation, and test sets using an 80-10-10 split to facilitate model training and evaluation. Each game sequence was segmented into subsequences of 60 moves each, and any subsequence shorter than this length was padded sequences to length 60.

We added an embedding layer to convert the SAN tokens into vector representations, which were then processed using the frozen ECA-pretrained model. A linear output layer was used to transform the outputs to the vocabulary size corresponding to the SAN tokens. These input and output layers were trained while the rest of the model was frozen. The model was trained using cross-entropy loss, the Adam optimizer (Kingma, ), and a learning rate scheduler with warm-up. Early stopping was also implemented.

### 4.3 Hardware and Software

The experiments were conducted using PyTorch version 2.1.2 and the Transformers library (version 4.41.0), with CUDA version 12.4 for GPU acceleration. The models were trained on 12 NVIDIA H100 GPUs, each with 80 GB of memory, running on Red Hat Enterprise Linux 8.8.

Figure: Figure 2: Relationship between downstream task performance and rule complexity. Left: Eight representative ECA rules, two from each of Wolfram’s four complexity classes. Performance of models trained on these rules is highlighted in the top row. Top: Model performance in relation to the Lempel-Ziv complexity of each rule. The left and center panels show efficiency (1 divided by number of epochs to reach 80% validation accuracy) for the easy and hard reasoning tasks, respectively. The right panel shows move prediction accuracy for the chess task. The rules depicted on the left are highlighted in the plot with triangles and annotated with the rule number. The correlation coefficient is shown in the top-left corner of each plot. An asterisk next to the value indicates a significant relationship ($p<0.05$). Bottom: Downstream task performance based on Wolfram classification of each rule. Models trained on chaotic and complex rules perform better than models trained on uniform and simple rules.
Refer to caption: extracted/5911414/figures/complexity_vs_perf/complexity_vs_perf.png

## 5 Results

In this section, we present our results exploring the relationship between system complexity and emergent intelligence in LLMs. The following sections detail our analyses of task performance and attention patterns across models trained on varying rule complexities.

### 5.1 Relationship between Intelligence and Complexity

Figure: Figure 3: Relationship between downstream task performance and rule complexity for other complexity measures. Rows depict easy reasoning, hard reasoning, and chess move prediction tasks, while columns show compression complexity, Lyapunov exponent, and Krylov complexity, respectively. We observe the same general patterns that we see with the Lempel-Ziv complexity in Figure 2.
Refer to caption: x1.png

Figure 2 presents the model performance across three downstream tasks (easy reasoning, hard reasoning, and chess move prediction) as a function of the complexity of the ECA rules the models were pretrained on. The top row highlights the relationship between performance and the Lempel-Ziv complexity of the rules, while the bottom row categorizes the performance by Wolfram’s complexity classes. For clarity, two representative rules from each complexity class are displayed on the left, with their corresponding performance annotated in the top plots.

For the reasoning tasks, models generally achieve near-perfect accuracy when trained for sufficient long time. Therefore, instead of reporting absolute accuracy, we focus on model efficiency, defined as the inverse of the number of epochs required to reach 80% accuracy. The chess task is sufficiently difficult that models do not achieve perfect performance, and so we report the final accuracy. As rule complexity increases, we observe a clear positive correlation in all tasks, with more complex rules leading to greater efficiency. This correlation is significant for each of the tasks ($p<0.05$).

In terms of Wolfram’s classification, rules from Classes I and II (uniform and periodic) show lower average efficiency in the reasoning tasks compared to those from Classes III and IV (chaotic and complex). Complex rules especially outperform the other classes on the chess move prediction task. This pattern suggests that models trained on more complex rules tend to perform better on harder downstream tasks. Results with respect to other complexity measures are shown in Figure 3.

We observe that models trained on certain Class III (Chaotic) rules, such as Rules 105, 146, and 150, have poorer performance on the hard reasoning and chess move prediction tasks. This behavior is expected due to chaotic systems lacking the structured patterns necessary for effective learning. In other words, they may be too random to predict, leading to weaker downstream performance. These results highlight the existence of a “sweet spot” of complexity conducive to intelligence, where the system is still predictable yet hard to predict.

### 5.2 Models Learn Complex Solutions For Simple Rules

Figure: Figure 4: Attention scores for the final 10 states prior to the target state, showing that models trained on more complex rules rely more heavily on past states for prediction. Left: Visualization of the last 10 states and the target state for representative rules from each of Wolfram’s complexity classes. Center: Attention scores for each of the last 10 states, highlighting that models trained on complex and chaotic rules focus more on recent states, while models trained on uniform rules exhibit consistently low attention. Periodic rules demonstrate a repeating attention pattern, suggesting that the model is learning to attend to earlier cycles of the same state rather than general state history. Right: Average attention across the final 10 states for all rules, plotted against Lempel-Ziv complexity. A strong positive correlation (r=0.66) indicates that models trained on higher complexity rules attend more highly to historical states during prediction.
Refer to caption: x2.png

The elementary cellular automata (ECA) are inherently memoryless, meaning the state at the next time point is determined only by the current time point, without any consideration of past states. For each model, a straightforward solution exists: simply learning the 8-bit ECA rule and applying it to the current state to predict the next state. However, alternative solutions may also be possible, where the model leverages historical states for its predictions. The key question is whether the model is merely learning the trivial solution or if it is integrating information from the state history.

To explore this, we analyze the self-attention scores with respect to the last state in the input sequence, which the model uses to predict the next state (see Section 3.2). Specifically, we examine the attention values corresponding to the final ten states before the target state. Figure 4 illustrates the average attention across all layers and heads for the different ECA rules, as well as the attention at each of the last 10 states for one rule from each complexity class.

Our findings reveal that models trained on rules that produce more complex dynamics tend to allocate higher attention to the last ten states, with a strong positive correlation (r=0.66) between rule complexity and average attention. This suggests that models trained on complex rules integrate information from past states to make their predictions.
In contrast, models trained on simpler rules display lower attention across the last ten states. For example, Rule 168 (uniform) shows consistently low attention, indicating that previous states are not being utilized in the prediction process. Rule 179 (periodic) demonstrates a recurring pattern in its attention scores, where the model focuses on every other state. This behavior is expected by the nature of Rule 179, which produces an alternating pattern that repeats every two time steps (Figure 4). Thus, the model appears to be learning only this alternating cycle, rather than general state history.
The simpler attention structure suggests that these models learn a trivial solution, which is in line with their downstream performance.

We emphasize that for both simple and complex rules, a trivial solution exists: learning the 8-bit instantaneous rule and applying it to only the current state. Such a model would have no attention on previous states, as it only needs the current state to make its prediction. The fact that the complex models are attending to previous states indicate that they are learning a more complex solution to this simple problem, and we conjecture that this complexity is what makes the model “intelligent” and capable of repurposing learned reasoning to downstream tasks.

Figure: Figure 5: Performance comparison between models trained on short-term and long-term prediction tasks. Each point represents an ECA rule, with the x-axis showing the performance of models trained on short-term (1-step) prediction and the y-axis showing performance of models trained on long-term (5-step) prediction. The points are colored by the Lempel-Ziv complexity of each rule. The dashed line represents equal performance between the two models. Points below the line indicate better performance for short-term prediction models.
Refer to caption: x3.png

We had initially expected that predicting one step in the future would be too easy, and would result in every model learning the trivial (8-bit rule) solution. As such, we trained models to predict 5 steps ahead. Surprisingly, we found that models predicting only the next step not only learned non-trivial solutions, but even outperformed models predicting five steps ahead (Figure 5). This result suggests that even when predicting the immediate next state, the models are learning nontrivial solutions (provided that the underlying rule is complex), capable of capturing complex patterns beyond the trivial rule-based approach.

In this section, we explored the relationship between system complexity and emergent intelligence. Our results show that downstream model performance improves with pretraining on more complex rules, but can deteriorate with excessive complexity or chaotic behavior, signifying a sweet spot of complexity conducive to intelligence. Attention patterns further reveal that models trained on complex rules integrate historical information into their predictions, suggesting that they are learning more sophisticated solutions to relatively simple problems. We hypothesize that this complexity in the learned representations is a key factor enabling models to generalize and perform well on downstream tasks.

## 6 Related Work

#### Elementary Cellular Automata and Complexity

Elementary Cellular Automata (ECA) serve as foundational models for exploring complexity arising from simple rules. Early research demonstrated that even minimalistic CA systems, governed by basic and local interaction rules, could generate intricate patterns over time (Wolfram, ). Wolfram’s extensive investigations into one-dimensional CA revealed that certain rules produce behaviors ranging from stable and periodic to chaotic and complex (Wolfram, ). In his work Cellular Automata and Complexity, Wolfram classified one-dimensional CA into four classes based on their dynamic behaviors, highlighting the rich complexity that simple systems can exhibit (Wolfram & Mallinckrodt, ).

#### Computation at the Edge of Chaos

The concept of computation at the “edge of chaos” suggests that systems poised between order and disorder exhibit maximal computational capabilities and complex behavior. Langton () introduced this idea, demonstrating that cellular automata operating at this critical transition can perform complex computations. Packard () explored how systems adapt toward the edge of chaos, suggesting that evolution may favor systems that balance between stability and chaos. Mitchell et al. () investigated evolving cellular automata to perform computations, finding that rules near the edge of chaos are more capable of complex tasks. These studies provide a theoretical foundation for our work, as we observe that models trained on rules with higher complexity exhibit greater intelligence in downstream tasks.

#### Emergence of Intelligence Through Complexity Exposure

The hypothesis that intelligence can arise from exposure to complexity is supported by studies in artificial life and complexity science. Bedau () discusses how complex behaviors and adaptation emerge from simple rules in artificial life systems. Langton () introduced the concept of ”computation at the edge of chaos,” proposing that systems poised between order and disorder exhibit maximal computational capabilities and complex behavior. Our conjecture that creating intelligence may require only exposure to complexity aligns with these perspectives.’ Crutchfield & Mitchell () explored how evolutionary processes can lead to emergent computation in cellular automata, demonstrating that complexity in the environment can drive the evolution of computational abilities. Kauffman () discussed self-organization and complexity in biological systems, suggesting that complex interactions can lead to emergent properties like intelligence.

#### Emergent Abilities in Large Language Models

Recent advancements in large language models (LLMs) have shown that not only increasing model size but also exposing models to more complex and diverse data can lead to the emergence of new capabilities not present in smaller models or models trained on simpler data. Wei et al. () discuss emergent abilities in LLMs, highlighting that certain reasoning tasks become solvable only when models reach a certain scale and are trained on sufficiently complex data. Brown () demonstrate that LLMs like GPT-3 can perform few-shot learning, indicating that exposure to a wide range of linguistic contexts and complexities enhances the models’ adaptability and understanding. Hoffmann et al. () emphasize the importance of data scaling laws, showing that increasing the amount and complexity of training data can lead to better performance than merely increasing model size, suggesting a trade-off between model capacity and data complexity. Kaplan et al. () introduce scaling laws for neural language models, illustrating how performance improves predictably with model size, dataset size, and computational resources, but also noting that the nature of the data plays a crucial role.

## 7 Discussion

In this work, we utilize LLMs trained on elementary cellular automata (ECA) to study how intelligent behavior may emerge in large language models (LLMs) when trained on increasingly complex systems. Our findings reveal several important trends that contribute to understanding the relationship between complexity and model behavior.

#### Optimal Complexity: The “Edge of Chaos”

We observe that the best model performance occurs in systems operating at high but not excessive complexity, previously referred to as the “edge of chaos” (Langton, ). Models trained on Class IV ECA rules, which exhibit structured yet complex behaviors, perform optimally, suggesting that intelligence may emerge in systems that balance predictability and complexity. On the one hand, if a system is too simple and predictable, like those governed by Class I and II rules, the model quickly learns a trivial solution and fails to develop more sophisticated reasoning. On the other hand, highly chaotic rules from Class III provide too much randomness, akin to training on noise, where the lack of meaningful patterns prevents the model from finding useful structure. The sweet spot arises when complexity is high enough to challenge the model but still retains underlying patterns that the model can exploit. This balance between order and randomness seems particularly conducive to fostering intelligent behavior, as it forces the model to develop more effective reasoning and processing strategies.

#### Complex Solutions for Irreducible Systems

Our findings demonstrate that large models are capable of learning complex, non-trivial solutions even when simpler, more trivial ones are available, likely because they are overparametrized. For lower-complexity ECA rules, the models often adopt a trivial solution, focusing only on the current state since no history is needed. However, when exposed to more complex rules, models tend to leverage prior states, as indicated by the attention patterns in Figure 4. Despite the memoryless nature of ECA systems, overparameterized models explore a broader search space, and solutions that integrate historical information may be more robust. We hypothesize that by learning to incorporate past states, the model develops generalizable logic that can be reused across tasks. In contrast, a model relying solely on a trivial, state-specific rule would struggle to transfer its knowledge to more complex downstream tasks. Thus, the ability to learn from past states may be key to the model’s success in adapting to diverse problems.

Certain ECA rules, such as Rule 110, are known to be computationally *irreducible*, meaning their behavior cannot be predicted without directly computing each step (Wolfram, ). However, some studies suggest that even in these systems, partial predictability can be achieved under certain conditions (Israeli & Goldenfeld, ). This implies that models learning more complex, non-trivial solutions can actually outperform simpler, irreducible approaches by leveraging approximate but efficient predictions. Rather than directly calculating each state, models can explore patterns and generalize from past states, potentially leading to solutions that are not only more robust across tasks but also more efficient than the irreducible solution.

#### Broader Impact

Our findings connect to a larger body of work on the emergence of intelligence in large language models (LLMs). Understanding how LLMs develop sophisticated reasoning capabilities when trained on relatively simple data could offer new insights into why and how intelligence emerges in these models. This research may help shed light on some of the open questions surrounding LLMs, particularly how their internal representations evolve and how certain training conditions lead to more transferrable reasoning abilities.

In future work, this framework can be further explored by training larger LLMs on synthetic data generated by simple rule-based systems. Incorporating measures of complexity, such as those used in this study, could provide a valuable tool for prioritizing and curating data, ensuring that models are exposed to information with the right balance of structure and randomness. This aligns with recent advances in data curation, where data quality and complexity, rather than quantity, is increasingly emphasized in improving model performance (Zhao et al., ; Cao et al., ; Liu et al., ).

Additionally, this work may have implications for our understanding of human intelligence, which is proposed to have evolved as a mechanism for interacting with a complex and hard-to-predict world (Euler, ). The idea that intelligence arises in systems operating at the “edge of chaos” parallels cognitive science theories suggesting that human brains function at a critical state between different dynamics (Cocchi et al., ; Hesse & Gross, ; O’Byrne & Jerbi, ). By exploring the conditions under which LLMs develop intelligence, we may gain new insights into the fundamental processes that underlie both artificial and human cognition.

## Appendix A Visualizing Learned Representations Using CKA Similarities

Figure: Figure 6: UMAP projection of CKA similarities between models trained on ECA rules. Each point represents a model, colored by its corresponding rule complexity (Lempel-Ziv, Compression, Lyapunov, and Krylov complexities). Models trained on rules with similar complexities cluster together, indicating that they learn similar representations. Chaotic rules like Rule 105 and 150 are notably closer to models trained on lower complexity rules, while models trained on complex, non-chaotic rules cluster more distinctly.
Refer to caption: x4.png

Figure: Figure 7: UMAP projection of CKA similarities between models trained on ECA rules, annotated with the rule number.
Refer to caption: x5.png

To investigate the learned representations of the pre-trained models, we use center kernel alignment (CKA) to compute similarities between each pair of models. CKA is a method for comparing representations learned by neural networks, measuring how well the internal weight matrices of two models align (Kornblith et al., ). In our analysis, we compute pairwise similarities across all models trained on different ECA rules, with complexity levels ranging from low to high.

To visualize these similarities, we project the CKA similarity embeddings into two dimensions using UMAP, a popular dimensionality reduction technique. The resulting plots, shown in Figures 6 and 7, reveal that models trained on rules with similar complexity tend to cluster together, indicating that they have learned similar internal representations.

Interestingly, models trained on highly chaotic rules, such as Rule 105 and Rule 150, are located closer to models trained on lower-complexity rules, suggesting that despite the high complexity of their training data, their learned representations are more akin to those trained on simpler dynamics. In contrast, models trained on complex but not chaotic rules tend to form distinct clusters, indicating that they develop more unique internal structures compared to models trained on either simple or highly chaotic rules.

These results suggest that complexity plays a significant role in shaping model representations. Models trained on rules with moderate complexity tend to develop distinct, well-formed internal representations, while chaotic rules may push models toward more generic representations, aligning them with simpler rules.

## Appendix B Long vs Short Term Prediction with other complexity measures

Figure: Figure 8: Efficiency comparison between models trained on short-term and long-term prediction tasks, colored by the other complexity measures. We observe the same trend as with Lempel-Ziv complexity in Figure 5.
Refer to caption: x6.png

Figure 8 presents an efficiency comparison between models trained on short-term (1-step) and long-term (5-step) prediction tasks akin to Figure 5, but with the remaining complexity measures. We observe the same trend as with the Lempel-Ziv complexity. Models tend to perform better when trained on shorter prediction horizons, challenging the assumption that longer prediction horizons lead to better understanding of underlying structures.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: In my view the paper is quite interesting and a good fit for ICLR, and would like see it accepted if some of its weaknesses are addressed. Some questions:

- (see above) What role does the temporal structure of CA sequences play in learning generalisable representations?
- Is the amount of data required from CA pretraining comparable to the amount of data required using natural langage (or similar complex data), to reach similar generalisation effects?
- what's the trade-off between complexity and volume of data
soundness: 3
strengths: I like about this paper that it uses a relatively simple approach to create data with varying (and quantifyiable) degrees of complexity. The approach could help in exposing LLMs to large amounts of data, potentially leading to smaller requireemtns of corpora during pretraining. The "edge-of-chaos" idea has been investigated in neural networks before, but not as a result of pretraining with data from CAs.

The approach of pretraining on CAs and finetuning the outputs later seems like a great fit to learning representations / ICLR - an interesting way to derive good features for training models.
summary: The paper explores whether LLMs can develop generalisable reasoning abilities by pretraining on structured but simple datasets, specifically sequences generated by cellular automata. The authors hypothesise that exposing models to complex patterns, even if the underlying system is deterministic and rule-based, could improve the models’ performance on reasoning tasks. By pretraining LLMs on cellular automata generated data, they test whether this training improves performance on abstract reasoning benchmarks, such as tasks from the Abstraction and Reasoning Corpus and chess move prediction (after finetuning).

The results show that models exposed to complex patterns, those generated by "edge-of-chaos" traces, perform better on these reasoning tasks compared to models pretrained on simpler or more chaotic traces. This suggests that pretraining on richer, more complex patterns helps LLMs develop broader generalisation abilities, even when the training data lacks semantic content.
weaknesses: While I like the idea and the approach of the paper, the choice of words could be improved and language be more precise. The choice of "Intelligence" in the title and throughout the paper is awkward, the work doesn’t necessarily show "intelligence" in the sense of true reasoning or understanding: it demonstrates however that structured complexity in data can lead to the development of generalisable features or representations, that is a better choice would be to describe the approach as learning useful features or representations for predictions or other complex tasks.

The language around the CA rules is also imprecise - all CA rules are similarly simple, but some of the rules lead to complex (or chaotic, or simple...) outputs / traces. It is the generated data that exhibits the complex patterns in case of the class iv CAs. 

As an ablation, it might also be interesting to explore if it is the *structured* complexity that leads to better features, or if complex patterns are sufficient. This could be achieved by using surrogate outputs from CAs (ie scramble the temporal order), to allow the distinction between the complex (spatial) pattern or the structured, sequential nature of the data.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Could you comment on the differences between "complexity" and "diversity" in ECA model? Is any of the complexity measures you incorporate captures diversity of the data as well? Are complex data always more diverse and vice-versa? 

-  In figure 2 violin plot, it seems like it's not only mean values but also variances that are correlated to the complexity measure and downstream tasks which I found interesting.  Could the authors provide comment?

- Related to the weaknesses I wrote above, I am curious about baseline performance of the models before pre-trained on ECA data. 

- I see different tendency on Krylov complexity compare to the other on fig 3 i.e. high variance on both very low and very high complexity measure. Is there any reason that you could think of?
soundness: 2
strengths: - I really enjoyed reading the paper. The authors explore the impact of rule complexity of the training data on the model's downstream task performance. I believe it is important and timely subject as there are already many literatures suggesting the importance of data property on acquired intelligence of the neural network models. 
- The paper is well motivated and the authors made good connection to the related previous works on complexity and computational capabilities.
- Particularly, I liked the ECA data model that the authors use, which provides a good control of complexity of the rule-based dynamical system. Also, the authors used multiple measures of complexity rather than single definition. 
- The paper is well-written, it was easy to follow and understand the key claims that the authors make.
summary: The authors explore how the complexity of the rule of the data generative system influences the model capabilities on the downstream tasks. They use elementary cellular automata (ECA) as a rule-based dynamic system to control complexity of the training data trained GPT-2 derived transformer architecture and evaluated the trained model on reasoning/chess move prediction tasks. They found that higher but not too chaotic complex system leads to higher downstream task performance. Also, they found that the model trained with more complex system tends to obtain more complex solution rather than a simple rule, which the authors hypothesized as a reason for better downstream task performance.
weaknesses: - While I am fairly convinced with the motivation and general results, but the presented quantitative result is rather weak in the results and figures they provide. Is the difference between accuracy across complexity, for example, in figure 2, shows fairly low r value.  
For wolfram's complexity class categorization, does the p-test shows significance? The absolute accuracy difference between the categories look fairly low. And the baseline performance without the ECA pretraining is not provided. 

- Did the authors run experiments with multiple seeds?

- The claimed 'emergent' intelligence is only evaluated on the downstream tasks after post-tuning, and in limited kinds. Also, the baseline performance without the pre-training on ECA is not given. 

- While authors found that the model trained with complex system tends to obtain more complex solution than the actual rule and made a interesting conjecture that this might be the reason for better downstream performance, no rigorous experiment to further investigate this point is provided. 

Further questions and comments are given in the next section.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could you include more downstream tasks, like nim games or cylinder chess, to expand the evaluation?

2. Could you test on smaller models, such as ResNet-50, BERT, or Mamba-170m, to validate the results?
soundness: 3
strengths: The paper’s topic is innovative, as I believe the emergence of intelligence is fostered by increasingly large architectures capable of handling complex data. The authors devote significant attention to explaining the concept of the "Edge of Chaos," ensuring a thorough understanding of this idea. Their explanation of the results is also convincing, and the paper’s visual illustrations are highly effective.
summary: The authors of this paper utilize a large language model (GPT-2) to demonstrate "Intelligence at the Edge of Chaos," showing that models achieve optimal performance in moderately complex, but not overly chaotic, systems. They train on tasks based on elementary cellular automata (ECA) and evaluate on various downstream tasks to explore the link between system complexity and emergent intelligence. The results align well with their hypothesis, though additional experiments could further strengthen these findings.
weaknesses: The experiments are somewhat limited. Adding more downstream tasks, such as nim games or toroidal chess, could enhance the findings. Additionally, testing the hypothesis on smaller models could provide further support, rather than relying solely on results from larger models.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How does this relate to Reservoir Computing (RC)? A randomly initialized RC can also perform well on nonlinear and chaotic tasks. How do you distinguish between the sources of good performance—complexity versus reservoir effects?
2. Beyond the trivial explanation mentioned above, what other reasons might explain why the transformer learns to use attention? Have you experimented with varying model sizes? It's possible that more powerful models might not require attention, while smaller models (or those trained with high L2 regularization) might need attention to allocate more computational resources.
3. Does this relate to the principle of computational equivalence?
soundness: 2
strengths: This paper proposes a novel approach to studying connections between complexity and intelligence. By pre-training transformers on complex patterns, it aligns with large language model (LLM) studies, which have demonstrated that high-quality training data leads to better performance on testing tasks. This study conducts thorough experiments across various tasks, examining the topic from different perspectives. It reveals significant performance differences in transformers trained on different cellular automata.
summary: This paper explores a potential connection between complexity and intelligence using data from elementary cellular automata (ECA) — a class of discrete dynamical systems that exhibit simple, complex, and chaotic dynamics. The authors find that transformers trained on complex and chaotic dynamics perform better on downstream tasks. They first train transformers on these dynamical systems, then adapt the pre-trained transformer to other tasks while freezing the transformer weights and only training the encoder/decoder layers.
weaknesses: The most critical issue in this paper is the method of generating training data. As mentioned in lines 146-152, you first generate a large spatiotemporal evolution, then extract only a small spatiotemporal window from it. This approach inadvertently introduces randomness into the evolution because information outside the spatial window is lost, yet its effects continue to diffuse into the observed space. The authors appear to overlook this randomness, claiming the system is deterministic. They then use a linear encoder/decoder to map states into hidden representations and predict the next state from these hidden states.

This mismatch may lead to several problems, potentially explaining some of the results observed in your paper:

1. Your deterministic model may not be strong enough to learn highly random patterns;
2. The attention pattern discovered in Figure 4 might be caused by this mismatch: For simple tasks, external unknown information may not diffuse into the space, so the cut windows contain truly deterministic dynamics, hence the transformer doesn't need to have attention on previous states. However, for complex patterns, information from states outside the window is necessary for prediction. Consequently, the transformers use previous states for better prediction, resulting in the observed attention patterns;
3. This could lead to a trivial, yet unfalsified explanation of your discovery: the reason transformers trained on complex patterns perform well on downstream tasks might be because they have more attention. And the reason they have more attention might be caused by the unknown information outside the spatial window.

To address this issue, you could predict only part of the next state. For example, given state indices 1-60 as input, you could predict only states 2-59, thereby mitigating the problem. I’m wondering if you still observing the results when removing such randomness.

Some other minor issues:

1. The models should not been named as large language model (LLM) — they are not large, and not natural language. I would suggest using transformer instead;
2. Around line 158, please define “binary vector”. Is it a vector of spatial pattern at given time?
3. In line 232, why there isn’t rule 110 in ARC Easy / Chess Move? It is Turning complete, may need more attention on it.
4. It will be good to compare to a randomly initialized transformer. Would complex ECAs beat random transformer?

Meta-review: This paper investigates whether transformers, in particular GPT-2, can learn to predict simple cellular automata, and also how they do it. This is an unexpected and quite interesting topic, and the investigation is well executed. The uncommon topic/perspective makes it worthy of spotlighting, IMHO. There are no particular weaknesses.

Decision: Accept (Poster)