ICLR 2025

Published as a conference paper at ICLR 2025
INTELLIGENCE AT THE EDGE OF CHAOS
Shiyang Zhang∗,1,2
Aakash Patel∗,1
Syed Rizvi1
Nianchen Liu 3
Sizhuang He1
Amin Karbasi1
Emanuele Zappala4
David van Dijk†,1
1Yale University, 2Columbia University, 3Northwestern University, 4Idaho State University
ABSTRACT
We explore the emergence of intelligent behavior in artificial systems by inves-
tigating how the complexity of rule-based systems influences the capabilities of
models trained to predict these rules. Our study focuses on elementary cellu-
lar automata (ECA), simple yet powerful one-dimensional systems that generate
behaviors ranging from trivial to highly complex. By training distinct Large Lan-
guage Models (LLMs) on different ECAs, we evaluated the relationship between
the complexity of the data generated by the rules and the models’ ability to learn
effective general representations, as reflected in their performance on downstream
tasks.
Our findings reveal that models trained on more complex data exhibit
greater predictive ability, as demonstrated by their performance on reasoning and
chess move prediction tasks. Both uniform and periodic systems, and often also
highly chaotic systems, resulted in poorer downstream performance, highlighting
a sweet spot of complexity conducive to intelligence. We conjecture that intelli-
gence arises from the ability to predict complexity and that creating intelligence
may require only exposure to complexity.
1
INTRODUCTION
The emergence and nature of intelligence within computational systems have long been subjects of
fascination and rigorous study in the fields of artificial intelligence (AI) and theoretical computation.
Traditional AI methodologies predominantly involve training models on high-quality datasets inher-
ently imbued with human intelligence—such as natural language corpora, expert-annotated datasets,
or data reflecting human cognitive processes (Coleman et al., 2019). This approach operates under
the assumption tha...