Combinatorial Creativity: A New Frontier in Generalization Abilities
Authors: name: Samuel Schapiro, name: Sumuk Shashidhar, name: Alexi Gladstone, name: Jonah Black, name: Royce Moon, name: Dilek Hakkani-Tur, name: Lav R. Varshney
Organization: Spiral Works

Artificial intelligence (AI) systems, and Large Language Models (LLMs) in
particular, are increasingly employed for creative tasks like scientific idea
generation, constituting a form of generalization from training data
unaddressed by existing conceptual frameworks. Despite its similarities to
compositional generalization (CG), combinatorial creativity (CC) is an
open-ended ability. Instead of evaluating for accuracy or correctness against
fixed targets, which would contradict the open-ended nature of CC, we propose a
theoretical framework and algorithmic task for evaluating outputs by their
degrees of novelty and utility. From here, we make several important empirical
contributions: (1) We obtain the first insights into the scaling behavior of
creativity for LLMs. (2) We discover that, for fixed compute budgets, there
exist optimal model depths and widths for creative ability. (3) We find that
the ideation-execution gap, whereby LLMs excel at generating novel scientific
ideas but struggle to ensure their practical feasibility, may be explained by a
more fundamental novelty-utility tradeoff characteristic of creativity
algorithms in general. Importantly, this tradeoff remains persistent even at
scale, casting doubt on the long-term creative potential of LLMs in their
current form. Together, our conceptual framework and empirical findings provide
a foundation for understanding and improving creativity in modern AI models,
bridging the gap between human and machine intelligence.