Title: Benchmarking World-Model Learning
Primary area: datasets and benchmarks
Keywords: world model learning, benchmark, representation-agnostic evaluation, reward-free learning

Abstract: Model-learning agents should gather information to learn world models that support many downstream tasks and inferences, such as predicting unobserved states, estimating near- and far-term consequences of actions, planning action sequences, and detecting changes in dynamics. Current methods for learning and evaluating world models diverge from this goal: training and evaluation are anchored to next-frame prediction, and success is scored by reward maximization in the same environment. We propose _WorldTest_, a protocol for evaluating model-learning agents that separates reward-free interaction from a scored test phase in a different, related
environment. WorldTest is open-ended—models should support many different tasks unknown ahead of time—and agnostic to model representation, allowing comparison across approaches. We instantiated WorldTest with _AutumnBench_, a suite of 43 interactive grid-world environments and 129 tasks across three families: masked-frame prediction, planning, and predicting changes to the causal dynamics. We compared 517 human participants and three frontier models on AutumnBench. We found that humans outperform the models, and scaling compute improves performance only in some environments but not others. WorldTest provides a novel template—reward-free exploration, derived tests, and behavior-based scoring—for evaluating what agents learn about environment dynamics, and AutumnBench exposes significant headroom in world-model learning.