Title: Origins and roles of world representations in neural networks
Primary area: interpretability and explainable AI
Keywords: World Representation, Fine-tuning, Generalization, Transformers

Abstract: While neural representations have been extensively studied in large practical models, the controlled conditions that govern their emergence and their downstream role in model adaptation remain poorly understood. We introduce a World–Data–Model framework that separates the underlying world (fixed city coordinates) from data generation (seven geometric tasks) and model training, enabling causal study of how tasks shape representations and how those representations mediate fine-tuning. We show that training on single tasks yields divergent representational geometries despite an identical underlying world, whereas multi-task training drives alignment, providing controlled evidence for the Platonic Representation Hypothesis. Building on this, we investigate adaptability by adding 100 synthetic “Atlantis” cities and fine-tuning on subsets of tasks. Surprisingly, representational divergence measured in single-task pretraining predicts downstream failure: tasks with low representational alignment not only fail to support cross-task generalization but actively hinder it by isolating new entities from the shared manifold. Overall, our framework establishes a model system for holistically studying the origins, alignment, and adaptability of neural world representations.

TL;DR: We introduce a synthetic setup to study how emergent world representations affect generalization.