Learning Universal Predictors
Authors: name: Jordi Grau-Moya, name: Tim Genewein, name: Marcus Hutter, name: Laurent Orseau, name: Grégoire Delétang, name: Elliot Catt, name: Anian Ruoss, name: Li Kevin Wenliang, name: Christopher Mattern, name: Matthew Aitchison, name: Joel Veness

Meta-learning has emerged as a powerful approach to train neural networks to
learn new tasks quickly from limited data. Broad exposure to different tasks
leads to versatile representations enabling general problem solving. But, what
are the limits of meta-learning? In this work, we explore the potential of
amortizing the most powerful universal predictor, namely Solomonoff Induction
(SI), into neural networks via leveraging meta-learning to its limits. We use
Universal Turing Machines (UTMs) to generate training data used to expose
networks to a broad range of patterns. We provide theoretical analysis of the
UTM data generation processes and meta-training protocols. We conduct
comprehensive experiments with neural architectures (e.g. LSTMs, Transformers)
and algorithmic data generators of varying complexity and universality. Our
results suggest that UTM data is a valuable resource for meta-learning, and
that it can be used to train neural networks capable of learning universal
prediction strategies.