NeurIPS 2025

On the creation of narrow AI:
hierarchy and nonlocality of neural network skills
Eric J. Michaud1,3∗
Asher Parker-Sartori2
Max Tegmark1,3
1 Department of Physics, Massachusetts Institute of Technology
2 Department of EECS, Massachusetts Institute of Technology
3 The NSF AI Institute for Artificial Intelligence and Fundamental Interactions
Abstract
We study the problem of creating strong, yet narrow, AI systems. While recent
AI progress has been driven by the training of large general-purpose foundation
models, the creation of smaller models specialized for narrow domains could be
valuable for both efficiency and safety. In this work, we explore two challenges
involved in creating narrow AI systems, having to do with basic properties of how
neural networks learn and structure their representations. The first challenge regards
when it is possible to train narrow models from scratch. Through experiments
on a synthetic task, we find that it is sometimes necessary to train networks on
a wide distribution of data to learn certain narrow skills within that distribution.
This effect arises when skills depend on each other hierarchically, and training on a
broad distribution introduces a curriculum which substantially accelerates learning.
The second challenge regards how to transfer particular skills from large general
models into small specialized models. We find that model skills are often not
perfectly localized to a particular set of prunable components. However, we find
that methods based on pruning can still outperform distillation. We investigate the
use of a regularization objective to align desired skills with prunable components
while unlearning unnecessary skills.
1
Introduction
Today, the most competent AI systems in any particular domain are general systems that are relatively
competent in every domain. The best models at math and coding are also broadly knowledgeable about
a very diverse array of topics, from Roman history to home cooking recipes to medical di...