ICML 2025

Modular Duality in Deep Learning
Jeremy Bernstein 1 Laker Newhouse 1
Abstract
An old idea in optimization theory says that
since the gradient is a dual vector it may not be
subtracted from the weights without first being
mapped to the primal space where the weights
reside. We take this idea seriously in this paper
and construct such a duality map for general neu-
ral networks. Our map, which we call modular
dualization, forms a unifying theoretical basis for
training algorithms that are a) fast and b) scalable.
Modular dualization involves first assigning op-
erator norms to layers based on the semantics of
each layer, and then using these layerwise norms
to recursively induce a duality map on the weight
space of the full neural architecture. We derive
GPU-friendly algorithms for dualizing Embed,
Linear and Conv2D layers—the latter two meth-
ods are based on a Newton-Schulz iteration. We
conclude with small experiments demonstrating
the speed, scalability and novel numerical prop-
erties of duality-based optimizers. Our methods
were used in the Muon optimizer, which recently
set speed records for training NanoGPT and was
scaled up to a 1.5 billion parameter transformer.
1
Introduction
This paper pursues a rigorous and first-principles theoretical
framework for designing neural network training algorithms.
We hope that such a framework will facilitate the design of
a next generation of fast and scalable optimizers that are
automatically tailored to different neural architectures.
While gradient descent is the workhorse of modern machine
learning, the most vanilla form of the algorithm does not,
in our view, pass a basic type check. For a gradient update
to type check, we insist that the gradient must be passed
through a duality map before being multiplied by a learning
rate and applied to the weights:
1MIT CSAIL, United States. Correspondence to: Jeremy Bern-
stein <jbernstein@mit.edu>, Laker Newhouse <lakern@mit.edu>.
Proceedings of the 42 nd International Conferenc...