ICML 2025

Enforcing Idempotency in Neural Networks
Nikolaj Banke Jensen 1 Jamie Vicary 2
Abstract
In this work, we propose a new architecture-
agnostic method for training idempotent neu-
ral networks. An idempotent operator satisfies
f(x) = f(f(x)), meaning it can be applied itera-
tively with no effect beyond the first application.
Some neural networks used in data transformation
tasks, such as image generation and augmenta-
tion, can represent non-linear idempotent projec-
tions. Using methods from perturbation theory we
derive the recurrence relation K′ ←3K2 −2K3
for iteratively projecting a real-valued matrix K
onto the manifold of idempotent matrices. Our
analysis shows that for linear, single-layer MLP
networks this projection 1) has idempotent fixed
points, and 2) is attracting only around idempo-
tent points. We give an extension to non-linear
networks by considering our approach as a sub-
stitution of the gradient for the canonical loss
function, achieving an architecture-agnostic train-
ing scheme. We provide experimental results for
MLP- and CNN-based architectures with signif-
icant improvement in idempotent error over the
canonical gradient-based approach. Finally, we
demonstrate practical applications of the method
as we train generative networks on MNIST and
CelebA successfully using only a simple recon-
struction loss paired with our method.
1. Introduction
Using neural networks as data augmentation tools is becom-
ing more widespread in areas such as signal processing and
generative artificial intelligence. In particular, networks of
the form f : X →X, mapping data within the same space
X, are frequently used in image augmentation (Lu et al.,
1Department of Computer Science, University of Oxford, Ox-
ford, UK. 2Department of Computer Science and Technology,
University of Cambridge, Cambridge, UK. Correspondence to:
Nikolaj Banke Jensen <nikolaj.jensen@cs.ox.ac.uk>, Jamie Vi-
cary <jamie.vicary@cl.cam.ac.uk>.
Proceedings of the 42 nd International Conference ...