ICML 2025

Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent
Santhosh Karnik * 1 Anna Veselovska * 2 3 Mark Iwen 4 5 Felix Krahmer 2 3
Abstract
We provide a rigorous analysis of implicit regu-
larization in an overparametrized tensor factoriza-
tion problem beyond the lazy training regime. For
matrix factorization problems, this phenomenon
has been studied in a number of works. A par-
ticular challenge has been to design universal ini-
tialization strategies which provably lead to im-
plicit regularization in gradient-descent methods.
At the same time, it has been argued by (Cohen
et al., 2016) that more general classes of neural
networks can be captured by considering tensor
factorizations. However, in the tensor case, im-
plicit regularization has only been rigorously es-
tablished for gradient flow or in the lazy training
regime. In this paper, we prove the first tensor
result of its kind for gradient descent rather than
gradient flow. We focus on the tubal tensor prod-
uct and the associated notion of low tubal rank,
encouraged by the relevance of this model for im-
age data. We establish that gradient descent in an
overparametrized tensor factorization model with
a small random initialization exhibits an implicit
bias towards solutions of low tubal rank. Our the-
oretical findings are illustrated in an extensive set
of numerical simulations show-casing the dynam-
ics predicted by our theory as well as the crucial
role of using a small random initialization.
1. Introduction
Analyzing implicit regularization during Neural Network
(NN) training is considered crucial for understanding why
*Equal contribution 1Department of Mathematics, Northeast-
ern University, Boston, USA 2Department of Mathematics and
Munich Data Science Institute, Technical University of Munich,
Munich, Germany 3Munich Center for Machine Learning, Mu-
nich, Germany 4Department of Mathematics, Michigan State
University, East Lansing, USA 5Department of Computational
Mathematics S...