Before revision:  We characterize the dynamics that gradient descent induces on the factorization, and establish a bias towards low tensor rank , in compliance with empirical evidence. Then, motivated by tensor rank capturing  implicit regularization of a non-linear neural network, we empirically explore it as a measure of complexity, and find that it stays extremely low when fitting standard datasets .
After revision:  We characterize the dynamics that gradient descent induces on the factorization, and establish a bias towards low tensor rank search, which we rigorously prove under certain conditions, and empirically demonstrate under others. Motivated by tensor rank capturing implicit regularization of a non-linear neural network, we empirically explore it as a measure of complexity, and find that it stays extremely low when fitting standard datasets .