Before revision:  This is a curious inequality: when applied to a discretization of a partial differential equation like -\Delta u = f, the last term measures the regularity of the residual u_k - u in a higher Sobolev space than the remaining terms: if u _k - u has large fourth derivatives (i.e. bi-Laplacian \Delta^2), then SGD will dramatically decrease the size of the second derivatives (i. e. \Delta) of u_k - u. For symmetric matrices, this inequality has an extension to higher-order Sobolev spaces.
After revision:  This is a curious inequality: when applied to a discretization of a partial differential equation like -\Delta u = f, the last term measures the regularity of the residual u_k - u in a higher Sobolev space than the remaining terms: if u _k - x is mainly comprised of large singular vectors, stochastic gradient descent leads to a quick regularization. For symmetric matrices, this inequality has an extension to higher-order Sobolev spaces.