Title: Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Stochastic gradient descent

Tikhonov regularization

convex optimization

minimum-norm solution
Abstract: The present article studies the minimization of convex, $L$-smooth functions defined on a separable real Hilbert space. We analyze regularized stochastic gradient descent (reg-SGD), a variant of stochastic gradient descent that uses a Tikhonov regularization with time-dependent, vanishing regularization parameter. We prove strong convergence of reg-SGD to the minimum-norm solution of the original problem without additional boundedness assumptions. Moreover, we quantify the rate of convergence and optimize the interplay between step-sizes and regularization decay. Our analysis reveals how vanishing Tikhonov regularization controls the flow of SGD and yields stable learning dynamics, offering new insights into the design of iterative algorithms for convex problems, including those that arise in ill-posed inverse problems. We validate our theoretical findings through numerical experiments on image reconstruction and ODE-based inverse problems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the stochastic gradient descent algorithm with Tikhonov regularization (reg-SGD), applied to the minimization of a convex smooth function on a Hilbert space. Theoretical results show that the iterates of the reg-SGD algorithm converge in norm, almost surely, to the minimum-norm solution with a careful choice of the step size and regularization parameters and under a certain condition on the minimum norm solution itself. When the latter assumption cannot be verified based on problem-specific structures, such as the source condition or the Łojasiewicz inequality, a second result on convergence in norm in the mean squared sense may still hold with more lenient assumptions. The authors offer both a theoretical analysis and a practical guidance for the optimal scheduling of the decaying step size and regularization parameters. Practical importance and feasibility of the algorithm are demonstrated on a Radon transform inversion arising in X-ray tomography problems.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths\n  1. The problem is clearly motivated, both theoretically and in practical examples and the analysis of the regularized SGD algorithm is technically solid with no blatant flaws.\n  2. Extension of the by now various results on SGD algorithm to Hilbert spaces and beyond is an interesting problem in and of itself in applied mathematics. The authors have made a contribution towards connecting the existing machine learning point of view towards SGD algorithms and the existing, albeit somewhat detached, literature on the analysis of Tikhonov regularization. \n\n* Weaknesses\n  1. The current exposition seems to only analyze SGD \"at large\" without delving into the specific ways in which the existing SGD algorithms are implemented. More specifically: in many machine learning tasks, and also in the tomography problem investigated in Section 3.2, the need for SGD often arises from the large number of observations, presumably \"independent\" in a precisely probabilistic or looser sense, hence the replacement of the full-scan gradient evaluation with a minibatch gradient evaluation. There are at least two sections in this work that may benefit from a more concrete connection to the assumption of gradient noise arising from minibatch sampling: **Assumption 1.2**, which I believe will be verifiable in this case; and the empirical investigations in **Section 3.2**, where the trade-offs involved in minibatch size and the performance of the reg-SGD remain unexplored.\n\nQUESTIONS:\n1. As elaborated in the Weaknesses Section, I suggest drawing concrete connections to the effects of minibatch sampling throughout the work. Specifically, this may be done through suggesting how **Assumption 1.2** may be verifiable and including the minibatch size as a tuning parameter in **Section 3.2**. Space constraints may be overcome by relegating them to the Supplements.\n2. How pliable do you believe the analysis of the method is to the following two extensions: (i) problems with *noise*, where a minimum-norm solution may not be optimal, as acknowledged in Section 4; and, (ii) problems with a *nonlinear operator*, which naturally arise in PDE parameter identification problems? It is conceivable both that the noise is also present in (ii) and that the need for SGD arises from a large number of independent observations.\n3. Section 1.3 leaves somewhat open the connection between this work and the parallel line of investigation into the generalizability of \"flat minima\" (and whether SGD can reliably find it). Can the analysis given in this work shed a light on a better understanding of when the SGD-found optimal are generalizable? Is the suggested generalizability in machine learning community due to the SGD \"imitating\" the effects of Tikhonov regularization, and if so, should the introduction of reg-SGD imply that the engineers revise their existing scheduling of parameter decay and early stopping rules to prevent \"excessive\" regularization?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes regularized SGD for the minimization of convex smooth function, with vanishing Tikhonov regularization parameter and proves convergence to the minimum norm solution. The analysis highlights properties about the the flow of SGD and stable learning, how to tune the step sizes and regularization parameters, and is supported by experiments.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written, easy to follow, and motivates the problem well while highlighting the novelty. The related work is thorough and the theoretical results are clearly stated.\n\nQuick note to check spelling and grammar, I spotted a few errors (e.g. \"continuous a line\" on line 83). \n\nSection 3.1 felt oddly placed or oddly named. I understand wanting to work through an example later in the paper, but motivating its importance is usually done quite early (I don't have a strong preference just wanted to raise this point).\n\nThere are many experiments in the appendices, some of which feel crucial to the main results. It is likely due to space concerns, but perhaps the authors may wish to re-assess which experiments are included in the main body.\n\nQUESTIONS:\n1) The authors might consider moving some of the experiments in Section A.2, which highlight the optimality of the theoretical schedules, to the main body of the paper, perhaps sacrificing one of the real examples of Section 3? I would understand the choice not to do this, but I do feel these appendix experiments are quite important for the point of the paper. \n2) Can the authors please do a thorough spelling/grammar check?\n3) Empirically, is the method sensitive to the tuned parameters? Can there be some computational savings by using approximations (throughout)?\n\nLIMITATIONS:\nSome limitations are discussed in the conclusion, mostly pointing out future directions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

comments: SUMMARY:
This paper provide convergence analysis for reg-SGD in optimization problems under convexity and  Łojasiewicz conditions problems. It establishes the $L^2$ and almost-sure convergence of SGD under the ABC conditions of the stochastic noise.

STRENGTHS AND WEAKNESSES:
**Strength** The paper is clear is easy to follow. I believe the result of this paper is correct.

**Weakness** 

1. The result is expected and not surprising to me. I think it is well-known that with regularization the algorithm will converge to the min-norm solution, although the existing analysis may not be comprehensive as this paper. I do not think this paper bring new insights to the community. 

2. The paper only studies problems under convexity and  Łojasiewicz conditions problems, but it lacks analysis of for general nonconvex optimization problems, which are the most widespread applications of SGD (or reg-SGD). For instance, neural network training.

QUESTIONS:
1. Could you please compare with the following recent work?

Boursier, E., Pesme, S., & Dragomir, R. A. (2025). A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation. arXiv preprint arXiv:2505.20172.

2. Are there any new insights that could potentially be useful be practice according to your theory?

LIMITATIONS:
See the weakness part.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n# Summary\n\nThis paper focuses on an algorithm combining SGD with Tikhonov regularization for minimizing a convex and smooth function $f$. \nThe paper is mostly theoretical, and provides numerous results showing that the algorithm converges towards the minimal norm minimizer of $f$, thereafter noted $x_*$. \n\nMore precisely, the authors leverage the introduction of the regularization term to obtain\n\n- Almost-sure and $L^2$ convergence of the iterates $X_k$ towards $x_*$,\n- Convergence of the values $f(X_k)$ towards $\\inf f$,\n- Convergence rates for the above quantities,\n\nall this under more or less strict assumptions on the parameters of the method (namely: the learning rate and the regularization parameter).\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n\nOverall I was very positively impressed by the paper.\n\n- It is very well written. While the topic is technical, the paper is quite easy to read\n- The contribution is very good. The results (see the summary) answer all the questions we could have for such stochastic method (well, one could always ask for high probability results...). The obtained rates are for most of them as good as what is currently known for vanilla SGD.\n- The assumptions are minimal, and match what are standard assumptions in the community: no bounded gradient assumption but a mere ABC-like condition (which is always verified for vanilla SGD), basic assumptions on the parameters\n\n# Weaknesses\n\n- The main complaint I have is the connection to the literature. The authors claim that there is one only work on gradient algorithms combining with a Tikhonov term. This is incorrect. The literature on the topic is quite consequent and goes back to the 70's, ranging from simple perturbations of the Landweber algorithm for inverse problems, to general methods for selecting a specific minimizer of a function f. Here are a few entries (there are many others, see the literature therein):\n  - \"Convergence of diagonally stationary sequences in convex\n    optimization\"\n  - \"Coupling forward-backward with penalty schemes and parallel splitting for constrained variational inequalities\"\n  - \"Splitting forward-backward penalty scheme for constrained variational problems\"\n  - for the applications in inverse problems (that the authors seem to favor):\n    - \"A modified Landweber iteration for solving parameter\n      estimation problems\"\n    - \"Iterative Regularization Methods for Nonlinear Ill-Posed Problems\"\n    - \"Iterative Methods for Approximate Solution of Inverse Problems\"\n- An other (minor) weakness is that I did not find the numerical experiments very compelling. I would have rather liked to see whether Reg-SGD can help find solutions which generalize better (or no numerics at all).\n\nQUESTIONS:\n# Questions/comments\n\n1. I would like the authors to review properly the existing literature on deterministic Tikhonov-gradient algorithms. Then I would like to see some comparison between their results / hypotheses of Section 2.1 with what was done in the deterministic setting, both in the continuous and discrete regime. I believe this would allow the reader to better appreciate the results. \n\n   1. For instance the third assumption in (6) seems strongly connected to assumptions which already appeared (albeit under different forms) in the literature. See\n\n      - p.132 in \"Coupling the gradient method with a general exterior penalization scheme for convex minimization\"\n      - Hypothesis H4 in \"Prox-Penalization and Splitting Methods for Constrained Variational Problems\"\n\n      and in particular look at how those papers make a connection with  the Łojasiewicz inequality (or other equivalent error bounds such as $f(x) - f(x_*) \\geq C dist(x, argmin f)^\\tau$ ). \n\n   2. Are the assumptions (7) the same that what is known in the deterministic case? Is there any work able to obtain convergence without making an assumption like in (6)? \n\n   If the above is done properly **I will raise my score**.\n\n2. Appendix D:  It is known that source conditions imply that a Łojasiewicz inequality holds onto some set, see Theorem 5.10 in \"Convergence of the forward-backward algorithm: beyond the worst-case with the help of geometry\". Can it be proved that the regularization path $x_\\lambda$ belongs to such set? If yes, then a connection could be made between the \"linear inverse problem\" scenario and the \"Łojasiewicz\" scenario.\n\n3. Authors claim that Tikhonov regularization helps preventing the explosion of the trajectory of SGD. I wonder how true is this claim, given that it is not yet fully known what can or cannot do SGD (here by SGD I mean sampling a function f_i when minimizing an expectation of f_i). For instance recent results (see \"Stochastic Gradient Descent Revisited\") indicate that SGD does not need bounded trajectories to converge in function values. More precisely, convergence of the values always hold, so under a mere coercivity assumption we can recover the boundedness of the iterates. \n\n4. You certainly want to tell the reader that Assumption 1.2 is not an assumption when $f = \\mathbb{E}[f_i]$ and $\\hat \\nabla f(x) = \\nabla f_i(x)$ , and always holds true with $A=4L$ and $C = \\mathbb{E}[\\Vert \\nabla f_i(x_*) \\Vert^2]$ .\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the Tikhonov regularized stochastic gradient descent (reg-SGD) algorithm for minimizing convex and $L$-smooth functions in separable real Hilbert spaces. \n\nAssuming a typical ABC-type condition on the noise sequence and without requiring any boundedness assumptions, the authors prove almost sure and $L^2$ convergence of the last iterates of reg-SGD to the minimum-norm solution. For polynomial stepsize and regularization schedules, they further establish sublinear convergence rates in both almost sure and $L^2$-convergence senses, with refined results under the \\L ojasiewicz condition. \n\nNumerical experiments are provided to support the theoretical results.\n\nSTRENGTHS AND WEAKNESSES:\nS1. This paper establishes almost sure and $L^2$ convergence of reg-SGD to the minimum-norm solution without imposing additional boundedness assumptions. Moreover, the almost sure boundedness of the iterates is proved, highlighting the stability of reg-SGD to vanilla SGD. \n\nS2. This paper provides clear guidance on explicit choices of polynomial stepsize and regularization schedules that guarantee convergence of reg-SGD, enhancing the applicability of the analysis in pratical applications.\n\nW1. Typos. \n1) Line 228, 631, 742: It should be \"$\\Vert X_k - x_{\\lambda_{k+1}}\\Vert_{\\mathcal{X}}^2$\". \n2) Line 666-667: It should be \"$\\lambda_k - \\lambda_{k+1} = o(\\alpha_k \\lambda_k)$\". \n3) Line 701-702: It should be \"$\\cdots - \\frac{p}{2\\varepsilon^\\prime C_\\alpha}\\Vert x_{*}\\Vert_{\\mathcal{X}}^2$\". \n4) Line 726, 750: It should be \"$p+q\\leq 1$\". \n5) Line 728: It should be \"$p+q = \\min\\\\{1+p-\\beta,2q-\\beta\\\\}$\". \n6) Line 749: It should be \"$E[\\varphi_k\\ \\vert F_{k-1}] \\leq \\cdots \\varphi_{k-1} + \\cdots$\".\n\nW2. Gaps in the Assumptions of Theorem 2.1. The claim in Line 658-659 that $\\varphi_k\\to 0$ almost surely via Corollary B.8 lacks sufficient justification. To apply Corollary B.8, it is necessary to verify that  \n\n$\n1-\\tau_k = \\min\\\\{\\alpha_k\\lambda_k, 2\\alpha_k\\lambda_k(1-\\frac{L+\\lambda_k}{2}\\alpha_k)-(\\frac{L+\\lambda_k}{2}A+A+2L+2\\lambda_k)\\alpha_k^2\\\\} \\geq 0\n$\n\nholds on $B_\\infty(N)$ for sufficiently large $k$. However, under the assumptions of Theorem 2.1, this inequality may not be satisfied. As a counterexample, define $C_1 \\triangleq \\frac{LA}{4} + \\frac{A}{2} + L$, set $\\lambda_k = \\frac{1}{\\sqrt{k}}$ for $k\\in Z_{+}$, and let \n$$\n\\alpha_k = \\begin{cases}\n\\frac{2}{C_1}\\lambda_{k}\\quad &\\mbox{if}\\ k = 2^n\\ \\mbox{for some}\\ n\\in \\mathbb{N}\\\\\\\\\n\\frac{\\lambda_k}{\\log k}\\quad &\\mbox{otherwise}.\n\\end{cases}\n$$\nThen it follows from   \n$\n\\sum_{n\\geq 0} \\frac{1}{2^n} < \\infty,\\quad \\sum_{k\\geq 3}\\ \\frac{1}{k(\\log k)^2} < \\infty\\quad \\mbox{and}\\quad \\sum_{k\\geq 3}\\ \\frac{1}{k\\log k} = + \\infty,\n$\nthat $\\sum_k \\alpha_k^2 < \\infty$ and $\\sum_k \\alpha_k \\lambda_k = \\infty$, satisfying the requirements of Theorem 2.1. However, for any $k = 2^n$ with $n\\in \\mathbb{N}$, we have \n\n$\n1-\\tau_k \\leq 2\\alpha_k \\lambda_k - 2C_1 \\alpha_k^2 = 2\\alpha_k \\lambda_k (1-C_1\\frac{\\alpha_k}{\\lambda_k}) = \\alpha_k \\lambda_k (1-C_1 \\cdot \\frac{2}{C_1}) = -\\alpha_k \\lambda_k < 0,\n$\n\nwhich implies that $1-\\tau_k < 0$ occurs infinitely many times. Consequently, the key condition for applying Corollary B.8 is violated, and the almost sure convergence of $\\varphi_k$ cannot be guaranteed under the stated assumptions.\n\nW3. Assumptions in Theorem 2.2 and proof argument in Line 668. See W2.\n\nW4. Proof argument of Theorem 2.4. In Line 756, the derivation of $\\Psi_k\\in \\mathcal{O}(k^{-1+q})$ by an argument analogous to the proof of Theorem C.5 is potentially problematic. When $q>0$ and $q = 1-p$, the proof of Theorem C.5 requires the condition that $2C_\\lambda C_\\alpha > 1-q$ (see Line 698-699), which may not be satisfied under the assumptions of Theorem 2.4 when $\\beta < 1-q$.\n\nW5. Proof argument of Theorem 2.4. In Line 758-759, the application of Corollary B.7 to establish almost sure convergence of $\\{\\varphi_k\\}$ is not fully justified. The corollary requires a uniform lower bound on $\\{\\varphi_k\\}$, but the almost sure boundedness of $\\{\\tilde{E}_k\\}$ alone is insufficient, since $\\varphi_k = \\tilde{E}_k k^\\beta$ with $\\beta > 0$ by definition. This raises concerns about the applicability of the corollary in this context.\n\nQUESTIONS:\nPlease see the weakness part and should address W2.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}