Title: Learning Provably Improves the Convergence of Gradient Descent
Venue: NeurIPS 2025
Accepted: yes
Keywords: Learn to Optimize

Gradient Descent

Neural Tangent Kernel
Abstract: Learn to Optimize (L2O) trains deep neural network-based solvers for optimization, achieving success in accelerating convex problems and improving non-convex solutions. However, L2O lacks rigorous theoretical backing for its own training convergence, as existing analyses often use unrealistic assumptions-a gap this work highlights empirically. We bridge this gap by proving the training convergence of L2O models that learn Gradient Descent (GD) hyperparameters for quadratic programming, leveraging the Neural Tangent Kernel (NTK) theory. We propose a deterministic initialization strategy to support our theoretical results and promote stable training over extended optimization horizons by mitigating gradient explosion. 
Our L2O framework demonstrates over 50% better optimality than GD and superior robustness over state-of-the-art L2O methods on synthetic datasets. 
The code of our method can be found from https://github.com/NetX-lab/MathL2OProof-Official.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a significant and often overlooked issue in the Learning to Optimize (L2O) field: the formal proof of convergence for the training process of L2O models. The authors focus on the Math-L2O framework and leverage Neural Tangent Kernel (NTK) theory to prove that learning hyperparameters for Gradient Descent (GD) can provably enhance its convergence rate. A novel initialization strategy is also introduced to support the theoretical claims and ensure training stability.\n\nSTRENGTHS AND WEAKNESSES:\nThe primary strength of this work lies in its novelty and significance. Many L2O frameworks implicitly assume that the underlying neural network models are trainable and will converge during training.\n\nQUESTIONS:\nWhile the contributions are significant, the paper's clarity could be improved in several key sections. The following points should be addressed to strengthen the manuscript.\n\nLIMITATIONS:\nIn section 3.1: the goal of this section is to analyze the limitations of existing frameworks, but the presentation could be more specific and balanced.\n\n- In Lines 157-162, the descriptions of prior work use general terms like \"stringent conditions\" and \"necessary conditions\" without immediately specifying what those conditions are. It would be more impactful to state the specific limitations upfront before the empirical analysis.\n- The section provides a detailed empirical analysis of the limitations of LISTA-CPSS (e.g., training instability and violation of sign consistency conditions). However, a similar analysis for the Math-L2O framework, which is also introduced as a target of analysis in this section, appears to be missing. The text should clarify the specific limitations of the original Math-L2O framework that this work aims to overcome.\n\nIn section 3.2: This section introduces the core conceptual framework of the paper, but its presentation is difficult to follow.\n- Can you explain that why $X_{T}^0$ occurs two sides of equation (9)?\n- In Line 202, the paper asserts that Equation (10) demonstrates a sub-linear convergence rate of at least $O(1/T^2)$. Can you illustrate it more details?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proves that the L2O methods probably improve the convergence of gradient descent.\nThe authors establish that L2O can improve GD convergence rates by a factor of $r_k$ and prove training convergence of the L2O network using NTK theory in the over-parameterized regime. To bridge theory and practice, they propose an initialization strategy that satisfies the theoretical assumptions. Experimental results demonstrate improved convergence stability of the trained optimizer and validate the effectiveness of the proposed initialization scheme.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- It provides solid theoretical analysis for L2O training convergence using NTK theory in the over-parameterized setting.\n\nWeakness\n\n- Convergence improvement claims require clarification (Section 3.2):\n     - The relationship between the LHS of Eq. (8) and the RHS of Eq. (9) in line 198 is unclear. Does this indicate that an L2O algorithm's initial state matches non-learning GD algorithms? If so, does the Deterministic Initialization in Sec. 5 satisfy this requirement?\n     - The author claims \"Eq. (10) demonstrates a sub-linear convergence rate of at least $O(1/T^2)$\", which is unclear since Eq. (10) shows a convergence rate of $O(r_k \\beta / T)$ and $r_k$ is independent of $T$.\n     - $r_k$ in Eq. 10 seems not to appear in Sec. 5 or Sec. 6.\n     - A formal theorem with precise definitions and assumptions is needed to substantiate these claims.\n- Novelty clarification needed: \n    - The technical contribution when applying NTK theory to prove linear convergence of L2O training in Sec. 4, compared to the cited work (Nguyen 2021), should be stated more explicitly.\n\nI will reconsider my evaluation if these concerns are addressed.\n\nQUESTIONS:\n1. Regarding the improvement of L2O over GD:\n   - What are the upper and lower bounds of this improvement?\n   - Does this improvement stem primarily from learning optimal hyperparameters (as mentioned in line 205), or is there another explanation?\n   - What is the worst-case/generalization performance of the trained L2O optimizer? Is it consistently superior to GD across all test instances with varying problem parameters?\n2. Regarding neural network width requirements:\n   - Section 4 requires NN width to be O(Nd), which is reflected in experiments (e.g., width 5120 in line 320). Do smaller widths (e.g., O(d) as suggested by universal approximation theory) perform adequately in practice?\n   - Is there a direct performance comparison among the three neural network architectures in Table 1?\n\nLIMITATIONS:\nSee the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the convergence of learning an NN-based optimizer that is used to solve the quadratic optimization problem. In particular, the paper focus on Math-L2O, whose learned optimizer outputs a step size which is applied in the gradient descent step of the base optimization problem. By leveraging the NTK-based analysis, the paper derives a linear convergence rate of the optimizer training. The paper also provided experiments to validate the theoretical results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The problem tackled by this paper is both significant and difficult. To further support the contribution of the paper, experimental results showing that the assumptions made in previous works are unrealistic are also included.\n\n2. The paper deals with a complicated deep MLP optimizer in the Math-L2O setting. This is both technically difficult and general in terms of the result.\n\n3. The convergence result proved by the paper is a clean linear convergence with no extra error terms.\n\n**Weaknesses**\n\n1. The main concern I have about the paper is that Theorem 4.3 shows that applying the output step size to the GD optimization problem with a fixed number of steps $T$, the final loss converges to zero as the training steps $k$ goes to infinity. This is particularly confusing as it seems that there is no requirement on $T$. In particular, this means that, as long as we do sufficient training to the optimizer model, its output step size can result in a convergence of the base optimization problem even with one step of gradient descent, which seems super intuitive. I suggest the author to provide a reasonable explanation of this surprising result.\n\n2. The choice of the step size in (13a) and (13b) seems complicated. Yet, the paper does not provide a good interpretation of what these expressions means and how the step size scales intuitively. Without such interpretation, it is possible that the step size are super small and lead to very slow convergence.\n\n3. Some of the notations are unclear. For instance,  $W\\_{[L]}$ never appears on the right-hand side of Eq.(2), where the dependency should be through $X\\_T$. In Eq.(6) and Eq.(7) I believe that parentheses are missing in the product terms, and similarly for the equation in Lemma 4.2, where the last parenthesis should include $||W\\_{\\ell}^{k+1} - W\\_{\\ell}^k||_2$. Moreover, It is a bit unclear what $F(W^k)$ really denotes, although it could be inferred from the text that it is $F(X\\_T, W\\_{[L]})$.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical study of Learn to Optimize (L20) framework, where the optimization of the main model is performed using an additional model that generates per-coordinate stepsizes. This work provides convergence guarantees for such an optimization problem when the original problem that we want to solve using the main model is quadratic. The authors provide a linear convergence rate for the Math-L2O framework. The theoretical claims are supported by empirical ablations.\n\nSTRENGTHS AND WEAKNESSES:\n***Strengths:***\n\n- Convergence guarantees for a quadratic objective in NTK regime (which is somehow needed to ensure that PL condition holds and consequently leads to linear convergence).\n- Experimental results demonstrate that Math-L2O framework might outperform classic algorithms like GD and Adam.\n- The authors demonstrate that previous assumptions used in previous approaches, like LISTA-CPSS, are not verified during training (Figure 3)\n\n***Weaknesses:***\n- The paper is poorly written, and the technical details are hard to parse. Due to typos and/or inaccuracies in writing, overloaded notation it is hard to evaluate the theoretical results (e.g. $M_i, W_{[L]}$ are never defined in the main part; line 224 involves so many new definitions without any intuition why they are even needed for a presentation, how they are used, and so on). I also didn't find a specific place where some properties are explicitly used (the definition of NTK is also absent). Therefore, in my view, evaluation of the theoretical results requires a significant improvement in the presentation.\n\n- Algorithms like Adam and SGD shine when training real networks of enormous sizes. Testing the performance of the proposed scheme against those algorithms on a toy quadratic problem is not a convincing result that can be used to claim \"up to a 50% improvement\". I acknowledge that deriving theoretical guarantees for Math-L2O seems to be a challenging task, but the evaluation has to be on at least MLP model on MNIST, which is a standard benchmark in DL (although small).\n\nQUESTIONS:\n- Can we use LSTM-type networks that suffer less from vanishing/exploding gradient problems instead of RNN-type updates?\n\n- Can the theory be generalized to non-ReLU types of activations?\n\n- Can the authors clarify why we obtain $O(1/T^2)$ rate (line 203)?\n\n- Line 223 is confusing. Are $\\alpha_0$, $\\overline{\\lambda}\\_{\\ell}$, $C\\_{\\ell}$ fixed? Can we always find such constants that (11) is verified? The next concern in this line is the assumption $\\\\{\\\\|W\\_{\\ell}^{k+1}\\\\|_2,\\\\|W\\_{\\ell}^{k}\\\\|_2\\\\}\\le\\overline{\\lambda\\_{\\ell}}$. The proof of it is also unclear. Can the authors show how Weyl's inequality is used?\n\n- The initialization (section 5) is unclear. The authors first set weight matrices to be zero, but then sample from a Gaussian distribution. Which one is used in the end?\n\n- Typo: line 546, \"eigenvalues\" are missing\n\n- The proof of Lemma A.2 is not full. How to obtain the final bound (31)?\n\n- Where is the assumption $0 < P^k_t < 2$ verified which is used in some lemmas (e.g., Lemma A.1)?\n\n- Can the authors provide more details on why the bound $\\\\|W\\_j^{k+1}\\\\|\\le\\overline{\\lambda}_j$ holds?\n\n- Where do the authors use the NTK regime? Can the authors clarify this point, please? Is it used when assuming that $\\sigma\\_{\\min}(G\\_{L-1,T}^0) > 0$?\n\nLIMITATIONS:\n- It seems that the proof techniques are extremely hard to generalize beyond the quadratic case.\n- The authors discuss the limitations in Section D, mentioning that the explicit form for $X_T$ related to $X_0$ is crucial in the derivations. This is more complicated for other algorithms, e.g., Nesterov Accelerated Gradient. \n- I believe it is hard to extend this approach to neural networks of real size since the input of the second NN is $[X_t,\\nabla F(X_t)]$, which is a very high-dimensional nowadays.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}