Title: Sharper Convergence Rates for Nonconvex Optimisation via Reduction Mappings
Venue: NeurIPS 2025
Accepted: yes
Keywords: Non-convex optimisation

Convergence rates

Gradient descent

Solution manifold

Reduction mappings

Curvature

Reparametrisations

Bilevel optimisation

Preconditioning
Abstract: Many high-dimensional optimisation problems exhibit rich geometric structures in their set of minimisers, often forming smooth manifolds due to over-parametrisation or symmetries. When this structure is known, at least locally, it can be exploited through reduction mappings that reparametrise part of the parameter space to lie on the solution manifold. These reductions naturally arise from inner optimisation problems and effectively remove redundant directions, yielding a lower-dimensional objective. In this work, we introduce a general framework to understand how such reductions influence the optimisation landscape. We show that well-designed reduction mappings improve curvature properties of the objective, leading to better-conditioned problems and theoretically faster convergence for gradient-based methods. Our analysis unifies a range of scenarios where structural information at optimality is leveraged to accelerate convergence, offering a principled explanation for the empirical gains observed in such optimisation algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is about using reduction mappings to exploit geometric structure of the solution set to accelerate convergence. It characterizes when reduction mappings lead to provable gains in optimization efficiency, i.e. optimizing on a manifold with a landscape that makes optimization easier for first-order methods. They show that under affine or nonlinear reduction mappings, the smoothness (Lipschitz) constant of the reduced function $F$ is strictly smaller than that of the original function.$f$. Under nonlinear reduction mappings, they also show that the sharpness constant is improved. Thus, the better condition number $\\kappa$ of $F$ over $f$ and faster convergence through the use of methods such as gradient descent where the rate depends on $\\kappa$.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Paper is very clearly written. Their framework ties together preconditioned methods, normalization methods, bilevel optimization and other domains via reduction mappings. \n - Authors give provable convergence improvements of nonlinear reduction mappings by showing that both the smoothness and sharpness constants are improved in the reduced \n\nWeaknesses\n - Could be difficult to understand how to use this from a practical standpoint. While this is a theory paper, pseudo-code showing how the algorithm could be implemented in practice and experimental results (even with synthetic data) could make this easier to grasp for practioners.\n - Some limitations were acknowledged by the authors: local analysis only, the assumption of exact evaluation of $\\Psi(x_1)$, the unknown (problem dependent) cost of evaluating $\\Psi(x_1)$, analysis limited to deterministic setting\n\nQUESTIONS:\n- Neural collapse is observed in stochastic settings. Could faster convergence via reduction mappings also be observed empirically?\n - Preconditioning is recommended to take advantage of the structure from reduction mapping, but in practice, exact applications of preconditioning is difficult. Is there an approximate form of the pullback metric $R(x_1)$ that is simpler, e.g. diagonal?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new framework for analyzing the contribution that the local geometry of minimizers has on the optimization process. The framework is based on reduction mappings that reparameterize the problem in terms of the solution manifold, which reduces the dimensionality of the problem. When the optimization process incorporates a well chosen reduction mapping, the authors prove that the local smoothness and sharpness constants of the objective improve, leading to faster local convergence rates. This framework unifies a number of approaches, such as batch normalization and equiangular tight frame projections in deep neural networks, that utilize reduction mappings and gives a theoretical explanation for the observed benefits of these approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper proposes a novel viewpoint to unify many commonly used techniques in nonconvex optimization that demonstrates the potential theoretical benefit of that incorporating a smartly chosen reduction mapping can have on the local convergence properties of first order methods. The proposed framework is novel and has a high degree of flexibility, allowing it to cover many interesting cases such as batch normalization, ETF reparameterizations, and bilevel optimization. Sufficient conditions are given under which a well chosen parameterization will yield improved local convergence properties for first order methods.\n\nWeaknesses: The paper mentions many potential applications of this framework, however, the authors do not apply their framework to any problem beyond some simple, 2-d examples given in the appendix. While the reduction mapping approach is interesting, it is hard to validate how effective it is at improving the understanding of the methods used without applying it to an example application, such as batch normalization.\n\nQuality: The paper is technically sound and provides an interesting viewpoint on how reduction mappings can have a beneficial impact on the local geometry of optimization problems. The framework is quite general and includes many interesting potential applications. However, no concrete applications of the framework is provided in the paper, which makes it hard to validate how tight the bounds generated by this approach are when compared with a more direct analysis on a case by case basis.\n\nClarity: On a whole, the paper is well written, however, some of the best motivation and explanation of reduction mappings is present in the appendix (section A). It would be ideal if some of this motivation was worked into the main body of the paper, to improve the introduction into the framework, which quickly becomes technical in the main paper.\n\nSignificance: The framework proposed is quite general and appears to apply to many interesting scenarios in nonconvex optimization. It seems that it could have a significant impact on how to view many strategies incorporated in modern optimization methods, though this is not entirely clear, as no concrete examples are evaluated using the proposed framework.\n\nNovelty: To my knowledge, this is the first work to use reduction mappings to unify a variety of techniques currently in use in nonconvex optimization. It provides a unique viewpoint, that could be used to not only better understand methods that are currently in use, but to design new techniques with improved performance.\n\nQUESTIONS:\nAre the results tight in the sense of matching previously known bounds? It is mentioned in the appendix that prior work has shown that in the case of batch normalization, the smoothness and sharpness constants are altered, just as in this work. In this case, does the proposed framework generate (roughly) the same improvement in these constants, or does the generality of the framework reduce the tightness of the theoretical results?\n\nThe curvature and spectral gap conditions are sufficient to guarantee the improved smoothness and sharpness constants. Is it known if these conditions are also necessary, or are there other conditions that could be satisfied for which a similar improvement is possible?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a formal and general framework for understanding and quantifying the benefits of using \"reduction mappings\" in nonconvex optimization problems where the set of minimizers forms a manifold. The core idea is that by reparametrizing the problem to stay on a manifold that encodes known optimal structure, one can provably improve the local geometry of the objective function, leading to faster convergence for gradient-based methods.\n\nSTRENGTHS AND WEAKNESSES:\nThe main strength is the move from qualitative intuition to quantitative results. The paper provides explicit bounds on the new smoothness and sharpness constants. This requires non-trivial mathematical machinery, likely involving matrix analysis and differential geometry (as hinted by the use of pullback metrics).\n\nThe framework is powerful because it's general. It applies to both affine and nonlinear reductions and covers a wide range of practical scenarios mentioned in the introduction (neural collapse, matrix factorization, phase retrieval). It provides a principled explanation for why techniques like fixing parts of a model or using certain projections can accelerate training.\n\nWeaknesses:\n\nWhile the theoretical contribution is strong, there are areas that could be discussed for a complete picture, which is typical for a focused research paper.\n\n1. Purely Theoretical: The paper is entirely theoretical and lacks experimental validation. While this is acceptable for a theory paper, even a simple numerical experiment on a synthetic problem (like the one in Figure 1) would make the theoretical gains more tangible. \n\n2. Practicality of the Mapping Ψ: The framework assumes the optimal structure, and thus the reduction mapping Ψ, is known. In many real-world problems, this structure might only be known approximately, or it might emerge during training. The analysis doesn't account for errors in the mapping Ψ.\n\n3. Computational Overhead: The proposed algorithm, Preconditioned Gradient Descent (PrecGD), requires computing and inverting the matrix R = DΦᵀDΦ. The paper acknowledges this cost but states that designing efficient solvers is \"beyond the scope of this work.\" This is a fair statement, but it is the key barrier to making this framework immediately practical for complex, nonlinear mappings.\n\nQUESTIONS:\n1. On the Practicality and Origin of the Reduction Mapping Ψ.\n\nCould the paper provide a concrete, illustrative example from the literature cited (e.g., neural collapse [14], dictionary learning [22], or phase retrieval [31]) and explicitly formulate the corresponding reduction mapping Ψ? For instance:\nIn the context of neural collapse, where optimal classifiers collapse to an Equiangular Tight Frame (ETF), is Ψ a projection onto the ETF cone?\nIn low-rank matrix factorization, min ||X - UVᵀ||², where solutions are only unique up to an invertible transformation (UR, V(R⁻¹)ᵀ), how would one define Ψ to fix this ambiguity?\n\n2. On the Verifiability of the Key Geometric Assumptions (ε and Δ)\n\nCould the paper provide more intuition on the plausibility of these assumptions in real-world machine learning settings? Could you further connect your assumption of a spectral gap to the empirical findings on Hessian spectra in deep learning [66-70], which you cite? Does the \"few large outlier eigenvalues\" phenomenon directly support the existence of a Δmax > 0?\n\n3. Lack of Experimental Results.\n\nHow large are the \"spectral gaps\" (Δmax, Δmin) in practice? How does the performance degrade as the assumptions (e.g., non-alignment) are gradually violated? It would help to answer these practical questions if experimental analysis could be involved.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}