Sharper Convergence Rates for Nonconvex Optimisation via Reduction Mappings
Authors: Evan Markou
Thalaiyasingam Ajanthan
Stephen Gould
Venue: NeurIPS 2025 spotlight ()
Keywords: Non-convex optimisation
Convergence rates
Gradient descent
Solution manifold
Reduction mappings
Curvature
Reparametrisations
Bilevel optimisation
Preconditioning
Primary area: optimization

Abstract: Many high-dimensional optimisation problems exhibit rich geometric structures in their set of minimisers, often forming smooth manifolds due to over-parametrisation or symmetries. When this structure is known, at least locally, it can be exploited through reduction mappings that reparametrise part of the parameter space to lie on the solution manifold. These reductions naturally arise from inner optimisation problems and effectively remove redundant directions, yielding a lower-dimensional objective. In this work, we introduce a general framework to understand how such reductions influence the optimisation landscape. We show that well-designed reduction mappings improve curvature properties of the objective, leading to better-conditioned problems and theoretically faster convergence for gradient-based methods. Our analysis unifies a range of scenarios where structural information at optimality is leveraged to accelerate convergence, offering a principled explanation for the empirical gains observed in such optimisation algorithms.

Sharper Convergence Rates for Nonconvex Optimisation via Reduction Mappings
Evan Markou
Thalaiyasingam Ajanthan
Australian National University
Australian National University & Pluralis Research
evan markou@anu. edu.au
chalaiyasingam, waa.raaggaaaaaaaaaaaaaaann janthannnnu. edu.
Stephen Gould
Australian National University
stephen. goul de anu. edu. au
Abstract
Many high-dimensional optimisation problems exhibit rich geometric structures their set of minimisers, often forming smooth manifolds due to over parametrisation or symmetries. When this structure is known, least locally, can be exploited through reduction mappings that reparametrise part of the parameter space to lie on the solution manifold. These reductions naturally arise from inner optimi- sation problems and effectively remove redundant directions, yielding a lower- dimensional objective. In this work, we introduce general framework to under- stand how such reductions influence the optimisation landscape. We show that well-designed reduction mappings improve curvature properties the objective, leading to better-conditioned problems and theoretically faster convergence for gradient-based methods. Our analysis unifies a range of scenarios where struc- tural information at optimality is leveraged to accelerate convergence. offering a principled explanation for the empirical gains observed in such optimisation algorithms.
Introduction
First-order gradient methods are the workhorse for large-scale optimisation in machine learning and data science due to their simplicity and scalability. However, objective functions encountered in these settings often exhibit highly non-convex and intricate loss landscapes, stemming from various factors such as over paramctrisation, compositions of nonlincar functions, and underlying data distributions, to name few (+4). Despite this complexity, gradient-basee based methods perform remarkably well in practice, achieving local linear convergence under mild regularity conditions such the widely studied Polyak-Lojasiewin theoretical difficulty and empirical success motivates a deeper understanding of the geometry of loss landscapes and its role in shaping optimisation dynamics.
(PE) condition [.9.. This apparent tension between
us consider the geometry of the solution spaces, which are shown to have manifold-lie structures, rather than isolated points for many machine learning problems due to over wer-parametrisation sym- metries, or latent invariances [10 6). Such structured sets minimisers are not merely theoretical curiosities- -teeie naturally arise in variety of real- world applications. For instance, in deep neural networks, over- weer-paramerrisation often leads to entire manifolds of local minima [TO] related by sym- metries [ Furthermore, recent findings in neural collapse [14] reveal that optimal solutions often exhibit highly regular, symmetric, low-dimensional structures across layers [1] In matrix factorisation problems [2I] such dictionary learning 222 25 low-rank matrix completion [26 27). and tensor decomposition problems 28 29 solutions are only identifiable up to scaling or orthogonal
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ambient Surface of f
f(z1 02) = 2(z2 =m)²
Fi()) f(m1 (1(x1)
Mrj (1) (a E EB)
(s( f(m, Ma(z1))
Mny = ((m1,0) 12 ER)
F((2) f(n, (s(x1))
MF, ((01, 2eein(z1) RR)
32
37
Figure 1: Illustration of how well-designed reduction mappings iron out worst-case curvature. The opaque surface depicts the graph of the function ʃ R22 - R, lifted above the ambient domain (grey plane) for visualisation purposes. The function has single global minimum S {(0, 0)). In general, can be set of non-isolated points. The blue curve shows the restriction of f along the mapping V1 : 21 + T1, where the high-curvature quadratic component cancels, yielding flatter profile in T1. In contrast, the orange curve corresponds to the mapping U + O, which preserves most of the steep curvature of f. The grey-green curve traces the restriction along nonlinear sinusoidal mapping an example of poorly designed reduction, which introduces additional curvature into the problem. Dashed curves on the ambient domain represent the images of these mappings as one-dimensional submanifolds M. The submanifolds have a non-empty intersection with S.
reflecting the inherent symmetries of these models. Similar invariances exist in problems such as phase retrieval [31 and blind deconvolution [35H40]. These symmetries induce structured sets of minimisers, often forming smooth manifolds or discrete equivalence classes, reflecting invariance under these transformations (30. These examples all share a common theme:
transformations 30
the objective exhibits invariances that endow the set of solutions with rich geometric structure.
In this work, we study how the geometric structure of the solution can systematically exploited to accelerate convergence. We focus on reduction mappings-reparamerees reparametrisations that encode known components of the solution manifold, and study their effect on the local curvature. In practice, these reductions often arise from inner optimisation problems, thereby reformulating the original problem into a bi-level optimisation problem and yielding a lower-dimensional objective. While intuitively promising, not all such reduction mappings are beneficial. Even if optimal in value, poorly designed reduction can distort local curvature and hinder convergence. We develop a rigorous framework to identify when and how these mappings lead to provable improvements in outer iteration complexity for gradient descent.
To build geometric intuition, Figure jillustrates how different reduction mappings affect the local curvature of the objective along their respective subspaces. For more details on this example, refer to Appendix A While some mappings reveal a well-conditioned, flattened profile, others retain or even exaggerate steep curvature. This highlights a central idea of our work: the convergence behaviour of gradient methods depends not just on the presence of structure, but how effectively it incorporated into the optimisation process.
1.1 Contributions
We provide systematic characterisation when reduction mappings lead to provable gains in optimisation efficiency. Specifically, we identify conditions under which the reduced objective exhibits a strictly smaller smoothness constant and strictly larger sharpness constant] Together, these improvements yield strictly better condition number, leading faster worst-case convergence
A general term encompassing Polyak-Lojasiewwn PL. quadratic growth (QG). and related properties.

---

rates for gradient- based methods applied to the reduced problem. The results hold under standard regularity assumptions and apply to both affine and nonlinear mappings. Specifically we contribute the following:
Theorem shows that for affine reduction mappings, the smoothness constant of the reduced objective strictly smaller than that of the full objective.
Theorem 2
extends this result to general nonlinear mappings, establishing improved smooth-
ness under mild regularity assumptions.
Theorem demonstrates that the sharpness constant of the reduced problem is strictly larger, implying stronger curvature near the minimisers.
Corollary combines the above to show that the condition number of the reduced problem is strictly better, leading to faster convergence of gradient descent under the PL condition.
For completeness, we include number of supporting lemmas and additional theoretical results the
appendices.
1.2 Related Work
Reparametrisations and geometry-aware optimisation have long been used to exploit structure and improve conditioning in nonconvex problems [41). Examples include normalisation methods [42 431. which can be interpreted as explicit reduction mappings improving feature geometry and convergence 44]; neural collapse [14] formulations, where our framework captures both fixed classifier parametrisations [45) and dynamic equiangular tight frame (ETF) projections [46]: and gauge-fixing strategies in problems with symmetries, offering lightweight alternative quotient manifold optimisation 47]48) Preconditioned methods also relate closely to our approach, with connections natural gradient descent (49 50], adaptive preconditioning [51 [22.. and studies of parameter space geometry under reparametrisation [53]. Our framework generalises these by treating reduction mappings intrinsic geometry-aware preconditioners, unifying and extending prior work on preconditioning and geometry adaptation. Finally, classical variable elimination and bilevel optimisation methods [54] [55] can be seen as special cases of reduction mappings. where our analysis goes beyond dimensionality reduction to provide precise improvements in conditioning and convergence rates. A more detailed discussion is provided in Appendix B
Setup, Notation, and Preliminaries
We consider general unconstrained optimisation problems of the form
minimise f f(x): ZE
where : - is C2 (twice differentiable), possibly non-convex function. We assume that the set of minimisers of is not discrete, but instead forms a non-isolated set. Specifically, we define the set of all local minima in a neighbourhood as
S {i R x is local minimum of f and f(x) = c]-
(2)
Without loss of generality, we assume the minimum of f is zero, i.e., c= 0. We further assume that, locally around any minimiser, the function f satisfies the Pt. condition with constant > 0 (u-PE.), that is,
1 [Vf(2
f(x) < 24
(PL)
Other related conditions commonly imposed in non-convex optimisation include the quadratic growth (QG) condition (56) and the error bound (EB) condition [57]. These can be defined follows, where
dist is the classical Euclidean distance: satisfies the quadratic growth condition with μ > 0 around
a minimum if
f(x) 2 / dist (I,
(QG)
2
Also, f satisfies the error bound condition with μ > 0 around a minimum if
p dist(x, S) [x()]]]
(EB)
3

---

Numerous works derive convergence rates for gradient-based methods under these assumptions [58 6I]. Another important condition is the Morse- Bott property [62 which generalises the classical Morse theory framework 665] While Morse functions require all critical points to be isolated and non- degenerate, Morse- Bott condition relaxes this by allowing the set of critical points to form smooth manifolds, as long the Hessian is non-degenerate in directions normal to these manifolds [62]. This setting is particularly relevant in optimisation problems where symmetries or invariances naturally give rise non-isolated minimisers lying structured sets and hence singular Hessians.
Let E be local minimiser of f with associated set of min- imisers S and S& be the tangent space of S at I. Then f satisfies the Morse-Bot property at T if
Definition 1 (Morse-Bott Property)
S is a C² submanifold around I and ker 4 f()) S&
(MB)
Furthermore, if there exists a uniform > such that for all vectors v normal to one has
(v, (()) 2 2 11ll
then we say that f satisfies the u-MB property.
Notably, for C2 functions, it has been shown (64] that these conditions essentially equivalent-uup to potential degradations in the constants or reductions in the neighbourhoods where they hold. Therefore, throughout this work, assuming any one of these conditions allows invoke the others interchangeably. Under (MB), we refer to set of minimisers S as the solution manifold.
Assumption 1 (Standing Assumptions on Function f) Suppose there exist constants L, 3, p > 0 and a compact neighbourhood N C. R2 around minimiser i' such that the following hold:
1. (Interpolation) The function f attains its infimum at zero, and the set of minimisers S is non-empty.
2. (Smoothness) The function f is L-Lipschitz continuous and 8-smooth on N; that is, its gradient is 3-Lipschitz continuous on N.
3. (PL condition) The function J satisfiex the PL condition every point in N.
2.1 Reparametrisation via Reduction Mappings
We begin by decomposing the variable x the function f into two components, 21 E R and 22 R such that nı + n2, with nı < Whenever the problem structure prior knowledge permits, we assume that the component 22 exhibits a known geometric structure at optimality. This a projection or implicit parametrisation-
allows to introduce an appropriate that explicitly encodes this structure by setting 22 as a function of #1- This leads to a reduction mapping, where 22 is fixed lie on its optimal structure, enabling us reformulate the problem by optimising only over the remaining variables æ1.
Definition 2 (Reduction Mapping and Reduced Function) Let V : R - R be a C2 mapping representing the known geomeiric structure of x2 al optimality. We define the reduction mapping c T. -> R. as
v(x1) = (I1, ((I)).
We then define the reduced objective F: TR - as the pullback of f along t as
F(x1) f (((())) f(1 (x1))
Since f and & are C², the reduced funccitfunntionn is also C?.
Definition (Graph Manifold) We define the graph manifold of the reduction mapping & as
Mr = (x(x)))= = (x1, M(x1)) | E E
(5)
Assuming VI is C2, the manifold Mp, which we will refer to as the feasible manifold, is a globally embedded 2 submanifold of R. of dimension nh, since t is globally injective and C
Assumption (Standing Assumptions on the Reduced Function F) Assume the following hold within a compact neighbourhood N C. IR" defined around a minimiser ã:

---

1. The reduced objective F is 3m-smooth on N.
2. The intersection Mr n is non-empty. We denote this set as SF = Mp n.S, which forms the set of minimisers of F.
Notation In the sequel, we the subscript f, denote all constants associated with the original objective f (e.g. for the smoothness constant, f for the (P)). Similarly, all constants related to the reduced objective F will be subscripted accordingly (e.g. 3F, p.)).
3
Main Results
We now present the main theoretical contributions of this work. Our results formally demonstrate how exploiting known geometric structure optimality via reduction mappings leads to improved smoothness and sharpness properties, which in turn result in improved convergence rates for gradient- based methods. Among the four equivalent sharpness conditions discussed previously, we will, for the purpose of this section, adopt the (MB) condition our reference framework for comparing sharpness constants. When discussing convergence rates, we will switch to the (PE) condition, which is more directly linked to iteration complexity. For clarity of exposition, we organise the results into two parts: smoothness improvements and improvements of the Morse-Bott constant.
3.1 Smoothness Improvement under Affine Reduction Mappings
We first establish that affine reduction mappings. by eliminating alignment with the worst-cas. curvature directions of the original problem. yield strictly improved smoothness constants for the reduced problem. This result is made precise in the following theorem. A noteworthy special case of affine mappings are constant mappings, whose analysis deferred to Appendix A
Theorem (Sharper Smoothness Constant for Reduced Functions with Affine Mappings) Let t - R. be C² function satisfying Assumption oo the compact neighbourhood N. Let V: + R be an affine mapping, and define the reduction mapping & ((z1) ) = (I1, (x1)), the
:
reduced function F, and the feasible manifold MF in Definitions 2 and 3 We consider the local feasible manifold M == N.N.
on be the singular values of V f(x arranged in descending order. Suppose that, for every x E M the largest singular value (denoted mmax) has multiplicity 2 1, with associated dominant subspace Emax.. Assume the following:
Let o1....,0n
1. There exists uniform constant E E (0, 1] such that for all unit vectors v E Emax.
Mg ull S E,
where P PraMpp is the orthogonal projection onto the tangent space T TzMlsc
2. There is a uniform spectral gap:
A Amax
inf MM
[omax (
>
Then, equipping R with the pullback metric induced by the embedding 4 the Riemannian gradient of F is Lipschitz continuous with constant 33 satisfying:
8F < B, -
Amax
(25 (2=
-2) 33.
Proof Sketch of Thcorem The proof relies on viewing the smoothness of the reduced function F as the largest curvature of along the feasible manifold M Since the dominant curvature directions of f are contained in the tangent space of .Mlos the restriction of the Hessian to M exhibits strietly smaller operator norm. By combining this observation with the spectral gap assumption, we obtain a strict improvement in the smoothness constant. The argument is formalised in Appendix
This result establishes that the smoothness constant of the reduced function F strictly smaller than that of the original function f. with the improvement governed by the geometric properties of the
5

---

intersection between the dominant curvature directions and the feasible manifold. These assumptions correspond to generic properties under mild conditions, as discussed Appendices [E.3] and E.4
We now translate this result into the standard Fuclidean setting, where the role of the pullback metric becomes explicit through the following corollary.
Corollar (Euclidean Smoothness Bound under Affine Mappings) Under the setting of Theo- remb / define the Euclidean smoothness constant of the reduced function F as a(F) = sup V Fl.
21
where the derivative of 4 is DΦ (DV) and M = -Amax (u D Then,
3 S ( [3, - Amax (2E - -²)]] < ( By.
In particular, when & is an orthogonal projection, ( = 2, and the sufficient condition ensuring ( F.)
< <By reduces to
Amax (2c =²) 1 3,.
Condition (+) requires the Hessian of f to exhibit significant spectral gap between its largest and subsequent eigenvalues. This scenario corresponds to highly anisotropic curvature, where a small number of dominant directions govern the largest curvatures of the loss landscape. Such spectral structures are commonly observed in wver-parametrised models machine learning, where empirical studies have consistently reported Hessians with few large outlier eigenvalues and bulk of near- zero eigenvalues [66H70]. This phenomenon supports the practical relevance of our condition in many deep learning settings. We note that the degradation observed in the Euclidean smoothness constant in Corollary I arises because the reduction mapping induces a non-Euclidean geometry on the reduced space. The natural metric in this setting the pullback metric induced by the mapping itself, under which the improvement in smoothness is directly captured, as shown in Theorem We will return to this point and its implications for convergence rates in Section
3.2 Smoothness Improvement under Nonlinear Reduction Mappings
We now extend the previous result to the more interesting case where the reduction mapping Vi (x11 nonlinear. In this setting, the feasible manifold Make becomes curved within the ambient space, introducing an additional curvature contribution the reduced function F. This contribution is captured by the correction term C(z1) in the Hessian (see Lemma 3 Intuitively, this term quantifies the bending effect of M and the extent to which the nonlinearity of ! adds curvature to F. When this correction remains sufficiently small relative the spectral gap between the largest curvature directions f and their restriction to M the reduction in the smoothness constant is preserved. This is formalised in the following theorem and the corresponding corollary for the Euclidean case.
Theorem (Sharper Smoothness Constant for Reduced Functions with Nonlinear Mappings) Let f, 4. F, and M be as in Theorem ! except that y (x1) is now a general Ca mapping. Assume:
1. There exist constants Q, Z V zz [21 (a1))l < Z.
| (+))) S Q and such that for all Iı MMles
2. The correction term satisfies curvature gap defined
QZ mite
< ō, where ml = Amin (D D D 1)) and ā is the
8 = Tunax (2 ((0(1))) I (V (V2 (Ta
Then, the reduced funcctton F has a Lipschitz continuous Riemannian gradient with
QZ ml))
3p $ By Amax (2= -E2)
< By.
In particular, when V is orthogonal projection, m (0) 1.

---

Proof Sketch of Theorem 2 The proof extends the affine case by accounting for the additional curvature induced by the nonlinearity V. This is captured by the correction term C(x1) ) in the Hessian of F, which arises due to the curvature of the feasible manifold M By controlling [[C(æ1)]l via bounds on D2 V and v., and ensuring that remains strictly smaller than the curvature gap å obtained from the projection step, we show that the overall smoothness constant of F is still strictly lower than that of f. The result follows by combining these bounds and applying
Weyl's inequality for perturbed operators. Full details are given in Appendix C
Corollary 2 (Euclidean Smoothness Bound under Nonlinear Mappings) Under the setting of Theorem the Euclidean smoothness constant of the reduced function satisfies
8/2) S M( [3 =²)] QZ ( 35.
where ( is the metric distortion factor defined as in Corollary 1 In particular, when Vi is an orthogonal projection, M =2, and the sufficient condition ensuring
$ <g simplifies to
mmxx <²) 2 (3 + QZ).
Reduction mappings often naturally arise in bilevel optimisation settings. where 22 v (211 is implicitly defined as the solution to an inner problem. Depending on the problem structure, these mappings can be affine or nonlinear, and our results apply equally in both cases. We formalise this in the remark below.
Remark (Inner Mappings Argmin Problems) Reduction mappings often arise when Vo defined implicitly as a local solution to an inner optimisation problem
(x1) arg min G(a1, u).
WEC
Under standard regularity conditions, such as constraint qualifications and strict second-order sufficiency (SSOSC), classical sensitivity results ensure that (x11) ) is locally C² [71]. Thus, our previous theorems for affine and nonlinear mappings apply directly in such settings.
To better understand the bounds on the correction term, when the mapping v arises from an inner argmin problem, the constants Q and Z have been explicitly quantified in Theorem in Appendix
3.3 Morse-Bott Constant Improvement under Reduction Mappings
In Appendix D establish that if the original function f satisfies the MBB) property, then the reduced function obtained via reduction mappings also satisfies the (MB) property, albeit potentially with a different constant. In the following theorem, we strengthen this result by showing that the (MB) constant of the reduced problem is in fact strictly improved compared to that of the original problem.
Theorem (Strict Improvement the Morse-Bott Constant under Smooth Reduction) Let
f: R + R be C2 function where the solution manifold S satisfies the μe JMBperorrtty within compact neighbourhood N. Let V R. - =2 be a C mapping and define the reduced function F(x,) f(11 (x1)), with local feasible manifold M -M N.N
At each s M let Hg denote the restriction of f()()) N₂S. Assume:
1. The smallest eigenvalue Auin (Ha) has multiplicity m, with eigenspace Emin- and for all U Emin. ||
M v/l < 11-6,
holds uniformly for some E E (0,
2. The spectral gap Amin inf Mep
(Dn-m(H)) - Amin (Hz)) > 0.
Then, for the
MB property of the reduced function F, obtain strictly improved constant
#F 2 u + Amin (2x = uf.
7

---

Proof The result follows by applying the same geometric argument as in Theorem | now to the positive definite Hessian H3 restricted N₃S. Since we operate entirely within the normal space and on the solution manifold, only eigenvalues and eigenspaces matter, and the nonlinearity of v has no effect.
In the Euclidean setting, the improvement in the Morse-Bool constant is further scaled by the pullback metric. Specifically, since the Euclidean constant is related the intrinsic pullback constant via (P) 2 mm!/)FF u.F. the distortion introduced by the metric, through its smallest eigenvalue m > 1, amplifies the effective (MB) constant. This shows that the pullback metric does not only improve the intrinsic condition number but also results in a better constant when measured under the Euclidean metric. In particular, when V is a projection mapping, we have m 1. and the Euclidean result coincides exactly with the pullback metric case.
Remark 2 (Morse-Bott Constant Equivalence (64) Since in Theorem 3 we showed that pF K for the MB) property, by equivalence it follows that the constants μu for the (PL). [B and (QG conditions will follow the same strict inequality.
4 Convergence Rate Results and Discussion
In this section, we discuss the algorithmic implications of our geometric analysis of reduction mappings. In particular, we focus on the impact of the improved smoothness and (MB) and by equivalence (PE constants obtained in Theorems and 3 on the convergence behaviour of first-order methods.
When the reduced problem is equipped with the pullback metric induced by the reduction mapping Φ our results show that the condition number of the reduced function strictly improved relative to that of the original problem f. This theoretical gain directly translates into faster local linear convergence rates when applying geometrically preconditioned gradient descent (GeoPrecGD) on the reduced objective. We formally state this result below. A broader discussion on metric choices, algorithmic implications, and practical considerations follows.
Corollary (Faster Linear Convergence of the Reduced Function) Under the settings of Theo- equipping Th& with the pullback metric induced by t yields strictly improved condition number for the reduced problem
rems 2 and 3
BF
- 16g-
B1
KFF
#1
As result, preconditioned gradient descent applied to F achieves a strictly faster local linear convergence rate under PL condition compared to gradient descent on f (5]. Specifically:
The rate factor improves from O(exp(-t/nı)) O(exp(-t/rE)),
The iteration complexity to achieve accuracy e iimproves from Ols log(1 (o)( to
O (og(/e))(1/))))
Below, we make explicit the form of the geometrically preconditioned gradient descent algorithm applied to the reduced problem F under the pullback metric induced by db
1++1)
(t) R V 2
with R = D D
(GeoPrecGD)
This method performs steepest descent under the pullback metric, aligning the descent directions with the intrinsic geometry induced by the reduction mapping fundamental principle in optimi- sation [72]. While preconditioning introduces additional computational cost due to the inversion of R, such overhead can often be mitigated by exploiting the structure of Rt shown in prior 52 51] In particular, for affine mappings. R constant and cheap to apply, whereas for nonlinear mappings, structured approximate solvers be used. Designing efficient implementations these methods is primarily mapping dependent and beyond the scope of this work, where our focus is the iteration complexity benefits arising from the improved conditioning.
works 733 74 50
75
76
instructive special case is when V is an orthogonal projection; common choice for many reparametrisations. If V is affine, then R simplifies, and its inverse is given by

---

--'=1-1DV" DV, implying no additional computational overhead. More generally, if y is nonlinear but still defines an orthogonal projection onto a manifold, the situation remains favourable. particularly near convergence in the defined neighbourhood. As iterates approach the solution mani- fold, the derivative D & locally behaves as a linear orthogonal projection onto the tangent space of the solution manifold, with the condition number of R remaining close to two. This favourable and stable spectrum near the solution means the preconditioned linear system within the (GeoPrecGD) step (Rz- VF) can be solved efficiently. Specifically, iterative solvers (e.g.. Conjugate Gradient) applied to this system benefit from rapid convergence due to the low condition number. Also, the structure of R allows for efficient and cheap inversion via the Woodbury formula, depending
whether 71 > f2 or vice versa.
We validate our theory with synthetic experiments in Appendix G Notably, for quadratic objectives, our geometrically preconditioned gradient descent equivalent to the full Newton method on the reduced problem; and for nonlinear least-squares, it is equivalent to the Gauss-Newton method applied to the reduced problem. Thus, our approach recovers Newton and Newton as special cases while extending beyond them to more general settings.
While preconditioning necessary and recommended [53] fully exploit the improved geometry induced by the reduction mapping, improvements can still be observed under Euclidean gradient descent. Specifically, if the condition holds, we still have ((.²) < By. leading a better condition number even without preconditioning. Even this condition fails, the improvement in the sharpness (F) constant pe y may still result in better overall condition number K'F" compared to ky- Thus, gains are still possible, although they may be limited.
If practitioner desires additional control over the condition number in Euclidean space for con- vergence guarantees, the reduction mapping can be modified induce approximate isometries via simple small-slope method. This approach is general and may warrant more systematic study; however, a full investigation of such strategies lies outside the scope of this paper. We sketch the idea below.
Remark (Slpe Method and Approximate Isometry) When using reduction mappings,
the pullback metric takes the form of R = D potential degradation in the Euclidean smoothness constant proportional to Amax (R). To mitigate
D Th D V, which in general is not identity, leading to
this, we propose a simple small-slope method, where the mapping is rescaled as Vp (z1) = a (x))
D V, whose condition number for a small a > This modifies the metric to Ro = a D L becomes
1 Amax (D D KC <(Ra)
1+ a2 (D D
Thus by making sufficiently small, R., becomes approximately isometric, improving the likelihood that 3(E) S By. This method offers a simple yet effective trade-off between improving conditioning and preserving geometric fidelity. Notably, if the geometric structure encoded by V is not scale- invariant, excessively small a may distort the mapping, potentially degrading its ability to capture the correct geometry of the Z2 variables. This trade-off off is less critical in many common cases susc as subspaces, cones, or orthogonal structures where the geometry is inherently maale-invariant. The parameter be treated as user-defined knob, balancing conditioning improvement against the precision of the reduction.
4.1 Limitations and Future Directions
Our analysis assumes that the reduction mapping V can be evaluated exactly. When is defined implicitly, this idealisation neglects the approximation errors that may arise in practice, potentially introducing bias in the reduced gradients and weakening the theoretical convergence guarantees. Moreover, the computational cost of evaluating V (211) is inherently problem-specifie and is not explicitly captured in the iteration complexity analysis. A rigorous treatment that jointly considers the benefits of reduction mappings and their evaluation cost remains open, problem-dependent direction for future work.
Beyond these considerations, our analysis is confined to local properties and deterministic first- order methods. Extending the theory to global convergence settings, particularly in nonconvex landscapes, is an important future direction. In this context, it would be valuable to explore how

---

reduction mappings reshape the landscape globally, including their potential to eliminate or introduce saddle points and spurious minima, extending the framework of [4T]. Another promising direction is applying our geometric framework to specific structured problems where the objective admits compositional forms, such as matrix factorisation, nonlinear regression, or neural network training, to obtain practical insights. Finally, extending the analysis to stochastic settings, for example by studying stochastic gradient descent (SGD) under reduction mappings and characterising its regularity conditions, would enhance the relevance of our results in large-scale applications.
5 Conclusion
In this work, we presented a unified geometric framework for analysing reduction mappings in optimisation problems with structured solution sets. By explicitly incorporating mappings that encode known geometric structures at optimality, we showed that the resulting reduced problems exhibit strictly improved smoothness and sharpness properties. This leads to enhanced local condition numbers and provably faster convergence rates when applying appropriately preconditioned first-order methods. Our analysis generalises seamlessly from affine nonlinear mappings, carefully accounting for the additional curvature induced by the bending of the feasible manifold. While the improvements fundamentally intrinsic the pullback geometry, we further showed that, under appropriate constructions and trade-offs, these gains can also manifest in the Euclidean metric. Throughout, we emphasised the generality and flexibility of our framework, illustrating that reduction mappings may be given explicitly or arise implicitly as solutions to inner optimisation problems, thereby connecting our approach to classical bilevel and composition formulations via implicit differentiation. We believe this geometric perspective offers a principled and broadly applicable toolset for designing more efficient optimisation algorithms that better exploit problem structure, with potential impact across areas such as matrix factorisation, deep learning, and other structured nonconvex problems.
References
[1] A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun, "The loss surfaces of multilayer networks," in Artificial intelligence and statistics, pp. 192-204, PMLR, 2015.
[2] K. Kawaguchi, "Deep learning without poor local minima," Advances in neural information processing systems, vol. 29, 2016
[3] I. Panageas and G. Piliouras, "Gradient descent only converges minimizers: Non-isolated critical points and invariant regions," arXiv preprint arXiv:1605.000400 2016.
[4] S. S. Du, X. Zhai, B. Poczos, and A. Singh, "Gradient descent provably optimizes over- parameterized neural networks," arXiv preprint arXiv:1810.02055 2018
[5] B. T. Polyak, "Gradient methods for minimizing functionals." Zhurnal vychislitel noi matematiki i matematicheskoi fiziki, vol. 3, no. 4, pp. 643-653, 1963.
[6] C. Liu, L. Zhu, and M. Belkin, "Loss landscapes and optimization in over-parameterized non- linear systems and neural networks," Applied and Compulational Harmonic Anulywis, vol. 59, pp. 85-116, 2022. Special Issue on Harmonic Analysis and Machine Learning.
[7] R. Bassily, M. Belkin, and S. Ma, "On exponential convergence of sgd in non-convex over- parametrized learning," arXiv preprint
2018.
[8] H. Karimi. J. Nutini, and M. Schmidt, "Linear convergence of gradient and moximal-gradionn methods under the pojaak-tojssiewicz condition," " in Joint European conference on machine learning and knowledge discovery in databases, pp. 795-811, Springer, 2016.
[9] P.-A. Absil, R. Mahony, and B. Andrews, "Convergence of the iterates of descent methods for analytic cost functions." SIAM Journal on Optimization, vol. 16, no. 2, pp. 531-547, 2005.
[10] Y. Cooper, "Global minima of overparameterized neural networks." SIAM Journal on Mathe- matics of Data Science, vol. 3, no. 2, pp. 691, 2021.
[11] R. Entezari, Sedghi, O. Saukh, and B. Neyshabur, "The role of permutation invariance in linear mode connectivity of neural networks," arXiv preprint arXiv:2110.060 2021.
10

---

[12] L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio, "Sharp minima can generalize for deep nets," in International Conference on Machine Learning. pp. 1019-1028, PMLR, 2017.
[13] B. Neyshabur, R. Salakhutdinov, and N. Srebro, sgd:h-sgd: narhhnormalized optimization in deep neural networks," Advances in neural information processing systems, vol. 28, 2015.
[14] V. Papyan, X. Y. Han, and D. L. Donoho, "Prevalence of neural collapse during the terminal phase of deep learning training." Proceedings of the National Academy of Sciences, vol. 117, no. 40, pp. 24652-24666 2020.
[15] P. Súkeník, M. Mondelli, and C. H. Lampert, "Deep neural collapse is provably optimal for the deep unconstrained features model," Advances Neural Information Processing Systems, 2023.
vol. 36, pp.
[16] P. Súkeník, C. Lampert, and M. Mondelli, "Neural collapse vs, low-rank bias: Is deep neural collapse really optimal?," in 38th Annual Conference on Neural Information Processing Systems, vol. 38, 2024.
[17] C. Garrod and J. P. Keating, "The persistence of neural collapse despite low-rank bias: An analytic perspective through unconstrained features," " arXiv preprint arXiv:2410.23169, 2024.
[18] A. Rangamani, M. Lindegaard, T. Galanti, and A. Poggio, "Feature learning deep classifiers through intermediate neural collapse." in Proceedings of the 40th International Conference on Machine Learning (A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, eds.), vol. 202 of Proceedings of Machine Learning Research, pp. 28729-28745, PMLR, 23-29 Jul 2023.
[19] Tirer and J. Bruna, "Extended unconstrained features model for exploring deep neural collapse," in International Conference on Machine Learning, pp.
PMLR, 2022.
[20] M. Fickus, J. Jasper, E. J. King, and D. G. Mixon, "Equiangular tight frames that contain regular simplices," Linear Algebra and its applications, vol. 555, pp. 98-138, 2018
[21] Y. Chi, Y. M. Lu, and Y. Chen, "Nonconvex optimization meets low-rank matrix factorization: An overview," IEEE Transactions on Signal Processing, vol. 67, no. 20, pp. 5239-5269, 2019.
[22] J. Sun, Q. Qu, and J. Wright, "Complete dictionary recovery over the sphere i: Overview and the geometric picture," " IEEE Transactions on Information Theory, vol. 63, no. 2, pp. 853-884, 2016.
[23] J. Sun, Q. Qu, and J. Wright, "Complete dictionary recovery over the sphere ii: Recovery by riemannian trust-region method," IEEE Transactions on Information Theory, vol. 63, no. pp. 885-914, 2016.
[24] D. Gilboa, S. Buchanan, and J. Wright, "Efficient dictionary learning with gradient descent," in International Conference on Machine Learning, pp. 2252-2259, PMLR, 2019.
[25] Q Qu, Y. Zhai, X. Li, Y. Zhang, and Zhu, "Analysis of the optimization landscapes for overcomplete representation learning," arXiv preprint arXiv:1912.02447 2019.
[26] R. Ge, C. Jin, and Y. Zheng, "No spurious local minima in nonconvex low rank problems: A unified geometric analysis," International conference on machine learning, pp. 1233-1242, PMLR, 2017.
[27] R. Ge, J. D. Lee, and T. Ma, "Matrix completion has no spurious local minimum," Advances in neural information processing systems, vol. 29, 2016.
[28] R. Ge and T. Ma, "On the optimization landscape of tensor decompositions." Advances in neural information processing systems, vol. 30, 2017.
[29] R. Ge, F. Huang, C. Jin, and Y. Yuan, "Escaping from saddle points onlite-online stochastic gradient for tensor decomposition," in Conference on learning theory, pp. 797-842, PMLR, 2015.
[30] Y. Zhang, Q Qu, and J. Wright, "From symmetry to geometry: Tractable nonconvex problems," arXiv preprint arXiv:2007.0677 0665000777766..... 2020.

---

[31] J. Candes, Y. C. Eldar, T. Strohmer, and V. Voroninski, "Phase retrieval via matrix completion." SIAM review, vol. 57, no. 2, pp. 225-251, 2015.
[32] E. J. Candes, X. Li, and M. Soltanolkotabi, "Phase retrieval via wirtinger flow: Theory and algorithms," IEEE Transactions on Information Theory, vol. 61, no. 4, pp. 1985-2007, 2015.
[33] J. Sun, Q. Qu, and J. Wright, "A geometric analysis of phase retrieval," Foundations of Compu- tational Mathematics, vol. 18, pp. 1131-1198, 2018.
[34] A. Fannjiang and T. Strohmer, "The numerics of phase retrieval," " Acta Numerica, vol. 29, pp. 125 228, 2020.
[35] S. Ling and T. Strohmer, "Blind deconvolution meets blind demixing: Algorithms and per- formance bounds," IEEE Transactions on Information Theory, vol. 63, no. 7, pp. 4497-4520, 2017.
[36] Y. Zhang, H.-w. Kuo, and Wright, "Structured local minima in sparse blind deconvolution," Advances in neural information processing systems, vol. 31, 2018.
[37] H.-W. Kuo, Y. Lau, Y. Zhang, and J. Wright, "Geometry and symmetry in short-and-sparse deconvolution," in International Conference Machine Learning, pp. 3570-3580, PMLR, 2019.
[38] Y. Lau, Q. Qu, H.-W. Kuo, P. Zhou, Y. Zhang, and J. Wright, Shhor-and-spant deconvolution-a geometric approach." " arXiv preprint 1908. 10959, 2019.
[39] Y. Li and Bresler, "Global geometry of multichannel sparse blind deconvolution on the sphere," Advances in Neural Information Processing Systems, vol. 31, 2018.
[40] Q. Qu, X. Li, and Z. Zhu, "A nonconvex approach for exact and efficient multichannel sparse blind deconvolution," Advances in neural information processing systems, vol. 32, 2019.
[41] E. Levin, J. Kileel, and N. Boumal, "The effect of smooth parametrizations on nonconvex optimization landscapes," Mathematical Programming, vol. 209, no. 1, pp. 63-111, 2025..
[42] S. loffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," " International conference on machine learning, pp. 448-456, pmir, 2015.
[43] J. L. Ba, J. R. Kiros, and G. E. Hinton, "Layer normalization," arXiv preprint arXiv: 160..v1607.004500 2016.
[44] S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, "How does batch normalization help optimiza- tion?," Advances in neural information processing systems, vol. 31, 2018
[45] 7. Zhu, T. Ding, J. Zhou, X. Li, C. You, J. Sulam, and Q. Qu, "A geometric analysis of neural collapse with unconstrained features," Advances in Neural Information Processing Systems, 2021.
vol. 34, pp.
[46] E. Markou, T. Ajanthan, and S. Gould, "Guiding neural collapse: Optimising towards the nearest simplex equiangular tight frame," Advances in Neural Information Processing Systems, vol. 37, pp. 35544-35573, 2024.
[47] P.-. Absil, R. Mahony, and Sepulchre, Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008.
[48] N. Boumal, An introduction to optimization on smooth manifolds. Cambridge University Press, 2023.
[49] S.-i. Amari, "Natural gradient works efficiently in learning." Neural Computation, vol. 10, no. pp. 251-276, 1998.
[50] G. Zhang, J. Martens, and R. B. Grosse, "Fast convergence of natural gradient descent for over- over-parameterized neural networks," Advances in Neural Information Processing Systems, vol. 32, 2019.
12

---

[51] V. Gupta, T. Koren, and Y. Singer. "Shampoo: Preconditioned stochastic tensor optimization," International Conference on Machine Learning, pp. 1842-1850, PMLR, 2018.
[52] K. Jordan, Y. Jin, V. Boza, You, F. Cesista, L. Newhouse, and J. Bernstein, "Muon: An optimizer for hidden layers in neural networks," 2024.
[53] A. Kristiadi, F. Dangel, and P. Hennig, "The geometry of neural nets' parameter spaces under reparametrization." Advances in Neural Information Processing Systems, vol. 36, pp. 17669- I 17688, 2023.
[54] D. P. Bertsekas, "Nonlinear programming." Journal the Operational Research Society, vol. 48, no. 3, pp. 334-334, 1997.
[55] S. Gould, B. Fernando, A. Cherian, P. Anderson, R. S. Cruz, and E. Guo, "On differentiating parameterized argmin and argmax problems with application to bi-level optimization," arXiv preprint arXiv:1607.05447 2016.
[56] J. F. Bonnans and A. loffe. "Second-order sufficiency and quadratic growth for nonisolated minima," Mathematics of Operations Research, vol. 20, no. 4, pp. 801-817, 1995.
[57] Z.-Q. Luo and Tseng. "Error bounds and convergence analysis of feasible descent methods: general approach," Annals of Operations Research, 1993.
[58] M. Anitescu, "Degenerate nonlinear programming with quadratic growth condition," SIAM Journal on Optimization, vol. 10, no. 4, pp. 1116-1135, 2000.
[59] I. Necoara, Y. Nesterov, and F. Glineur, "Linear convergence of first order methods for non- strongly convex optimization." Mathematical Programming, vol. 175, pp. 69 19-107,200 2019.
[60] J. Liu and S. J. Wright, "Asynchronous stochastic coordinate descent: Parallelism and conver- gence properties," SIAM Journal on Optimization, vol. 25, no. 1, pp. 351-376, 2015.
[61] D. Drusvyatskiy and A. S. Lewis, "Error bounds, quadratic growth, and linear convergence of proximal methods," Mathematics of Operations Research, vol. 43, no. pp. 919-948, 2018.
[62] R. Bott, "Nondegenerate critical manifolds," Annals of Mathematics, vol. 60, no. pp. 248-261, 1954.
[63] R. Bott, "Lectures on Morse theory, old and new," Bulletin (New Series) of the American Mathematical Society, vol. 7, no. 2, pp. 331 358, 1982.
[64] Q Rebjock and N. Boumal, "Fast convergence to non-isolated minima: four equivalent condi- tions for C² functions," Mathematical Programming, 09 2024.
[65] Milnor, M. SPIVAK, and R. WELLS, Morse Theory. (AM-5I), Volume 51. Princeton University Press, 1969.
[66] V. Papyan, "Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians," arXiv preprint 3rXiv:1901.08244: 2019.
[67] V. Papyan, "Traces of class/cross-class structure pervade deep learning spectra," " Journal of Machine Learning Research, vol. 21, no. 252, pp. 1-64, 2020.
[68] L. Sagun, L. Bottou, and Y. LeCun, "Eigenvalues of the hessian in deep learning: Singularity and beyond," arXiv preprint arXiv: (o:::111177000 1.07476, 2016.
[69] L. Sagun, U. Evei, V. U. Guney, Y. Dauphin, and L. Bottou, "Empirical analysis of the hessian of over-parametrized neural networks," arXiv preprint arXiv:1706. 04454, 2017.
[70] B. Ghorbani. S. Krishnan, and Y. Xiao, "An investigation into neural net optimization via hessian eigenvalue density," in International Conference on Machine Learning, pp. 203222222222 22411, PMLR, 2019.
[71] Bonnans and A. Shapiro, Perturbation analysis of optimization problems. Springer Science & Business Media, 2013.
13

---

[72] J. Bernstein and L. Newhouse, "Old optimizer, new norm: An anthology," arXiv preprint arXiv:2409.20325, 2024.
[73] X. Xu, Y. Shen, Y. Chi, and C. Ma, "The power of preconditioning overparameterized low-rank matrix sensing," in International Conference Machine Learning, pp. 3861 1-38654, PMLR, 2023.
[74] T. Cai, R. Gao, J. Hou, S. Chen, D. Wang, D. He, Z. Zhang, and L. Wang, "Gram- "Gram-gauss-nee gauss-newton method: Learning overparameterized neural networks for regression problems," arXiv preprint arXiv: 1675, 2019.
[75] G. Zhang, L. Li, Z. Nado, J. Martens, S. Sachdeva, G. Dahl, C. Shallue, and R. B. Grosse, "Which algorithmic choices matter at which batch sizes? insights from noisy quadratic model," Advances in neural information processing systems, vol. 32, 2019.
[76] T. Tong, C. Ma, and Y. Chi, "Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent," Journal of Machine Learning Research, vol. 22, no. 150, pp. 1-63, 2021.
[77] G. Andriopoulos, Dong, L. Guo, Z. Zhao, and K. Ross, "The prevalence of neural collapse in neural multivariate regression." arXiv preprint arXiv:2409.04188 2024.
[78] Nocedal and S. Wright, Numerical optimization. Springer series in operations research and financial engineering, New York, NY: Springer, ed. ed., 2006.
[79] J. Martens and R. Grosse, "Optimizing neural networks with kronecker-factore approximate curvature," International conference on machine learning, pp. 2408 2417, PMLR, 2015.
[80] Kato, Perturbation Theory for Linear Operators. Classics in Mathematics, Springer Berlin Heidelberg, 1995.
[81] M. Dana and K. D. Ikramov, "On the codimension of the variety of symmetric matrices with multiple eigenvalues," Journal of Mathematical Sciences, vol. 137, no. 3, pp. 4780-4786, 2006.
[82] A. Sard, "The measure of the critical values of differentiable maps," Bulletin of the American Mathematical Society, vol. 48, 12, pp. 883 890, 1942.
[83] M. Hirsch, Differretiaal Topology. Graduate Texts in Mathematies, Springer New York, 1997.
[84] I.. Hörmander, The analysis of linear partial differential operators III: Pseudo-differentia operators. Springer Science & Business Media, 2007.
[85] E. Bao and K. Zhu, "Computable, obstructed morse homology for clean intersections." arXiv preprint arXiv:2409.11565, 2024.
14

---

A A Gentle Start
To build intuition for the broader theoretical developments in this work, we begin with two illustrative examples. Each example is carefully chosen to highlight different geometric structure of the minimisers: a function with a unique isolated minimum and function with a continuum of non- isolated minima. For each case, we examine the curvature of the landscape through the lens of the Hessian and its maximal eigenvalue, offering insight into the local geometry.
We then apply range of reduction mappings to these examples- designed to restrict the optimisation
lnwer-dimensional subspace or manifold observe how these mappings affect the curvature of the objective function. Of particular interest is how the maximum curvature behaves under these reductions, as this has direct implications for the conditioning of the problem and the performance of iterative methods.
The section concludes with focused analysis of constant mappings, special class where the reduction ignores part of the variable space entirely. This case serves as a useful analytical baseline, offering contrast structured mappings that actively exploit the problem geometry, whereas constant mappings remain agnostic to how this structure influences the reduced variables.
A.1
Example 1: Single Isolated Minimum
Consider the function f IR2 IR defined by
x211 = M(x -21)2
(6)
where M > 1 is large constant. This function exhibits strong anisotropy: the quadratic term in 22 introduces high curvature along the 22 -irrirect. As a result, the Lipschitz constant of the gradient (ie., the smoothness constant) By is on the order of M, reflecting the steep variations in the landscape. We note that this function is used in Figure I with M = 2 for ease of visualisation.
To make this explicit, we compute the gradient and Hessian of f as follows
v ff()))22) 12m1 + 2M
( 122)
122)
22 2M
22
V²f(I1,22)
(7)
2M =x1)
-2M
2M
The eigenvalues of the Hessian are given by
(2+ 4M) 1 v (2 + 4M 16M
(8)
2
If one instcad fixes 22 via a constant mapping that lies on the solution manifold, the effective maximum curvature in the 2 z1-direction becomes 2 2M, since 22 is no longer allowed to adapt to changes in TT, and the mismatch induces high curvature.
Now consider an affine mapping, which can arise from bilevel setting such that
un (21) = arg min [(11, 22)
22
For the function above, it is easy to verify that the optimal inner solution satisfies a}(xn) = 211 Along the reduced trajectory (I1, x5 (x1)) = (x1, the function simplifies to
F(xi)= = f =5x1) = 27
(10)
In this formulation, the steep curvature contributed by the term M - )2 is eliminated entirely, and the resulting function F has smoothness constant that is independent of M.
This example illustrates that allowing Z2 to adjust optimally can "iron out" steep variations that would otherwise hinder optimisation. However, this is true only when the mapping is well-designed. To see that not reductions lead to improved curvature, consider instead a nonlinear mapping given by
x2(x1)= 21 2 sin(
(11)
Under this mapping, the reduced function becomes
F(x1) . + 4M sin sin? (x1).
(12)
15

---

The curvature of this reduced function is now influenced by the oscillatory term sin (21). and its second derivative includes component proportional 4M cos(2x1). leading increased curvature compared to the original function f.
This contrasting mappings can introduce new sources of curvature rather than mitigate them. Hence, curvature reduction is not an automatic consequence of reparametrisation-- depends critically on the mapping being well-aligned with the geometry of the solution manifold and its effect on the optimisation trajectory. stru-stre settings, such when =21) derived from minimising over 22, the resulting outer function F can exhibit substantially improved curvature characteristics.
makes clear that not all mappings beneficial: poorly chosen misaligned
In Figure 2
plot the curvature
(x)
<(x)=
(13)
[1 + (f(
over the domain x [-0.5, 0.5] with = 10 for the reduced cases. For comparison, we also indicate the corresponding maximum curvature value (i.e., the largest eigenvalue of the Hessian) for each case, including the original 2D function.
Unconstrained
80
max K
82
Fixed
Linear Nonlinear
60
Curvature
max 2 41
40
max 22
20
max
0
0.6
0.5
-0.4
0.3
-0.2
0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Ø1
Figure 2: Curvature profiles for three reduction mappings. The full (unconstrained) function f(21. x2) x 10(x² =2 exhibits high curvature (approximately 41). Fixing 22 = 0 yields ((xe = 11x;, with curvature 22 at = 0. The bilevel reduction Pincar(z1) = Φ has lower curvature (2 at .i = 0), while the nonlinear mapping induces strong oscillations and increases the maximum curvature to 82.
A.2 Example 2: Non-Isolated Minima
Consider the function f IR2 - I. defined by
(m11 (0(1) + (x2 - sin(x1))
(14)
where the flat-bottom quartic is defined as
(a) -a),
21 a, 21 E [a, 8],
((x1) =
0,
(15)
(i) - -8),
21 > 3.
The gradient of / is given by
(4'(z)) - 2(x2 sin( cos(x1))
(16)
2(x² - sin(x1))
16

---

where
4(01 - a)²
xn < a
((1))=
0,
11 E |a $].
(17)
4(20) - B)s,
3 > B.
The Hessian is
+ - sin(x1)) (innzi)))n((1) + 2 cos
-2cos(x1)
(18)
-2 c0s(2x1)
where
12(x1 - a)); 0, 112(x) - B)2
11 < a I1 E |a, 3]
(x1
(19)
21 > 3.
Since the Hessian is symmetric, its eigenvalues are real and given by
x= (a (a+ct v(a - c)2 4b2)
(20)
where = V2 Sazz J.b= V 21 -2 cos(x1), and Zgaz S = 2. These expressions follow from the standard characteristic equation for symmetric 2 matrices. In our analysis, we record the largest eigenvalue of the Hessian restricted to the region
V2
[a, B), where v"(x1 0.
We consider three reduction mappings V R + R. each inducing a reduced function (( f V. (z))).. The quality of each mapping can be assessed by the curvature of the resulting reduced function, particularly over the flat region [a, (3), where (x1x1) 0.
Nonlinear (Implicit) mapping: V/ (x1) = sin(x1). This choice corresponds to the exact solution of the inner minimisation problem
1 (x1) arg min (x1,
(21)
and cancels the coupling term, i.e., (x2 - sin(21)) = 0. The resulting reduced function is (x = (xx))= p(x)),
(22)
which is piecewise quartic with flat basin over [ax, 3). Since both the first and second derivatives vanish in that interval, Fi has zero curvature there. This mapping represents the ideal scenario for dimensionality reduction, as it perfectly aligns with the optimal structure of the original function.
Fixed mapping: (2(21) 0.
This mapping treats 22 constant and neglects the structure the inner problem. The reduced function becomes
F2(21) = ((m) ) + sxn)
(23)
where the additional term sin 2 (x1) introduces artificial curvature even inside the flat region, due to the mismatch between the fixed choice x2 = 0 and the optimal one 22 = sin(#1) While simple to implement, this mapping results in highly curved landscape that can
significantly hinder optimisation.
Linear mapping: (2((1) I1. This mapping is a linear approximation to the sine function near the origin and yields
F((z)) = p(x,) + (x1 - sin(
(24)
Inside the interval [a:, 3]. 6" = 0, but the additional term (z) - -ii((inn)) introduces nonzero curvature. However, unlike the fixed mapping, the misalignment with sin(21) smaller, especially near the origin. Thus, F3 achieves lower curvature than F2, but does not eliminate coupling completely. It represents a computationally cheap, yet structurally-infor approximation.
These three reductions are visualised in Figure where each curve lies on the surface of and projects to its corresponding manifold Mr, in the base plane. Figure plots the curvature k(z)) of each reduction and confirms that the implicit mapping achieves the smallest curvature, which vanishes on [a, 3). where we have picked -0.5 and 8 = 0.5.
17

---

Ambient Surface of f
f(21, 02) p(x1) + (22 sin()))))))
Fi f(m1 (1(x1)
Mrj {(m),in((x)) : R)
(s( f(n, Va(z1))
Mny = ((01,0) 121 :2) ER)
(( (s(e))
MFs :21
S
3/2
21
Figure Visualisation of the ambient function ((x1, ((x1)) (x2 (22=-sn(z1)) with y flat-bottom quartic, and three lifted restriction curves corresponding to different reduction mappings v (21). ). The black curve S denotes the global minimisers of f. forming a non-isolated solution manifold parametrised by æ1 E [-0.5, 0.5| and sin(x1). Each lifted curve lies on the surface f. and the dashed curves on the base plane show the image of each corresponding manifold My.-
A.3 Constant Mappings: A Special Case
As special case of our reduction framework, consider a constant mapping of the form
V (x1) = Ø2
for some fixed Ø2 E R
(25)
The reduced function then
F(x1) (x1,
(26)
with reparametrisation map ((x1) = (2), whose Jacobian satisfies
D (x11) = [n (m) (2))))
[ = Ing
(27)
This isometric property ensures that the geometry the reduced space is Euclidean, and hence optimisation proceeds without distortion or the need for metric preconditioning.
Although the mapping is static, it completely eliminates the sy-directions. If the original function f exhibits strong curvature in those directions, the reduced function F may be significantly better conditioned. In this way, constant mappings can yield nontrivial curvature reduction, even without adapting to the structure of f as varies.
Under standard assumptions (e.g., smoothness and PL conditions), the reduced function F can inherit the same convergence guarantees as affine mappings, provided the non-tangency condition and a uniform spectral gap hold. The key advantage here is that these guarantees apply directly in the Euclidean setting, without requiring any geometric correction.
The main limitation of constant mappings lies in their lack of adaptivity. Since the mapping is fixed, it cannot adjust to the optimal choice of (211 as ø1 evolves during optimisation. In settings where Ii and 22 tightly coupled, this lead to misalignment between the reduced function F(x1) = (I1, 52) and the true solution manifold of f. While 22 may belong to the global minimiser set of f. it need not remain optimal for all values of T1. As result, F' may fail to faithfully reflect the structure of f away from neighbourhoods where (I1, .22) E S, potentially slowing convergence yielding suboptimal solutions.
This scenario frequently arises overparametrised models with symmetric minimiser sets. For instance, in deep neural networks, the optimal classifier weights often lie in the set of Equiangular Tight Frames (ETFs). A constant mapping implicitly selects a particular representative 32 this set and holds it fixed throughout training. While this preserves global optimality in principle, it breaks
18

---

max R.
Unconstrained Fixed
Linear Nonlinear
3
Curvature
mag F.
max 2.22
max
1.5
0.5
0.5
1
1.5
Figure 4: Curvature comparison across different reduction mappings applied to the function f(1 x22 (0(11) (I2 sin(x1))² where oP is flat on [-0.5, 0.5] and quartic outside. The black dashed line represents the maximum eigenvalue of the full Hessian when no reduction is applied. The blue curve shows the curvature induced by fixing #2 O, resulting in high curvature due to mismatch with the optimal
sin(x1). The red curve corresponds to the linear reduction 22 = 21, which aligns more closely with sin(x1) and yields lower curvature. The green curve represents the bilevel (implicit) reduction function inside the flat region, driving curvature to zero. Each dashed horizontal line indicates maximum curvature for its corresponding method.
sin(2) (3) ). which eliminates coupling and flattens the
the symmetry in way that may be incompatible with the evolution of The reduced function F thus becomes a projection of onto a fixed slice of the solution manifold, which may not be well-aligned with the optimisation trajectory unless additional structure coordination is imposed.
In summary, constant mappings offer geometrically clean and computationally simple reduction strategy. They can yield strong local convergence when well-aligned with the solution structure, and are especially attractive in models where redundant variables can be safely eliminated. However, their rigidity makes them less suitable for problems requiring global coordination between variables or adaptivity along the optimisation path. In practice, they serve as a valuable analytic baseline and an effective modelling choice when the structure of the solution is known or can be fixed priori.
B Extended Related Work
Reparametrisations widely adopted to exploit problem structure and improve the conditioning of nonconvex optimisation problems [41]. Our framework offers a unified lens to view these diverse approaches special cases of reduction mappings--hether constant, affine, nonlinear, explicitly defined, or arising implicitly-and systematically analyses their effect on problem conditioning and the convergence first-order methods. Below, outline several representative cases.
Normalisation and induced feature geometry. A canonical example is normalisation layers, such as batch normalisation [42] or layer normalisation [43), which perform explicit nonlinear mappings that constrain the scale and centring of activations parameters. These reparametrisations have been shown to accelerate convergence by implicitly altering the smoothness and sharpness constants [44). In our framework, this can be interpreted introducing reduction mapping T that enforces fixed feature geometry on specific variables, such as unit variance or zero mean.
Neural collapse and ETF reparametrisations. the context of neural collapse, our framework naturally captures both prevalent approaches. Fixing the classifier to equiangular tight frame (ETF)
19

---

corresponds to a constant reduction mapping that constrains the classifier parameters to predefined geometric structure 44 Alternttvvly. dynamically solving for the nearest ETF given the current features has been shown to accelerate convergence [46]. and fits within our framework as a nonlinear or implicitly defined mapping. enforcing optimal alignment with the feature space. Such reduction mappings can also been extended to regression settings [77]. In all cases, our analysis applies directly, providing theoretical guarantees on improved conditioning and convergence.
Optimisation on quotient manifolds and gauge fixing. In optimisation over quotient mani- folds [47 [8), the goal is to account for known invariances by operating directly on equivalence classes, such optimising over the Grassmannian or Stiefel quotient manifolds. While these ap- proaches formally respect the problem's 's intrinsic symmetries, they can incur significant per-iteration costs due to expensive manifold operations like projections and retractions. Our framework provides an alternative by enabling explicit gauge fixing via reduction mappings, where one selects a repre- sentative from each equivalence class example, fixing scaling or orthogonal symmetry through mechanism
a carefully designed mapping V. This strategy can be viewed as (ower-dimensional space equipped with the pullback
that reduces the optimisation problem metric induced by the reduction mapping. This allows standard first-order methods to operate more efficiently while still respecting the problem's 's local geometry. Thus, our approach can potentially offer a lightweight and practical alternative to quotient manifold optimisation, particularly suitable for large-scale or deep learning scenarios where iteration efficiency is paramount.
Preconditioned methods and metric-aware optimisation. Preconditioning is a classical approach in optimisation and numerical linear algebra to accelerate convergence by modifying the problem geometry through change of metric (78). This perspective underpins variable metric and quasi- Newton methods (78). where gradient steps are taken relative to preconditioner that improves conditioning. In deep learning, natural gradient descent [49] formalises this idea by using the Fisher information matrix as Riemannian metric aligned with the model's 's statistical structure, with recent works showing that such methods can achieve accelerated convergence in overparametrised neural networks [50]. Similarly, recent efforts have revisited scalable preconditioning strategies, such as frottecke-ffcttree methods [79], Shampoo [51], and Muon [52], which exploit problem structure to efficiently approximate second-order information in large-scale settings. More broadly, the geometry of parameter spaces under reparametrisation has been studied to understand how such choices influence conditioning and learning dynamics [53]. Our framework complements and generalises these approaches by introducing reduction mappings where the pullback metric induced by & serves as an intrinsic, problcm-adaptive preconditioner. This allows first-order methods exploit problem geometry efficiently, without explicit second-order computations, while benefiting from strictly improved condition numbers. Moreover, our analysis applies uniformly to both affine and nonlinear mappings, unifying classical preconditioning with modern geometry-aware methods and extending their reach to a broader class of structured nonconvex problems.
geometric preconditioning mechanism,
Reduction mappings and problem dimensionality reduction. Classical strategies for problem simplification often rely on variable elimination techniques, such as projection methods, partial optimisation, or block coordinate descent, where certain variables are explicitly or implicitly optimised out reduce problem dimensionality [54]. A more general formulation arises in bilevel optimisation, where the inner problem implicitly defines mapping from the outer variables to the lower-level solution [55]. In these settings, implicit differentiation offers a principled mechanism to compute gradients of the reduced objective, effectively collapsing the problem to the outer variables alone. Our framework generalises and unifies these approaches by viewing them as reduction mappings whether -whthee explicitly defined or arising from inner problems solved to optimality- and providing a geometric lens to analyse their impact on problem conditioning. Crucially, our focus extends beyond mere dimensionality reduction, offering sharp characterisations of how these mappings improve local smoothness and sharpness constants of the reduced problem, and thus enhance the convergence rates of first-order methods.
20

---

C Proof of Main Theorems
This section contains the complete proofs of the main results stated in the paper. Each proof is presented in a separate subsection for clarity. Some of the arguments rely on technical lemmas, which are deferred to Appendix for readability.
C.1 Proof of Theorem 1
Proof Since F(x1)= = I (11, ((r)), we have the reduction mapping
I1 ((x)) =
E Rm
(x11)
By the chain rule, the gradient F is given by
VF(x)) = ((1)) (T((()) e R
where the Jacobian of o has the form
Ins (J (
D
with J(x1) = D (x)) m
and the gradient f (.(((z))) E R22
We define a Riemannian metric on R induced by & (i.e., a pullback metric) as
g(u, v) = (u, v) = T Rv, with R = (2((1) (*))).
Because the solution mapping (x1) ) is assumed affine, R is constant. Also, we have D2 v (xx) = 0. so no additional correction term appears in the Hessian F. Differentiating F((1)) with respect to 21 yields (see Lemma
v²F(x1) = D v2 5 (o(z)) ((((()))(()))))))))) (x1)).
Define A = (((((())) and note that A/y = F²I(I1) is the restriction of A to the ni
dimensional subspace
im (D (x1()))) M
We define the following operator norms
|||
max
ww A w| max maxxtmmma (mmax(mm)))) - -Amin -A)) 0max
|| koll
and
||yy|| = max WEY |
T A Awl max Amax(All)) - -Amin (Alv) = Fmax (Alv).
Also, by definition, the Lipschitz constant of V By = suP AA], and, similarly, the Lipschitz constant for the Riemannian gradient VF is Br = supz,eMMos | A|ylla
Step 1. Restriction of the Hessian. By the Rayleigh quotient characterisation (see Lemma have
Xmax (Aly) S (A), Amin (Alv) 2 Amin (A),
so,
Fmax (Alv) S maxx(A),
with equality if and only if there exists unit vector w E V satisfying
|w Aw] maxx (A).
Since Tmaxx (A) is achieved only along vectors in the maximum singular-vector subspace
Emax spanfy e N Av= Xe, K d (A))
21

---

we have equality if and only if some unit vector w E V is in Emax.
Step 2. Uniform Non-Tangency of Emax with V.
The uniform non-tangency condition (where its general validity is expressed precisely in Corollary by setting that
0) implies that for every unit vector Emax there exists uniform & > 0 such
||P wll S 1 E. Now, consider any unit vector w E Since w lies in V, we can use the following argument for any unit vector U E Emax:
1. Write the vector Eaa in its orthogonal decomposition with respect V
t P (I (I-Py) Py)
2. Since w E V. is orthogonal every vector in the orthogonal complement, in particular to [I py)c. Therefore, we have.
w T u = w [Pv++ + Pv) o] = w Pv
3. Applying the Cauchy-Schwand inequality, we obtain
w v | (Pv ( v) Since ||l||| and by the uniform non- non-tangency condition, follows that
- || ||P u|l |Py
w T vl S --E.
Step 3. Bounding the Absolute Rayleigh Quotient Let A Σ d U be orthonormal eigendecomposition of A with eigenvectors {ui), and define a, = [Asl. ordered so that 01 vector V. write
Op mmax (A) > 0p+1 2 2 On- For any unit
w
Og 02.
so that
Σ aa==1
Let Ei af be the squared norm of the projection of onto the top singular subspace. Then
|w Awl = Σ Xa
S Σ Ca.a Fmax (A) 02 +
Σ o,a
1=1
1=1
x=p+1
Since 0; S Spt1 for i > p, the tail sum is bounded:
Σ Daaa S 50t+1
(1) (1
yielding
A w Fmax (A) 82 + opti(1 82).
Using the uniform non-tangency condition 8² < (1 E)² and taking the maximum over all V. we obtain
mmax (Alv) WEV, ||=]|==
TTAAu Aww < mmax (1- opt1 [1 (1 (1=
Step 4. Expressing the Gap via the Singular- Value Gap
define the spectral gap between the largest eigenvalue and the second largest as
Amax S mmax(() opt1.
Rewriting 0p+1 S Tmax (A) A we have.
Fmax Tmax(A/)) (Aly) S Fmax (A) (1 - 6)2 (omax(A) Amax) [1- (1 - s)²
= 0max (A) - A Ama [ - (1 =]
= (A) A Amax (25 - 2²)
22

---

Taking the supremum over all points in each domain, it follows that
BF sup max (Alv)
S sup a max (A) max (2s = = 3g
A A (28 5²)
in M
sEN
Thus, by our assumption that Amax > O, we obtain the desired strict inequality:
Br S 8g Amaxe (2c =2) 3g.
This completes the proof.
C.2 Proof of Corollary
Proof The result follows directly from Corollary supremum over 11 E yields
Applying this to F2F(x1) and take the
[E) SF
sup F2 s M( sup V F
(1)
) M Bp.
Finally, substitute the bound Br S Bg A max (2s from Theorem conclude
3/
< M ( mmx (2r =²]]
< ( 8y
with (o) be equal to 2 by the non-expansive property of the projection mapping V.
C.3 Proof of Theorem
Proof From Lemma
we have that the Hessian of F can be written
F(n)) ((2)) V2 ( D (xx1) + C(x1),
where the correction term due to the nonlinearity of (()) is
C(æ1) DDa (z) V a. (4xx1))
Here denotes the contraction of the third-order Hessian tensor D v Raaxnixn with the gradient vector v E TR over the first index, that is,
0f
C(z1)
D (R). 0rg,
(#(z))
1=1
Define A= == (2)(((2))) B == tangent space of V To( as Aly Ff)(1 such that
(+(()) D (x1), and the restriction of A into the
Aly - BCC
Since Aly and are symmetrie, it follows that C(x1) is symmetric matrix. For B, symmetric matrices, we apply the Weyl's inequality for the pullback metric R
max B C) S max (B) + ||C]]
From Theorem we have
Tuuxx B) maxx(A) Amax (2e =z2)
80,
Tmax (Alv) S (Aax(() Amax (2s =²) ||C||
where we have ||C|| R < ||C]I/m(() We need control now the Euclidean operator of the correction term C. By the sub-multillicative property of the norm and our assumption on the bounds, get
||]|| D (< | f(+(x)) s QZ,
where since 2 (z1) ) is a higher order tensor, we take the appropriate norm for the sub-multin sub-multilicctive property to hold. More specifically, standard choice is to use the tensor operator or injective norm. If we have N-th order tensor T E I Xnx its operator norm is defined by
|T| sup (=)()) (2)
(N) 0000 1, Vi}.
23

---

where T(x(1), ( (N)) is the scalar obtained by contracting T with the vectors z(), ..... x 2M)) We also have the uniform curvature gap defined
= mmax (A) max B) 2 17.
Since the bound Q Z on the correction term is assumed to be strictly less than the curvature gap the improvement in smoothness obtained by projecting onto the tangent space is preserved. In other words, the additional curvature introduced by the nonlinearity does not overwhelm the inherent reduction from the projection, ensuring that the overall smoothness constant of the reduced function remains strictly below 5y.
Formally, by taking the supremum over the region of interest, the Lipschitz constant 3P of the Riemannian gradient V satisfics
QZ Br S B Amax (2E-E²) + m < Bg
For projection mappings V we have 0 = D D I, and = D D 3 21. Hence m (+) 1. This completes the proof.
D Additional Theoretical Results
This appendix presents supplementary theoretical results that support the main developments in the paper. While not required for the core proofs, these results offer deeper insight into the structure of reduction mappings and their implications for optimisation and geometric analysis.
Specifically:
provides quantitative bound the correction term induced by composing function with a smooth minimising mapping V, derived via the implicit function theorem. Section D.2 establishes that Morse-Bott structure is preserved under reduction, assuming mild regularity and a clean intersection condition.
Section D.1
D.1
Correction Term Bound Under Argmin Mappings
The proof of the main result below relies expressions for the first- and second-order derivatives of implicit argmin mappings. These are provided in Appendix Subsection E.
Theorem 4 (Quantitative Bound on the Correction Term) Let f G R x R + R be C3, Assume that for each iı in an open neighbourhood U R the solution mapping
R" + R be C2, and let
V(x)) arg min G(m,,u),
wER=2
well-defined and C², Suppose further that for every xı eld
(i) Strict second-order sufficient condition (SSOSC).
H(x,) (,)((I),((()) 2 o Im 0 0> 0.
(ii) Bounded higher-order derivatives. There exist constants L12, L21 L3s L11 > 0 such that at (æ1, (zz)),
zfu G 4 L12,
V & Lei, | < Lz, | s La1-
Write
V = (VeyJ, S.e.VVaall Vaal), v(x1) Vo, ((r1),
[V (
Define
[ |
[V/(,,, ((x))>>0 ge 00
(xx)))l
0, otherwise,

---

Let
(Z)): 1 R +R C(z1): R R R.
be defined by
C(z1)(h, = (u(z1), D2 (x)
Define the operator norms
[ ((=))]
sup |
D2 (a,
[[C(x))]|=
sup
(h,))
||||=||||==
and set
L L12 L21 Lis + La Li1
Then for every Xı EU, the following bounds hold:
L 22
i -
[|((x))] ETT V(Dill
|((x))]
σz E
Moreover, if one regards D² v (21) as linear map Sym (I"") + R with top singular subspace R and defines cos θ > (x1))l//((((z)) then the refined estimate
i
L E, cos 0
[[C(x))| < o |[(())) cos < σ
also holds.
Proof We assume the mapping arises from unconstrained problem for simplicity; analogous results hold in the constrained setting.
1. First derivative via IFT. By the SSOSC assumption,
H Ga,, (z1)) ł al = | S
Applying the first-order implicit function theorem (Lemmal)
yields
< Lu
( = - G, ||
v
Second derivative via higher-order IFT. Let
A
G. B 2 G, C V
Then the second-order expansion (Lemma gives
D
A +
B
Using the bounds
| L.
|| < kn
||||| S Lia,
L21,
|C| s La,
we obtain
1
1 1122
L11
L2
i
D 1
T2ı
L
0
o²
3. Partial-gradient bound. Since ||
| fll Ly, we have
||a||| = Ε | < L,
Moreover, & < 1 strictly whenever V., f(1 (x1 * 0, since then
|| |V =. fl|
||||||
The degenerate case & = 1 occurs only if V f 0, i.c., the full gradient lacks 21 component.
25

---

4. Operator norm of the correction form. By Cauchy Schhaaz,
L (u, D2 (1 S |ol| 1 w/ S σ² El..
||]|
sup |
||A
Let Σ denote the top singular subspace of D2 v. Projecting onto Σ, we obtain the refined bound
||C|| - D2 |||| ||]]| cost,
where cos = ||Puu|/||||
Equality cos 0 = 1 holds only when v E E, i... when u is fully aligned with the top-amplification directions. Otherwise, cos strictly whenever v(x1) é Σ.
D.2 Preservation of Morse-Bott Structure under Reduction Mappings
Assumption 3 The intersection Mr n is clear ensuring Sp is C submanifold.
Remark 4 (Correspondence of critical sets under reduction mappings) Let 4(x1) (I1, (xx)) and F(x1) == f(((x1)). Then a global C² di-iiffeomorphism onto its graph manifold Mr. If the critical set S intersects My cleanly, their intersection corresponds diffeomorphically to the reduced critical set S5 = {x1 F( C. (z) = 0}, with
SF SnMP
Moreover, D (x1) is a linear isomorphism between their tangent spaces.
Theorem 5 (Morse-Bott Reduction) Let f R - TR. be a C2 function which, on a compact neighbourhood N of a level c, satisfies the Morse-Bott property: its critical set S (x E N V Vf(x) 0, f(x) = c}, is smooth submanifold and
ker
TTS
Ve E.
while every positive eigenvalue of 2 ((x satisfies X 2 M1 > uniformly.
Let & R. gmitna be C embedding and denote its image MF ((R")). Write 4(x1)
(x), (ZI)), V R + n2 is the induced projection. Define the reduced functiinn
F:Rm + R,
F(x1) = =(D(x1))
and set
=F=
= NF VF(z1) = 0, F(x1) c}.
Also suppose that for the intersection S MM Assumptions Bnnd hold. The F is Morse Bott Np:
1. Sy is smooth submanifold of p
2. At each xı Sr, ker (V²F(x))) Tay Sr, and every positive eigenvalue of (2F(1)) satisfies X₂ 2 PF uniformly.
Proof Since, by assumption, the submanifolds S and Mr intersect cleanly, their intersection S Sn.rr smooth submanifold of R.. Moreover, since + is C2 embedding, the pre-image SF = (S MM)) is a smooth submanifold of "R diffeomorphic to S o. Mr (see Remark 4).
At critical point o E Sp, now establish the identity
ker ( (V²F(zi)) = SP.
The proof consists of two parts.
We assume that the intersection Mr S is clean so that Sp is a well-defined smooth submanifold. This assumption is essential for establishing the Morse- Bott property for the reduced function F. A discussion on the detinition of clean intersections and their genericity is provided in Appendix FF
26

---

(1) Sp C ker ker(V2F(zi))
tangent w E Te; Sp must satisfy VF =0 F c to first order. At a critical point VF(xf) = the only nontrivial condition is
w=
so that w E ker ker(v)((a)))
(2) ker (<)())) Sp:
If (( 0, then
0 D=Do(a)) (V)(((x))) u
Injectivity of D + gives (²(((x))) D w| so
D (2j w ker (Vf(((<i)))
Since / satisfies the Morse-Bott property, we have
ker V2 (2(((()))) T T
By definition, we have D (xr1) E Ta(a)) hence
( E T n Ta(a))M-= (S Mp).
By the pre-image description of Sp, we conclude that must lie in Ta
Combining the two inclusions yields
ker (V²F(x))) = Sp:
which shows that the reduced function F(11 satisfies the Morse-Bott property. On compact Sp. the smallest positive eigenvalue of 2 attains minimum of p. 0.
E Supporting Lemmas
This appendix collects technical results used throughout the main body of the paper. The lemmas grouped into four self-contained subsections, each addressing specific analytical geometric aspect of the theory:
Derivatives: Closed-form expressions for the first- and second-order derivatives of implicitly defined mappings, as well as the Hessian of the reduced function induced by a smooth reparametrisation.
Rayleigh Quotients Comparison: Classical inequalities comparing spectral quantities of symmetric matrices under embeddings into lower swerr-dimensioal subspaces, used to control the eurvature of reduced objectives.
Uniform Angle Bounds and Genericity: Results showing that the eigenspaces associated with extreme curvature directions intersect the tangent space the feasible manifold only trivially. This condition holds generically and yields uniform lower bounds on the angle between the subspaces.
Genericity of Single Eigenvalues: A measure-tiie theeoretc argument demonstrating that symmetric matrices with repeated eigenvalues form zero-measure subset. This justifies the generic spectral gap assumptions used in several proofs.
E.1 Derivatives
Lemma (First-Order Derivative Expression for Implicit Functions) Let Rm x R R be at least twice continuously differentiable. Suppose that for each there exists unique y (x) satisfying the optimality condition
Dy 1(, v. (x)) 01xn
27

---

and assume further that the Hessian D h(x, y (x)) is invertible for all a in neighbourhood U around a local minimiser. Then the first-order derivative of y (2) with respect to a is given by
Dy*(x) h
h(z (x)) D² b(a,y' (x)).
Proof Starting from the optimality condition and differentiating it with respect to x yields
DD M( (x)))) =0 D2y hxx (x)) D Ma, (z)) Dy (x) =0
Dy = --[Di, (x, (x)) D h(a, y (x))
The uniqueness of y" (x) is guaranteed by the classical implicit function theorem.
Lemma (Second-Orde Derivative Expression for Implicit Functions) Let h 18m x R" R be C and suppose that y² (x) is the unique solution of
Dy Dyl,,, (x)) O, x xEM;
with
H(x) = D M2,, (x))) E (R),
for all x E U. Define
J(z) Dx)=
h (z, y (z)) E R"x"
Further introduce the following third-order multilinear maps at (z, (x))::
A(x) == D he R RRT &R"
OR (xx) = D D hER" 6 RM "R"
C(x) = D DE
Let
TA &6 R. RR) L(Sym²Rm RR) be the symmetrisation operator : (T,
(T(u, o] + T(u, ul). Then the second derivative D2 y" (x) E L(Sym R ,R") given by the formula
D² (2y . A(x) (B((
J(z)) + c(x) (J(x) & J(x))
where & is the tensor product and is tensor contraction matching covariant and contravariant indices.
Proof
For convenience, we define the vector-valued function F: mR m R
R where
F(x, v) = D, h(r, v), D2 F == D2y D h. Dy F = D' h H(x).
From Lemma 0 we have that for the direction Rm
D Flu D) F(u) + Dy F[Dy" 0. Differentiating again in direction E _Rm we obtain
D Flu) [b] + D (D, FDy [u]] [u]
We now expand each of these two terms by applying the produce and chain rules in turn.
D Flu) [u]
D Flu, v) + D2 Flu, Dw"pl]
B(w)(u, [v]l
D (D,, F[DD"([]] [o] (zx) [u], r]
D F y [w]. [v]] + D, F(D yf [u,
[D= [M]. [v]]
Summing the two expansions and recalling that D, F D is the Hessian in y, we get
D [DD ju, v]] (A(z)(u, w BB
B y c)) Dy [v]]).
28

---

Since h2 = H (z) is invertible, and denoting [(x) = D (x)), we conclude
D2 y= u] - (A()),u, v] + B())uu, [v)] + B (zx) ua
(())
Because the function smooth, the mixed partials should commute by the Schwarz's theorem. The tensors A and C are already symmetric, only need to deal with the two B terms. To that end, we introduce the symmetrisation operator (u, v) = 1 (T o]. Tlu, ul), and we conclude
v) + (w), J(v]]} Re-expressing this result in the compact tensor tensor-prod producctcccontraation notation exactly yields the lemma's 's statement.
D y" (u, oj =
(A(x)(u, + / BB(
Lemma 3 (Restricted Hessian of the Reduced Function) Let f: Ru - IR be twice contin- uously differentiable and let V (xx) ): R R be a twice continuously differentiable mapping.
Define the reduced function functionn F: I R by F(x1) (on)==((n)) ((1)), ((t)) and denote the reduction map
21 (a1)) = (((i)) with J(x)) = Dry ((1)).
Then the Hessian of F is given by the compact representation
FFF(1)= [r J()) (2)(') [(ai) + C(a)
where x= (x11) and the correction term
C(x1) D2, (x)) Vzzs(x");
captures the curvature of the constraint manifold Mr {(1 (x1)) R" The symbol
denotes a tensor contraction operation at the appropriate indices.
Proof We start by differentiating the reduced function F(m1) (1n (mn)). By the chain rule,
VF(21) Ve, + (D., (x1)) Voaf(="))
where x² = (x1, (x1)). Differentiating V (F()) with respect to æ1 gives
v²F(x:) Dz, (vz,(xx')+ (De, ((1))) vonf(x')
=[2,5(0") V2 [2 Da (((1
+ Daj (Dzn (xx)))
Vazs(r") + (Das ((x1)) Day 1
Since Dz, [Zz5(x")) 2
aza f(x))
f(x) Dz, (x))) and using the notation ((1]
Da, (x1). we can rewrite the above
F2F(x)) V2, (x*)) V2 22
[(x) J(x1)
+ +D., (J(zI))
V fxx" + [211 V2 f(x') + fxr)
Grouping like terms, we obtain
v²F(z1) V2, = f(x") v2 f J(z))((r)))))))) ((11) V Venn f(=))+++++++ (xx))(T22 VV f(z"(")) ((x1)
+ DD2 v(z1) Vorf(x")
ni Xn
n2 x1
This expression can be compactly represented in block form as
[ ( + D2 v vezf(x")
v²F(#))= = | J(z1) v2 v²)(")
which completes the derivation.
29

---

E.2 Rayleigh Quotients Comparison
Lemma (Rayleigh Quotient on an Embedded Subspace) Let Ruxn be symmetric matrix and let B E Ruxk (k < n) have full column-rande Write
V = im B CR, BBB00.
Define the ambient extremal Rayleigh quotients
v Au
a A
Ammxx
max VER" V. v TTv.
Amin(A)
min vER u140 TTv
and the embedded (or generalised) Rayleigh quotients
21 B AB
min 1/' B AB
Amax (Alv) :=
max
Amin (Alv) YERK, V B By
VERK 1770 y B B
Then
Amin (A) S Amin (Alv) S Amax (Alx) S (A).
Moreover, the right (resp. left) inequality in (x) is strict if and only if Emax(A) 7 {0} (resp. Funin (A) nnv= (0)).
Proof Put w == By E Because B has full column rrakk the map y + a bijection between R \ {0} and V {0} and
y Ry [By)TBB)) = 11 B ABy (By) A(By) T AAw
Hence
w T T
w A w WEV, wg w w
Amaxe (Alv) =
max WEV, w, w
Amiu (ala) min
Applying the Courant-Fische variational characterisation to the ordinary Rayleigh quotient on the proper subspace V C IR" gives the chain of inequalities (+). If, in addition, Faax (A) n {0) (respectively Eminn (A) n
(0)), then the extremal value attained over V is strictly smaller
(respectively strictly larger) than the ambient one.
Corollary (Euclidean Rayleigh Bounds via Gram-Matrix Extremes) assumptions of Lemma 4 and set
Keep the notation and
m()) Amii (R),
(R) Amax (R)
(0 < S M).
Define the coordinate Rayleigh quotients
B AB TB
y B AB
Xmax (BTAB) max YERE WAA
AB)
Xmin (B AB) = min
y yT
yER* 9ę40
y
Then
m (R) Amin (A) & Amaa (B AB) S. A (B AB) S (R)) Amax (A).
Moreover, the right (resp. left) inequality in (x) is strict if and only if Emax (A) 0 {0) (resp.
Emin (A) (()))
Proof
Because (R) R (R) for every + one has
(R) yT y Ry & y
Combining this with the definition of the generalised Rayleigh quotient gives
y T ATTBy < M(n) y T AB T AB
(n) y¹ (l)
B AB
2
y RTY Taking the maximum (respectively minimum) over y + and invoking Lemma 4 yields
T
y'y
y Ry
ml (R) Amin (Aly) S (B" AB) & (BT AB) S (R) Amax (Aly). Applying again the bounds Amin(() S Amin (Aly) and Amax (Alv) S Amax (A) gives the chain (+).
Strictness of the outer incqualities propagates from the corresponding strictness in Lemma
30

---

E.3 Uniform Angle Bounds and Genericity
Lemma (Local Compactness of the Intersection Sn Mr in a bounded neighbourhood) Let function f be C We define the manifold of local minimisers as
(ar E TR. Vf(x) = 0, f(x) = fs).
and the graph of continuous mapping V
Mr = ((11, ((()) 21 E ]. Also, let N C t be an closed and bounded set. Write
gluc = S MF == Mr N.N.
Then both Sloe and M are closed subsets of the compact set N, and therefore is compact.
sloe nMpp
Proof We can write the solution manifold S an intersection of two pre-images
S = (Us)) ((()))
Since is C2 and {fs} C R. {0} c R are closed sets, follows that their respective pre-images are closed in I Hence, is closed set.
Similarly, the feasible manifold can be defined the pre-image of function g I R2
R
such that
MF = ((11 g(r x2 ((()) = 03 = =3
closed in t2
Finally, since S and Mr are closed, their intersection with the neighbourhood. A a closed subset of compact set. Any closed subset of compact set is compact, and thus gloc Ml is compact.
Lemma 6 (Uniform Angle Bound for Isolated Spectral Subspace) Let R" be compact set and let H. : N + Sym(n) be continuous map. Let MP C IR" be smooth submanifold such that K = N C My is compact.
For each K choose an eigenvalue M(x) E spee (gp) (with multiplicity k 2 1) and assume uniformly isolated:
min - A(x)] 2 A >00. (Eapeo((a)MMACC
Define the (possibly multi-dimensional) eigenspace
E(x) = ker (Hn (Ha-
and assume that it satisfies
E(x) (0), Wo K.
Then there exists > 0 such that for every E K and every unit vector v E E(x) one has
<(v, T,MF) 20
Proof Step Continuous subspace field produced by the Riesz projector. By hypothesis, for every E M we have
min (aa
11 x(x)) 2
YrENNMP.
Hence one can choose disjoint small closed contour (xx) of radius A around A(x). By construction this contour encloses precisely the X-cluster and avoids all other eigenvalues of Hg. Then, the Riesz projection onto the X-eigenspace,
$ (z H²) dz, 2Ti J.
R(x)
31

---

is well-defined and depends continuously on (cf. Kato 80 Ch. II $1.4).
Set
B(x) = im R(z).
Thus x: > (2)) is a continuous map Mp - Gr(R") into the Grassmannian ie., forms a continuous field of subspaces.
Step 2. Compact unit-sphere field. Define
= 1 K K= (=)00)) Ne UE E(x), ||o|| ||||==
Continuity of x + E(x) implies that the set is closed in the product
F x 8"-1, because the Lemma 5 Hence, the set compact.
base N n M compact (since Mr closed
Step 3. Uniform angle gap. For any (x, v) E K. we have v & Mr by the hypothesis E(x) n ME {0). We note that this trivial intersection is generic (Lemma 7). Because orthogonal projection decreases norm unless the vector in the target space,
p(x v) PM (v) <1
The function şp -N + [0, is continuous; compactness K. gives a global maximum å maxk to < 1 (by the extreme value theorem). Choosing 0 = arccos > 0 yields Z(v, T&Mp) 2 for all (z, v) XC, as claimed.
Corollary (Uniform Angle for the Xinn--ggenspace)
K = -N=NNNSMMM C R., where N is closed bounded neighbourhood of a local minimiser of E C S is the Morse-Bott critical locus of J, and ME a smooth submanifold (so K is compact see Lemma Set
H₂ = >=>(((x) NgS:
E KEK,
so that H is positive-defiitte Suppose
(a) the smallest eigenvalue Amin (x) (mult. m 2 1) of H. is isolated by a uniform gap An-m(z) - Amin(x) 2 Amin > 0,
(b) eigenspace Funnn (x) = ker ( H Amin (2: satisfies the trivial intersection
Emia (I) Tr = {0), Vz E K.
Then Lemma 6 yields constant such that, for every K and every unit u E Emin(x),
(v,TyMP) 2
Corollary 6 (Uniform Angle for the maax -inngular Subspace) Let == N n Mr c RR", where N is closed bounded neighbourhood of local minimiser of E C² Mp is smooth submanifold (so is compact see Lemma 5). For x E set
H V2 f(x),, Omax (I) = largest singular value of H
Assume
(a) the largest singular value mmax (2) (mult. 2 1) of H is isolated by a uniform gap Tmax (x) 0pt1(a) (3r) 2 Amax 0,
(b) the eigenspace Emax(2) (x)) = ker (H² m (x) satisfies the trivial intersection
Emax MrMF {0}, Ve E K.
Then, by Lemma 0 (with the eigenvalue cluster {o o (x)) of H2), there exists 0 such that for every x. E K. and every unit v Emax (I),
2 0.
32

---

Lemma 7 (Genericity of Itivially-Intersectinn Subspaces) Let 0 < k, < N with lo + +1 < N.
Write Gr(k, N) (resp. Gr(,,N) for the real Grassmannian of k- klplanes (resp. 1-planes) in EN and sel
Σ = ((4, V) E Gr(k, N) x Gr(I, u u (0))-
Then E is real-algebraie subset Gr(k, N) Gr(l, of codimension
codim Σ = N (k: + l) +1 1.
Consequently 2 has empty interior and Lebesgue measure 0; in particular generic pair of planes satisfies = (0).
Proof Step 1. Rank-drop criterion. Choose full-rank matrices U E N E R.Vxl whose column spaces are ll and V, and form the N (I: + 1) matrix M = [U VJ. We have
unv + {0)
rank M 1-+111
Thus the "bad" locus E is the image in Gr(k, N) Gr(1, N) of the determinantal variety
D= M E R (x++))) rank M S -
defined by the vanishing of all (k + 1) (k + minors of M.
Step 2. Codimension in matrix space. For general integers N, m, with r < m S N,
dim N x m matrices rank S r) = (N + m)r - 12,
hence
codimgaxxr { Trank S
Nm (N m)r + 22 = (N rm - r)
Taking = k + and r k - gives
codimgax(e4)) D= = (N (ks + 1).
Because ++1 < N by hypothesis, this number is 1.
Step 3. Passage the Grassmannians. The product of Stiefel manifolds St(k, N) St(L, c
R. x(++)) is open subset of the full-rank matrices, intersecting D with cannot decrease codimension. Next, the quotient map
St(k, N) Nt) - Gr(k, N) Gr(I,N), (Q1, + (co) a smooth submersion with compact fibres ((k)) x (1)), which preserves codimension. Consequently codimor(t,N)xxGHG.. E = N (k + +1 +1.
Step 4. Genericity. Because E is proper real-algebraic subset of positive codimension, it has Lebesgue measure and empty interior. Its complement is therefore dense (indeed Zariski open) and of full measure, so a generic pair (U, V) satisfies 24 n = (0).
E.4 Genericity of Single Eigenvalues
Lemma (Codimension of the Repeated Eigenvalue Locus (8, Let
S(n) {A Rax A A }
be the space of real symmetric x n matrices, and let.
(A E S(n): A has a repeated eigenvalue).
Then E is an algebraic subset of S(n) (namely, the zero set of the discriminant of the characteristic polynomial).
At a matrix Ag E where an eigenvalue No has multiplicity exactly 2 (with the other eigenvalues distinct and different from No), the condition that nearby matrix has a double eigenvalue imposes two independent (local) constraints on its parameters.
Consequently, the codimension of Σ in S(n) is least 2.
33

---

Remark 5 (Generic Simplicity of the Eigenvalues of Symmetric Matrices) Lemma shoows that the set of symmetric matrices with repeated eigenvalues is real real-analytic subset of Sw. which can be decomposed into a finite union of smooth submanifolds, each of codimension at least two. Since smooth submanifolds of codimension 2 have Lebesgue measure zero (by applying Sard's theorem (82)), it follows that this set has measure zero. Therefore, the complement- set of symmetric matrices with simple (distinct) eigenvalues full measure. In other words, simple eigenvalues are generic among symmetric matrices.
F
Transversality and Clean Intersection of Smooth Manifolds
To study the geometry constrained critical sets, we begin by recalling two foundational concepts that govern how smooth manifolds intersect: transversality [ Chapter 3] and clean intersection [84] Appendix C.3]. These notions describe the local structure and regularity of the intersection and play a central role in establishing genericity results for solution sets in optimisation and variational problems. To build geometric intuition, we illustrate examples of transverse and clean (but non-transverse) intersections in Figure
Let
N C Z,
be embedded submanifolds of a smooth manifold 2, with
dim M =,, dimN= = dim Z =1.
The classical condition of transversality ensures that two manifolds intersect in general position, meaning their tangent spaces at each point of intersection span the ambient space. This condition central in differential topology and guarantees that intersections behave stably under perturbation. However, in many geometric and optimisation contexts, transversality unnecessarily strong. For example, if the sum of the dimensions satisfies + < then transversality cannot hold unless the intersection is empty. Consequently, generic perturbations that enforce transversality may climinate meaningful intersections entirely.
This behaviour undesirable in applications where the intersection encodes feasible or optimal we wish to preserve. To accommodate more flexible and structured intersections, the notion of clean intersection provides a weaker but still geometrically meaningful alternative. Clean intersection allows the tangent spaces to align nontrivially, as long as the intersection remains a smooth submanifold with compatible tangent structure. This relaxation offers a more flexible framework that preserves non-empty intersections under generic, local perturbations, which is exactly the setting we consider in our problem.
solutions-
We now formalise both notions:
Definition 4 (Transversality) We say that M and N intersect transversally, denoted by M dh N, if for every point x E MMN the tangent spaces satisfy
T.M+ -TZ,
or, equivalently, the normal spaces satisfy
NeMON,N= =
Definition (Clean Intersection) We say that M and N intersect cleanly, denoted by M m N, if for every point x E M M,N, we have that the dimension d = dim(T,M TT,) ) is constant and
Tz(MON) = TaMOT
Remark Transversality implies clean intersection between manifolds but the converse is not always true.
To rigorously analyse generic properties of smooth manifolds and function spaces, we work within topological framework of residual sets. This notion allows us to make precise what it means for property to hold "generically" that is, to be true for a large and stable class of objects. In particular, we will be concerned with residual subsets in spaces of smooth functions immersions, equipped with the Whitney O" topology.
34

---

Definition 6 (Residual set & generic property) Let X' be topological space. A subset
A C
called residual it contains a countable intersection of dense open sets X. If is a Baire space (e.g. any Banach or Fréchet space, with its usual topology), then every residual set is itself dense in X.
A property e points in X is said to hold generically if the set {or X satisfies P)
residual in X.
Within this framework, differential topology offers a foundational result: transversality a generic condition, as we see in the following Lemma. That is, for a large class of smooth mappings, transversality to fixed submanifold holds generically under perturbation.
Lemma 9 (Generic Transversality of Submanifolds) Let M,N N C be submanifolds of a smooth manifold 2, and let F be the space smooth immersions of one of the submanifolds into Z (e.g.. M). Then, the set of maps & E F such that ((M) rh N residual F with the Whitney C" topology for 1. In particular fransverse intersection is generic under smooth perturbation.
Proof This result follows directly from standard transversality theory, specifically the Parametric Transversality Theorem Thom's Transversality Theorem. See, for example, [83] Chapter Theorems 2.1 and 2.9], where it is shown that the set of maps transverse to a fixed submanifold is residual in Whitney C" topology for any
We now apply these concepts to the setting of critical points. Let f E (RR) be a smooth function satisfying the Morse-Bott condition, and consider the of critical points fixed value as defined below. We are particularly interested in the intersection of this critical locus with a fixed feasible manifold Mr R While transverse intersection may fail due dimension constraints, clean intersection remains viable and meaningful condition and, crucially, it holds generically within the space of C2 functions.
Definition (Critical-locus map) Fix c E R and a C2 function f R= + IR in which the Morse-Bott property holds. Define
((x) cf( -XX
Its zero-set S -66-1(000) is the Morse-Bott critical locus.
Lemma 10 (Equivalence of Clean Intersection and Local Transversality [85 Lemma 2.6] Let
M,N c. Z be submanifolds of a smooth manifold Z, and suppose that the dimension dim dim,TTMM.. n
T N) locally constant for all E N. Then, the following are equivalent:
1. M M:N:
2. For each x E M n N, there exists a submanifold N" C containing N near x, with codim(N')) = dim M dim(Ta.M (TTMMTTTTTT such thut
M and M
locally near x.
particular, clean intersection between M and N near x equivalent to transversality of M with an auxiliary submanifold N" that locally agrees with N on the intersection.
Lemma 11 (Generic Clean Intersection of the Critical Locus) Let (0-100000000 be the Morse- Bott critical locus of a C² function f : g > R. at level e E IR as in Definition 7 and let MF c I." be a fixed embedded submanifold. Then, clean intersection between S and My is generic property in the space of C functions under the Whitney topology.
Proof By Lemma I0 clean intersection between S and MP near point E SnMP is equivalent to the existence of submanifold N C Rh" containing MD near & such that
S S Sn locally, and S m N.
35

---

That is, clean intersection is locally equivalent to transversality between S and an auxiliary submani- fold N
Since S is defined as the zero set of the critical-locus map ((x) = (f(x) c, V)()), varies
smoothly with in the Whitney C topology- By Lemma transversality S r .V is generic condition among Ca functions. Therefore, clean intersection S m MM holds generically in the space of C functions.
Remark (Density and Small Perturbations) Since every residual in a Baire space is dense, the generic set
{ (R") Cfit,(f) m MF).
not only large in the topological sense but also satisfies: for any given f and any >> O, there exists C (I8") with llg fllc= < such that Crite(g) M. MM.
In other words, any non-generic f can be made generically clean by an arbitrarily small C perturbation, without making the intersection empty.
G Experimental Validation of Results
To validate theoretical findings, we present three illustrative examples. These examples demon- strate the application and performance of our proposed reduction mapping method different optimisation scenarios, from simple 2D quadratic case a more complex high-dimensional nonlinear problem.
G.1 Example 1: 2D Quadratic Problem
We begin with a simple two-dimensional quadratic function visualise the trajectory of the optimisa- tion process. It follows the first example of the Appendix A where the objective function is given by,
G(ær, v) x²² + 10 =)²².
The reduction mapping, which constrains the optimisation to the manifold where y x, is defined as,
((r)
x
The reduced function F(x) =², with Hessian (2 2. In this case, the pullback metric tensor R DΦ DΦ = 2 is identical to the Hessian. Consequently, our geometrically preconditioned gradient descent method recovers the full Newton method. As expected for quadratic function. this achieves convergence in single step. Aigure/oillustrrle of the function. We can observe that the reduced method directly follows the valley of the function, leading faster convergence. The learning rates for each method based on the smoothness constant of the function, ie.,, 17 = 1/3.
the optimisation paths the contour plot
G.2 Example 2: High-Diimensioal Quadratic Problem
Next, consider digh-dimensional quadratic problem (n 40). The objective function is,
G( - + 11
|-
A
Kx
where R", y E R", and K is linear operator. We also set the parameter À equal ten and keep it fixed during training. The reduction mapping is given by the manifold where = Kx:
(xx)
=
K
T
The gradient and Hessian of the original function are,
x AK" (Kx - y)
and v'G(z,y) = 1 1+ K
-KKT
VG(x, v)
y X(1 Kx)
-XK (1+A)1
36

---

Figure 5a: Transverse Intersection
Figure 5b: Corresponding Intersection Set
Figure 5c: Clean & Non-Transverse
Figure 5d: Corresponding Intersection Set
Figure Se: Non-Clean & Non-Transwen Transverse
Figure 5f: Corresponding Intersection Set
Figure 5: Visualisations of three types of surface intersections in TR3 Left column shows intersecting surfaces, right column shows corresponding intersection eurves. Top (Transverse): The paraboloid
x22 + y intersects the plane = ++r+ + y transversely, with distinct tangent planes at every point of intersection. Middle (Clean, Non- Non-Transverse), The surface
-0.02 0.02y² is intersected by the curve (t) = (0, t, - + (t))), where (tt) 0 for | < 1, pp()) = 0.04 - 1) for 2(t 1)ah for > The curve lies exactly the surface for t [-1, the intersection is a smooth ID submanifold and hence clean. However, outside this interval, the curve deviates from the surface, and the tangent vectors do span 3 the intersection is not transverse.
-1, and (t)
Bottom CNon-Clean, Non- Transverse: The surfaces x² + y² and = x² + y² + x²y intersect along {i 0) U {y 0}. forming two parabolic curves that meet the origin. the origin, there is singularity, so the intersection is not smooth submanifold, hence neither clean nor transverse.
37

---

Optimisation Trajectories to Optimum
2D (steps=B02)
1D Prec (y-x, steps-1)
10 (y-x. steps-1)
Optimum
2
1.6
1.5
1.2
1.1
14
15
15
-6
44
-2
Figure 6: Optimisation trajectories for the 2D quadratic problem.
The reduced function F(x) G()(x)) simplifies to,
F(x) ブ |= + 2 [K
The gradient and Hessian of the reduced function are,
(x) = x + Kx, and (+(())I+KKT
For this linear least- squares problem, the Hessian of the reduced function is identical to the pullback metric R = DΦ D& = 1 + K. As in the first example, our geometrically preconditioned gradient descent is equivalent to the full Newton method applied to the reduced function, which converges in a single iteration. The experimental results for this case are summarised in Figures] 7 and, 9
g
The learning rates for each method are based Armijo's 's backtracking line search.
Objective convergence
Gradient norm convergence
10a
10a
G,(X.y)
G,Ix,y)
Objective (log)
Fix)
10a
Fix) Reduced PrecGD
10-2
Reduced + PrecGD
Ilgrad| (lag)
10
10-12
10
10-17
10
10-22
10 10_
20000
40000
60000 80000 100000 120000
23000
40000 60000 B0D00 100000 120000
Iteration
Iteration
Figure 7: Convergence rate for the high-dimenssonal quadratic problem.
G.3 Example 3: High-Dimensional Nonlinear Problem
Finally, address high nigh-dimensional nonlinear problem (n 40), with the same settings as before. The objective function is,
G( v) 1 11=11 |llll + ly a tanh (anh(K>)112.
38

---

Reduced eigenvalues
Reduced Hessian spectral gaps 1 -Al
1.0x 10
10a
(logg
10a
Gop (log)
101
Amax
Ants
10
10a
0
10
15
20 Index
25
30
35
40
10
15
20
25
30
35
Index
Full Hessian elgenvallues
Full Hesslan spectral gaps Mi A.l
Kg
10
10-2-
(log)
10-5
20a
10
10:11 11
10-14
Index
Index
Figure 8: Hessian eigenspectrum for the high-dimensional quadratic problem.
64/86.62
10a
10a
2310.8x
Wall time (s)
10
10
10-1
1.0x
Full
Reduced
Reduced+
Figure 9; Wall-clock time for the high-dimensional quadratic problem. This figure compares the wall-clock time required to reach convergence.
First, let define intermediate variables.
Kv. t= t=tnhhu)) s=sech((u) =1 r= at.
The gradient is,
[ -aaK" (Diaag(s))) y + Ar
VG(z,y)
The Hessian is,
2 G( y)
V2
sy G(z,z)
V²G(z,v)
V ya G y)
V
V G(r,y)
39

---

where,
2 G(r,)) v) = I Xa² K Diag(s) + Aork Diag(g) na
G(v,v) - -aaK (
V 2a G( (s)Diag(s))
V yy G(x,y) - -1+
In the V block, is vector defined as 2r sL The reduction mapping is defined by the nonlinear manifold Lan)(Kx))
((x) (a tanh Kx)
The reduced function is,
F(x)= = 2 2 2/1a tanh((a)]
|=:1 +
This problem structure, minimising (x))11 is nonlinear least-squares problem. The gradient and Hessian of the reduced function are given below,
F((x) + a² Diag(s)t and V2F(x) = a² Diag(g)
where y -
1Ooos 8. The pullback metric R De D& given by,
I=I+ a K x D
In this setting, our geometrically preconditioned gradient descent, which uses R as the preconditioner, is precisely equivalent to the Gauss-Newton method applied to the reduced function. This is a special case the the function nonlinear least-squares problem. The experimental results for the nonlinear case are presented Figures TO 11 and. T2 The learning rates for each method are set based on Armijo's backtracking line search.
Objective convergence
Gradient norm convergence
106
100
103
10-1
Objective (log)
10-12
[|rrad]| (log)
10-1
18-
15
10-1
10-7
10
30.
10
10.5
10
30
10000
10
10000
unconstrained
Full unconstrained Euclidean
10
Reduced Reduced PrecGD
10-16
Reduced PrecGD
0.00
0.25
0.50
0.75 1.00 1.25 1.50
1.75
2.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Iteration
le6
Iteration
le6
Figure 10: Convergence rate for the high-dimenssonal nonlinear problem.
These examples demonstrate the flexibility of our proposed framework. While it is general enough to apply to any objective function G and mapping 4, these experiments show that for common problem classes, it naturally recovers well-known, powerful optimisation algorithms such as the Newton and Gauss-Newton methods.
40

---

Reduced Hessian eigenvalues
Reduced Hessian spectral M Ad
10a
10r
lambbda (log)
10
Gap (lgg!
10a
Arac
10a
Index
Index
Full (x.y) Hessian eigenvalues
Full Hessian spectral gaps Mi+1 - A.l
2.0 x 10
10a
10
10a
lambda (log)
20
Gap (log)
10a
10a
10
Amac
10a
Amin
10
20
30
40
60
70
10
20
40
60
70
80
Index
Index
Figure 11: Hessian eigenspectrum for the high-dimensional nonlinear problem.
19592.2x
10a
Wall time (5)
20a
198.9x
10a
1.0x
Full
Reduced
Reduced Prec
Figure 12: Wall-clock time for the high-dimensional nonlinear problem, comparing the time to convergence. Note that the Reduced+PrecGD wall-clock time also includes the conjugate gradient steps needed to solve for the preconditioning at each iteration.
41

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We analysed the impact reduction mappings on local convergence rates in nonconvex landscapes. To support our analysis, we derived number of theoretical results.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss our limitations of our analysis in Section 4.1 of the paper.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
For all our theoretical results, we have provide full proof' the appendix and further proof sketches and analysis in the main paper. All assumptions are stated explicitly throughout the paper.
Justification:
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper is mainly a theoretical work but we have include some synthetic experiments for validation.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: provide sufficient instructions to reproduce the results.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We outline experimental setup in Appendix G.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Experiments were run for multiple seeds.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Answer: [Yes]
42

---

Justification: We outline our experimental setup in Appendix G.
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The paper conforms with the NeurlPS code of ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is societal impact of the work performed in this paper.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not pose such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The paper does not use existing assets.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicabble, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
16. Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Question:
Answer: [NA]
Justification: The core method development in this paper does not involve LLM usage.
43


Reviews:
[{"review_id": "B3pxd4qBR7", "reviewer": "Reviewer_6kyE", "summary": "This paper presents a formal and general framework for understanding and quantifying the benefits of using \"reduction mappings\" in nonconvex optimization problems where the set of minimizers forms a manifold. The core idea is that by reparametrizing the problem to stay on a manifold that encodes known optimal structure, one can provably improve the local geometry of the objective function, leading to faster convergence for gradient-based methods.", "questions": "1. On the Practicality and Origin of the Reduction Mapping Ψ.\n\nCould the paper provide a concrete, illustrative example from the literature cited (e.g., neural collapse [14], dictionary learning [22], or phase retrieval [31]) and explicitly formulate the corresponding reduction mapping Ψ? For instance:\nIn the context of neural collapse, where optimal classifiers collapse to an Equiangular Tight Frame (ETF), is Ψ a projection onto the ETF cone?\nIn low-rank matrix factorization, min ||X - UVᵀ||², where solutions are only unique up to an invertible transformation (UR, V(R⁻¹)ᵀ), how would one define Ψ to fix this ambiguity?\n\n2. On the Verifiability of the Key Geometric Assumptions (ε and Δ)\n\nCould the paper provide more intuition on the plausibility of these assumptions in real-world machine learning settings? Could you further connect your assumption of a spectral gap to the empirical findings on Hessian spectra in deep learning [66-70], which you cite? Does the \"few large outlier eigenvalues\" phenomenon directly support the existence of a Δmax > 0?\n\n3. Lack of Experimental Results.\n\nHow large are the \"spectral gaps\" (Δmax, Δmin) in practice? How does the performance degrade as the assumptions (e.g., non-alignment) are gradually violated? It would help to answer these practical questions if experimental analysis could be involved.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The main strength is the move from qualitative intuition to quantitative results. The paper provides explicit bounds on the new smoothness and sharpness constants. This requires non-trivial mathematical machinery, likely involving matrix analysis and differential geometry (as hinted by the use of pullback metrics).\n\nThe framework is powerful because it's general. It applies to both affine and nonlinear reductions and covers a wide range of practical scenarios mentioned in the introduction (neural collapse, matrix factorization, phase retrieval). It provides a principled explanation for why techniques like fixing parts of a model or using certain projections can accelerate training.\n\nWeaknesses:\n\nWhile the theoretical contribution is strong, there are areas that could be discussed for a complete picture, which is typical for a focused research paper.\n\n1. Purely Theoretical: The paper is entirely theoretical and lacks experimental validation. While this is acceptable for a theory paper, even a simple numerical experiment on a synthetic problem (like the one in Figure 1) would make the theoretical gains more tangible. \n\n2. Practicality of the Mapping Ψ: The framework assumes the optimal structure, and thus the reduction mapping Ψ, is known. In many real-world problems, this structure might only be known approximately, or it might emerge during training. The analysis doesn't account for errors in the mapping Ψ.\n\n3. Computational Overhead: The proposed algorithm, Preconditioned Gradient Descent (PrecGD), requires computing and inverting the matrix R = DΦᵀDΦ. The paper acknowledges this cost but states that designing efficient solvers is \"beyond the scope of this work.\" This is a fair statement, but it is the key barrier to making this framework immediately practical for complex, nonlinear mappings.", "was_revised": true, "final_justification": "The rebuttal addresses the concerns and questions well. The score is raised. Solid work.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We address each point below.\n\n**Weaknesses:**\n\n**1)** We agree that numerical experiments can enhance intuition for the theory. In response, we conducted synthetic experiments illustrating the predicted iteration complexity improvements. These build upon Example 1 (Appendix A) and will appear in the final version with visualisations and analysis. All experiments were repeated with varied seeds/initialisations and results averaged. \n- **Example 1: Quadratic Toy Problem (Figure 1)**  \n  Canonical function:  \n  $$f(x_1,x_2) = x_1^2 + 10(x_2 − x_1)^2$$  \n  with reduction mapping $x_2^\\star(x_1) = x_1$, yielding reduced problem $F(x_1)=x_1^2$. GD is used with maximal step sizes.\n- **Example 2: High-Dimensional Quadratic**  \n  Ill-conditioned problem ($K \\in \\mathbb{R}^{d\\times d}, d=40$):  \n  $$G(x, y) = \\tfrac{1}{2}||x||^2 + \\tfrac{1}{2}||y||^2 + \\tfrac{\\lambda}{2} ||y - Kx||^2$$  \n  Reduction via $y = Kx$ gives:  \n  $$F(x) = \\tfrac{1}{2} x^\\top (I + K^\\top K) x,\\quad R = I + K^\\top K$$\n- **Example 3: Nonlinear Mapping (Tanh)**  \n  Reduction $\\Phi(x) = [x; \\alpha \\tanh(Kx)]$, full objective:  \n  $$G(x, y) = \\tfrac{1}{2}||x||^2 + \\tfrac{1}{2}||y||^2 + \\tfrac{\\lambda}{2} ||y - \\alpha \\tanh(Kx)||^2$$  \n  Reduced:  \n  $$F(x) = \\tfrac{1}{2} ||x||^2 + \\tfrac{1}{2} \\alpha^2 ||\\tanh(Kx)||^2$$  \n  with pullback metric:  \n  $$R(x) = I + \\alpha^2 K^\\top \\mathrm{diag}(\\mathrm{sech}^4(Kx)) K$$\n\n|  Iterations to Convergence         | Full Problem  (GD) | Reduced (Euclidean GD) | Reduced (Pullback + PrecGD)                                  |\n|:---------:|:------------------:|:----------------------:|:------------------------------------------------------------:|\n| Example 1 | 800                | 1                      | 1                                                            |\n| Example 2 | 132,000            | 9,000                  | 1                                                            |\n| Example 3 | > 2,000,000        | 28,000                 | 20 (+1400 CG steps) |\n\nThese results confirm that reduction significantly reduces iteration counts and preconditioning further enhances this—even for high-dimensional or nonlinear problems.\n\n\n**2)** While our theory assumes exact reduction mappings $\\Psi$ for clarity, many practical cases admit explicit forms. Examples include BatchNorm, WeightNorm, or projections onto structured manifolds (sphere, Stiefel), often implemented via QR/polar decompositions. These are exact (not iterative) and fit our framework.\nMappings derived as a solution of inner optimisation problems (e.g., balanced factorisations or ETF projections) are indeed approximate. We acknowledged this in Section 4.1, and handling such mappings is a key future direction.\n\n\n**3)** Inverting the pullback metric $R = D\\Phi^\\top D\\Phi$ adds some overhead, but is tractable in many settings:\n\n- **Linear $\\Phi$**: $R$ is constant with closed-form inverse or low-rank structure; cost is negligible.\n- **Structured nonlinear $\\Phi$**: $R(x)$ often has favourable algebraic form (e.g. diagonal dominance, low-rank perturbation of identity). Efficient inversion via Woodbury or matrix identities is possible.\n- **Stable conditioning**: Well-behaved $\\Phi$ (such as nonlinear projection on manifolds) keeps $R(x)$ stable, enabling iterative solvers (e.g. conjugate gradient) with only a few steps, hence they are not substantially increasing the wall-time performance as we see in Example 3.\n\nThus, while efficient large-scale implementation is indeed an important direction beyond the current scope, we see no fundamental barrier to scalability in many settings. We have updated the discussion in the paper to better reflect these considerations.\n\n---\n\n**Questions:**\n\n\n**1)** *Neural Collapse*: In the context of Neural Collapse, the reduction mapping $\\Psi$ can be viewed as fixing classifier weights (equivalent mappings can be formulated for hidden layers via Deep NC characterisations) to a canonical or random ETF form, eliminating orthogonal symmetry. This constant mapping fits our framework.\nA richer formulation defines $\\Psi$ implicitly via the nearest-ETF projection, as in [46]:  \n$$U^\\star \\in \\arg\\min_{U\\in St(d,C)} ||\\bar{H} - UM||^2_F + \\tfrac{\\delta}{2} ||U - U_{\\text{prox}}||^2_F$$  \nThis defines $\\Psi(\\bar{H}) = U^\\star M$. Empirical validation in [46, Figs. 3,4] shows faster convergence and improved conditioning, matching our theoretical predictions.\n\n\n*Low-rank Factorisation*:\nHere, the solution is non-unique due to $\\mathrm{GL}(r)$ symmetry: $(U, V) \\mapsto (UR, V(R^{-\\top}))$. Quotient manifold methods handle this via equivalence classes, but do not favour specific representatives.\n\nOur framework allows explicit representative selection—fixing a gauge—which is crucial when structure or regularity is needed (e.g., in generalisation).\n\nFor example, an interesting choice is balanced factorisation: $U^\\top U = V^\\top V$. Let $A:=U^\\top U, B:=V^\\top V$. For any feasible $(U,V)$, compute the matrix geometric mean\n$$X: = A^{-1/2}(A^{1/2}BA^{1/2})A^{-1/2}$$  \nand choose the principal symmetric PD square root:\n$$R(U, V) := X(U,V)^{1/2}$$  \nso that $U^\\top U = V^\\top V$. This defines a reduction $\\Psi(U,V) = (UR(U,V), V(R(U,V)^{-\\top}))$.\nUnlike quotient optimisation, which simply eliminates symmetry, our mapping selects a preferred/specific representative from the equivalence class with desirable properties. Balancedness improves stability and generalisation in tasks like matrix completion and low-rank deep models.\n\nWith this framework, we give practitioners the ability to act with principled control over the solution structure, enabling them to design problem-specific mappings that not only resolve symmetries but also accelerate convergence toward desirable and application-relevant solutions. More broadly, many important applications motivate the need for structure-aware optimisation. In this paper, we have chosen to focus on developing the general theoretical framework. In upcoming work, we will provide a detailed treatment of key applications—illustrating how reduction mappings and the associated geometry can be leveraged in practical settings across machine learning and general nonconvex problems.\n\n**2)** Τhe assumptions made in our framework are both plausible and robust in real-world machine learning settings. For the spectral gap $\\Delta_{\\max}$​, the assumption is satisfied in any anisotropic Hessian, which is virtually always the case in modern deep networks. The only exception is the isotropic case (all eigenvalues equal), which is highly non-generic and mostly of theoretical interest. In this degenerate scenario, our reduction framework does not improve convergence over classical training—precisely as our theory predicts. More generally, as shown in Section E.4, symmetric matrices with non-simple spectra form a measure-zero subset, and our analysis explicitly accommodates eigenvalue multiplicities in the top eigenspace as long as there is a gap below.\nFor the non-alignment assumption $\\varepsilon > 0$, geometrically this requires that the tangent space of the reduced manifold does not contain the top curvature directions we aim to eliminate. Such near-perfect alignment is also non-generic and would require adversarially structured mappings or objectives. Moreover, even if alignment does occur, small, structure-preserving perturbations to the reduction mapping (e.g., reparameterisations, gauge fixes) are sufficient to break it—without altering the semantics of the model or predictor.\nRegarding the connection to deep learning Hessian spectra, we note that the goal is not merely to argue that $\\Delta_{\\max} > 0$—this is almost always true in nontrivial problems—but to emphasise that the magnitude of the spectral gap in practice can be very large, as shown empirically in [66–70]. This has two key implications: (i) our theoretical gains via reduction and preconditioning become more pronounced in such settings, and (ii) even in the purely Euclidean reduced case, the large $\\Delta_{\\max}$ leads to a strictly smaller smoothness constant, and hence faster convergence. This explains the accelerated behaviour observed in real-world examples like neural collapse and ETF-aligned features [46], even without preconditioning.\n\n**3)** In practice, spectral gaps $⁡\\Delta_{\\min}$ and $\\Delta_{\\max}$​ are typically positive and well-separated, as confirmed in our experiments and supported by prior work on deep learning Hessians. These studies consistently observe a bulk–spike structure, where a few large outlier eigenvalues are separated from the bulk of smaller ones. This directly implies a large $\\Delta_{\\max}$, while $\\Delta_{\\min}$, though smaller, remains non-negligible. Such structure provides a natural setting where our assumptions hold and the benefits of reduction and preconditioning are most pronounced. For completeness, we include the spectral gaps for the three synthetic examples from above.\n\n- Example 1: $\\Delta_{\\max} = \\Delta_{\\min} = 40$ (2D case)\n\n- Example 2: $\\Delta_{\\max} = 3540, \\Delta_{\\min} = 20$\n- Example 3: $\\Delta_{\\max} = 3.465 \\times 10^5, \\Delta_{\\min} = 1.646 \\times 10^{-6}$"}, {"review_id": "6jMCRh4AXZ", "reviewer": "Reviewer_rmzJ", "summary": "This paper proposes a new framework for analyzing the contribution that the local geometry of minimizers has on the optimization process. The framework is based on reduction mappings that reparameterize the problem in terms of the solution manifold, which reduces the dimensionality of the problem. When the optimization process incorporates a well chosen reduction mapping, the authors prove that the local smoothness and sharpness constants of the objective improve, leading to faster local convergence rates. This framework unifies a number of approaches, such as batch normalization and equiangular tight frame projections in deep neural networks, that utilize reduction mappings and gives a theoretical explanation for the observed benefits of these approaches.", "questions": "Are the results tight in the sense of matching previously known bounds? It is mentioned in the appendix that prior work has shown that in the case of batch normalization, the smoothness and sharpness constants are altered, just as in this work. In this case, does the proposed framework generate (roughly) the same improvement in these constants, or does the generality of the framework reduce the tightness of the theoretical results?\n\nThe curvature and spectral gap conditions are sufficient to guarantee the improved smoothness and sharpness constants. Is it known if these conditions are also necessary, or are there other conditions that could be satisfied for which a similar improvement is possible?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: This paper proposes a novel viewpoint to unify many commonly used techniques in nonconvex optimization that demonstrates the potential theoretical benefit of that incorporating a smartly chosen reduction mapping can have on the local convergence properties of first order methods. The proposed framework is novel and has a high degree of flexibility, allowing it to cover many interesting cases such as batch normalization, ETF reparameterizations, and bilevel optimization. Sufficient conditions are given under which a well chosen parameterization will yield improved local convergence properties for first order methods.\n\nWeaknesses: The paper mentions many potential applications of this framework, however, the authors do not apply their framework to any problem beyond some simple, 2-d examples given in the appendix. While the reduction mapping approach is interesting, it is hard to validate how effective it is at improving the understanding of the methods used without applying it to an example application, such as batch normalization.\n\nQuality: The paper is technically sound and provides an interesting viewpoint on how reduction mappings can have a beneficial impact on the local geometry of optimization problems. The framework is quite general and includes many interesting potential applications. However, no concrete applications of the framework is provided in the paper, which makes it hard to validate how tight the bounds generated by this approach are when compared with a more direct analysis on a case by case basis.\n\nClarity: On a whole, the paper is well written, however, some of the best motivation and explanation of reduction mappings is present in the appendix (section A). It would be ideal if some of this motivation was worked into the main body of the paper, to improve the introduction into the framework, which quickly becomes technical in the main paper.\n\nSignificance: The framework proposed is quite general and appears to apply to many interesting scenarios in nonconvex optimization. It seems that it could have a significant impact on how to view many strategies incorporated in modern optimization methods, though this is not entirely clear, as no concrete examples are evaluated using the proposed framework.\n\nNovelty: To my knowledge, this is the first work to use reduction mappings to unify a variety of techniques currently in use in nonconvex optimization. It provides a unique viewpoint, that could be used to not only better understand methods that are currently in use, but to design new techniques with improved performance.", "was_revised": true, "final_justification": "The authors responded to all of my questions in a satisfactory manner and I maintain my original recommendation of accept.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We address each point below.\n\n**Weaknesses:**\n\nIn response, we have added two new synthetic high-dimensional examples (see the response to reviewer 1 (6kyE) for details) illustrating the utility of reduction mappings and their associated preconditioning, with a detailed analysis of the iteration complexity for each method. Moreover, our framework applies directly to prior work such as [46], where the authors employed a nonlinear reduction mapping—derived from an argmin formulation—to optimise towards the nearest ETF in the neural collapse setting. Their results, on both synthetic and real-world data, demonstrate accelerated convergence compared to standard training, further validating the benefits of this approach.\nBatch normalisation can also be formulated as a reduction mapping. However, due to its reliance on batch statistics, the more interesting and practical mappings are inherently stochastic. The analysis of such stochastic reduction mappings is an interesting direction and is left for future work. In contrast, a particularly instructive and tractable case is weight normalisation (WN). Consider a single neuron with weight vector $w \\in \\mathbb{R}^d$. WN adopts a scale–direction parametrisation: \n$$\nw = \\Phi(\\theta) := g \\frac{v}{\\|v\\|}, \\quad \\theta = (v, g) \\in \\mathbb{R}^d \\times \\mathbb{R}.\n$$\nThis defines a (chart) representation of the sphere bundle $S^{d-1} \\times \\mathbb{R} \\to \\mathbb{R}^d$. In the reduction view, the unit-norm direction serves as a fixed gauge, and the known scale–direction structure is captured via $\\Phi$. Performing steepest descent in the pullback metric induced by $\\Phi$ yields exactly the preconditioned update recommended by our framework.\nLetting $n = v/\\|v\\|$, the pullback metric is\n$$\nR = \\mathrm{diag}\\left( \\frac{g^2}{\\|v\\|^2}(I - nn^\\top), 1 \\right).\n$$\nBecause $R$ is singular (due to redundancy in $v$), the correct preconditioner is its Moore–Penrose pseudoinverse,\n$$\nR^\\dagger = \\mathrm{diag}\\left( \\frac{\\|v\\|^2}{g^2}(I - nn^\\top), 1 \\right),\n$$\nor alternatively, in a minimal coordinate chart, the full inverse is\n$$\nR^{-1} = \\mathrm{diag}\\left( g^{-2} I_{d-1}, 1 \\right).\n$$\nIn both cases, the inverse is closed-form and computationally efficient. At the layer level, the Jacobian becomes block-diagonal (one block per row), so preconditioning reduces to a per-row projection onto the tangent space followed by a scalar rescaling. The full metric remains block-diagonal with a cost of $\\mathcal{O}(md)$ for a layer with $m$ neurons, which is negligible in practice.\nFuture work will address the stochastic extension of reduction mappings and explore more real-world applications. Our theory suggests that while normalisation layers are already known to improve training speed in standard Euclidean settings, performing geometry-aware preconditioning through the pullback metric can yield further improvements in convergence—without additional wall-time overhead—since the metric inverses are inexpensive to compute.\n \n*Clarity:* Thank you for the suggestion. Due to space constraints, we couldn’t fit the whole motivation in the main text. In the final version of the paper, we will provide additional motivation in the main text. \n\n**Questions:**\n\n**1)** For the case of batch normalisation bounds, our results are tight in the sense that we can move between the two results in a straightforward manner. Specifically, when the batch normalisation transformation is interpreted as a reduction mapping $\\Phi_{\\text{BN}}$​, the pullback metric becomes a projector–rescale operator, and our framework recovers the same structural improvements in smoothness and gradient norms observed in prior BN analyses [44]. The multiplicative factor $\\gamma^2/\\sigma^2$ and the projection onto the orthogonal complement of the mean and normalised input directions emerge naturally from the geometry of the mapping and match the scaling and projection terms in the BN gradients.\nThe difference lies in how the second-order effects are handled. In our general setting, when $\\Phi$ is nonlinear, the curvature correction term involving $D^2\\Phi$ is upper-bounded generically by the abstract quantity $QZ/m(\\Phi)$, which captures the interaction between the curvature of the mapping and the gradient of the loss. In contrast, the BN analysis computes these same curvature terms explicitly using the algebraic structure of the batch statistics, leading to closed-form expressions for the full Jacobian and its effect on the gradient and Hessian. In this sense, the BN bounds can be seen as a concrete instance of our general result, where the curvature term is evaluated exactly rather than bounded.\nTherefore, up to this difference in treatment of curvature, the two sets of results are equivalent in spirit and mechanism. Our bounds can be viewed as a generalisation of the BN case, and when applied to the BN setting, they yield bounds that are tight up to the curvature correction. In particular, when the reduction mapping is affine—as in frozen BN—the curvature term vanishes, and our bounds match exactly. For nonlinear cases (e.g., batch-dependent BN), the tightness of our bound depends on how well the curvature correction is controlled, but the mechanism of improvement remains the same. For a general setting, our results are minimax-tight. \n\n\n**2)** The curvature and spectral gap conditions we impose are sufficient but not necessary for achieving improved smoothness and sharpness constants. In particular, they are not required to guarantee a qualitative improvement — that is, an improvement may still occur in their absence. In fact, the necessary and sufficient condition for a strict reduction in curvature appears in the Courant–Fisher formulation used in Lemma 4: the improvement occurs if and only if the feasible tangent space avoids the top eigenspace of the Hessian.\nWhat the curvature and spectral gap conditions provide is a quantitative strengthening of this principle. They turn the abstract variational condition into a concrete, verifiable, and practically useful bound. In that sense, our assumptions are best viewed as sufficient conditions for controlled and predictable improvement, which can guide algorithmic design and offer explicit guarantees in real-world applications."}, {"review_id": "EGyEnV70kq", "reviewer": "Reviewer_1oNF", "summary": "This paper is about using reduction mappings to exploit geometric structure of the solution set to accelerate convergence. It characterizes when reduction mappings lead to provable gains in optimization efficiency, i.e. optimizing on a manifold with a landscape that makes optimization easier for first-order methods. They show that under affine or nonlinear reduction mappings, the smoothness (Lipschitz) constant of the reduced function $F$ is strictly smaller than that of the original function.$f$. Under nonlinear reduction mappings, they also show that the sharpness constant is improved. Thus, the better condition number $\\kappa$ of $F$ over $f$ and faster convergence through the use of methods such as gradient descent where the rate depends on $\\kappa$.", "questions": "- Neural collapse is observed in stochastic settings. Could faster convergence via reduction mappings also be observed empirically?\n - Preconditioning is recommended to take advantage of the structure from reduction mapping, but in practice, exact applications of preconditioning is difficult. Is there an approximate form of the pullback metric $R(x_1)$ that is simpler, e.g. diagonal?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Paper is very clearly written. Their framework ties together preconditioned methods, normalization methods, bilevel optimization and other domains via reduction mappings. \n - Authors give provable convergence improvements of nonlinear reduction mappings by showing that both the smoothness and sharpness constants are improved in the reduced \n\nWeaknesses\n - Could be difficult to understand how to use this from a practical standpoint. While this is a theory paper, pseudo-code showing how the algorithm could be implemented in practice and experimental results (even with synthetic data) could make this easier to grasp for practioners.\n - Some limitations were acknowledged by the authors: local analysis only, the assumption of exact evaluation of $\\Psi(x_1)$, the unknown (problem dependent) cost of evaluating $\\Psi(x_1)$, analysis limited to deterministic setting", "was_revised": true, "final_justification": "I appreciate the additional experiments and the authors response to my concerns. I maintain my rating at Accept.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We address each point below.\n\n**Weaknesses**\n\n**1)** We agree that practical clarity is important, even in a theory-focused paper. In response, we have included several synthetic experiments that demonstrate the practical benefits of our approach (see the response to reviewer 1 (6kyE) for details). These include:\n- a 2D toy example (Example 1, Appendix A),\n- a high-dimensional linear problem with ill-conditioned structure, and\n- a high-dimensional nonlinear experiment using a $\\tanh$-based mapping.\n\n\nThese experiments empirically validate the iteration complexity improvements predicted by our framework.\nAdditionally, we highlight that our framework connects with real-world settings such as neural collapse and ETF parametrisation, as explored in prior work [46]. We plan to apply our framework and cover more complex settings and real data applications in future work. \n\n**2)** Indeed, the limitations mentioned—local analysis, assumption of exact mappings, deterministic setting—are important and we fully agree that they offer meaningful avenues for future research. Our current focus was to establish a rigorous foundation, but we view extending the theory to stochastic and approximate regimes as a natural and promising next step.\n\n**Questions:**\n\n**1)** Yes, faster convergence of neural collapse due to reduction mappings has already been observed in stochastic settings in prior work [46]. In particular, the empirical results in that paper show significant gains under SGD when using ETF-reparametrisation mappings. Extending our theoretical framework to analyse convergence in such stochastic regimes is a key future direction.\n\n**2)** We agree that exact application of the pullback metric can be computationally expensive in general. However, in many structured applications, the metric admits efficient forms:\nLinear or affine mappings yield a constant $R$ that is easy to compute and invert.\nStructured nonlinear mappings (e.g. arising from normalisation layers or architectural constraints) often produce diagonal, block-diagonal, or low-rank $R$, enabling efficient approximations.\nFor more general nonlinear cases, approximate preconditioning techniques (e.g. diagonal approximation, conjugate gradients, or low-rank updates) can be used to trade off computational cost with fidelity to the pullback geometry.\nThese situations are common in practice, and we believe they offer a rich space for practical extensions of our theory, which we intend to explore further in future work."}]

Author rebuttal:

Meta-review: This is a strong and well-written paper that introduces a reduction-based framework for analyzing convergence rates of first-order methods in nonconvex optimization. The core idea is that by reparametrizing the problem to stay on a manifold that encodes known optimal structure, one can provably improve the local geometry of the objective function, leading to faster convergence for gradient-based methods. The framework proposed is quite general and appears to apply to many interesting scenarios in nonconvex optimization. It seems that it could have a significant impact on how to view many strategies incorporated in modern optimization methods. All the concerns raised during the reviewer discussions were adequately responded to in the rebuttal. This paper makes a valuable contribution and should be of broad interest to the NeurIPS community.
Decision: Accept (spotlight)