Constrained Optimization From a Control Perspective via Feedback Linearization
Authors: Runyu Zhang
Arvind Raghunathan
Jeff S Shamma
Na Li
Venue: NeurIPS 2025 poster ()
Keywords: Constrained Optimization; Control Theory; Feedback Linearization
Primary area: theory
TL;DR: This paper develops a feedback linearization framework for constrained optimization, with convergence guarantees, a connection to SQP, and a momentum-accelerated variant.

Abstract: Tools from control and dynamical systems have proven valuable for analyzing and developing optimization methods. In this paper, we establish rigorous theoretical foundations for using feedback linearization—a well-established nonlinear control technique—to solve constrained optimization problems. For equality-constrained optimization, we establish global convergence rates to first-order Karush-Kuhn-Tucker (KKT) points and uncover the close connection between the FL method and the Sequential Quadratic Programming (SQP) algorithm. Building on this relationship, we extend the FL approach to handle inequality-constrained problems. Furthermore, we introduce a momentum-accelerated feedback linearization algorithm and provide a rigorous convergence guarantee.

Constrained Optimization From a Control Perspective via Feedback Linearization
Runyu Zhang
Arvind Raghunathan Mitsubishi Electric Research Laboratories
Massachusetts Institute of Technology runyuzha@mit.co edu
raghunathan@merl.com
Jeff Shamma
Na
University of Illinois Urbana-Chama Obbana-Champaign j jshamma@i wit.mmiilliiiii.iiimiim edu
Harvard University nali@seas. harvard. edu
Abstract
Tools from control and dynamical systems have proven valuable for analyzing and developing optimization methods. In this paper, we establish rigorous theoretical foundations for using feedback linearization (FL)-a well-established nonlinear control technique-to solve constrained optimization problems. For equality- constrained optimization, we establish global convergence rates to first-order Karush-Ko Maeiush-Kuun-TTie Kaan-KuunnnTeeeee (KKT) points and uncover the close connection between the FL method and the Sequential Quadratic Programming (SQP) algorithm. Building on this relationship. we extend the FL approach to handle pequualit-cconsrainee problems. Furthermore, we introduce nomentum-acceeerran feedback lin- earization algorithm and provide rigorous convergence guarantee.
Introduction
Constrained optimization, also known as nonlinear programming, has found vast applications in sev- eral domains including robotics (2). supply chains [30], and safe operations of power systems [23]. First-order iterative algorithms are widely used to solve such problems, particularly in optimiza- tion and machine learning settings with large-scale datasets. These algorithms can interpreted as discrete-time dynamical systems, while their continuous-time counterparts, derived by considering infinitesimal step sizes, take the form differential equations. Analyzing these continuous-time systems provide valuable theoretical insights, such stability properties and convergence rates. This perspective is well-developed for unconstrained optimization, exemplified by the gradient flow the continuous time counterpart of gradient descent, as well as its However, for constrained optimization, this approach remains less
i Vf 126
6
70
31
accelerated variants 75
79
51.
thoroughly explored.
16, ) have explored the dynamical properties continu- ous time constrained optimization algorithms. These works leverage a feedback control perspective to design analyze the performance of optimization methods. Specifically, they propose frame- works that model constrained optimization problems as control problems, where the iterations of the optimization algorithm are represented by dynamical system, and the Lagrange multipliers act control inputs. The objective in this framework is to drive the system feasible steady state that satisfies the constraints. Within this framework, various control strategies can be employed to design the update of Lagrange multipliers, resulting in different control-based first-order methods. For example, it can be shown that Proportional-Intege (PI) control leads to Primal-Dus Gradient
Recent studies (c.f. [ 35
Dynamics (PDGD), whose properties well-studied [46] 66, 22 However, most of the works focuses on convergence for convex constrained problems.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we adopt the same control perspective as above, and specifically focus on using another approach. namely Feedback Linearization (FL) a standard approach nonlinear control (ef. 45 41)), to design the Lagrange multiplier. One key advantage of this method is its natural suitability for handling nonconvex constrained optimization problems. Although this approach [15), along with similar dynamical system perspectives
[72 I [2] 54), has been explored in the literature, its
theoretical properties are not yet fully understood. Several important questions remain open.
The first question concerns global convergence and convergence rates. While existing works estab- lished local stability [5]. global convergence and convergence rate have not been established. The second question concerns the relationship between the feedback linearization approach and existing optimization algorithms, specifically whether the discretization of the optimization dynamics de- rived from feedback linearization aligns with any known optimization method. Additionally, since most existing studies [5 72 focus exclusively on equality constraints, this raises the third ques- tion: how can the feedback linearization approach be extended effectively for inequality constraints? Lastly, it remains unclear whether ideas from acceleration in optimization- such as momentum- based techniques-can be incorporated to speed up feedback lienarization methods for constrained opti- mization.
Our contributions. Motivated by the open questions discussed above, we aim deepen the the- oretical understanding of the feedback linearization (FL) approach for constrained optimization by addressing these questions. Specifically, our contributions are as follows:
1. We establish global convergence rate first-order FL method for equality-constraines optimization (Section
(KKT) point for the
2. We demonstrate that the FL.-based optimization algorithm is closely related to the Sequential Quadratic Programming (SQP) algorithm, providing new perspective its connection to tablished optimization techniques (Section 3.2).
3. Building on this insight, we extend the method to handle inequality constraints, broadening its applicability (Section 4
Finally, we propose a momentum-accceennal empirically exhibits accelerated convergence in both equality constrained and inequality con- strained settings. Furthermore, we establish O vT convergence in the nonconvex equality con- strained setting (Section 5)
FL algorithm for constrained optimization, which
Due to space limits, a comprehensive review of related literature is deferred to Appendix
We use the notation [n],n E N to denote the set {1, 2,, ........ n). to denote the gradient of a scalar function f IR"" -> IR evaluated at the point x E R" and use 2 (x) to denote its corresponding Hessian matrix. We use (x) to denote the Jacobian matrix of a function : RR" + T evaluated at x E RR", i.e. [J)(=)) = (h,(z) i E [m], E [r]. Unless day
Notations:
We use (xx)
specified otherwise, we use || || to denote the L2 norm of matrices and vectors and use || 1 to denote the Tooe norm. For a positive definite matrix A, we use |X|| == || 2 X|] to denote the A-norm of X. For a set A, we use A to denote its complement. When no ambiguity arises, we denote de by ă, and abbreviate time-dependent variables such as xtt), A(t), and y(t) as x, X, and y.
dz
2 Feedback Linearization (FL) for solving equality constrained optimization
In this section, we briefly review related works that adopt control perspective, particularly focusing on the use of feedback linearization (FL) to address equality-constraiee optimization problems.
Control perspective on equality-constrainee optimization mization problem with equality constraints
15
Consider the constrained opti-
min f()
s.t. h(x)) = O,
(1)
where x FT F R" R R" R Here we assume that f, are differentiable, and additional assumptions will be introduced where needed to support the analysis. The first-order KKT conditions are given by
-V((x) - (n(x) = 00, h(x) =

---

The key idea is to view finding the KKT point a control problem (Figure with the system dynamics given by,
dx -(((tt)) (((()) ((t)),
y(t) h(x(t)),
where x represents system state, y - h(x) is system constraint variable and À is the control input. T(x) here is positive definite matrix and throughout the paper assume that there exists Amins Amax such that for all z,
Amin 3 Txx) 3 Amax
Note that at an equilibrium point x² of the system Fig. must satisfy: 2 = - Vf(x*) J/ (
Plant (f())+
Further, if x* is feasible, i.e. h(x*) = O, then get that x² satisfies the first order KKT conditions (2). Thus, the key idea manipulate the evolution of so that we stabilize the system to equilibrium and feasibility.
a. T(x)
Controller
Figure 1: Control Perspective for
To measure convergence, We define the KKT-gaap of As as follows, such that converging to a KKT point is equivalent to KKT-gap is zero
Constrained Optimization
KKT KKT-gap >) == max VV(() + (I,
(4)
To design the controller X(t) to reach feasible equilibrium, next introduce the feedback lin- carization (FL) approach, which is the main focus of this paper.
Feedback lincarization (FL) for cquality-constrainee optimization 15 (FL) [45 41 is classical control method for controlling nonlinear dynamics which generally takes
Feedback linearization
the following form:
à F(x) G(z)A
(5)
Directly designing stabilizing controller for the nonlinear system is a challenging task. The FL approach circumvents the difficulty by transforming the nonlinear control problem into an equivalent linear control problem, which is much easier to analyze, through a change of variables and a suitable control input. In particular, if G(z) ) is invertible, then one can let À := G(x) ((x)=110 (2 u()) is the new control input to be designed. Substituting À = G(x)
F(x)) where
F())) into the dynamics
5), the dynamics becomes - = u, which is now a linear system.
Similarly, in the equality constrained optimization problem, we write out the dynamics for y:
ý Jn(x) = Ja(x) Vf(x)
(=) A
F(x)
G(x)
Thus, by setting
A = (J,(CT)
(
we have that ÿ = u. then we can simply set u = - Ky where K is a Hurwitz matrix to guarantee that y asymptotically converge to zero. Thus the feedback linearization (FL.) dynamics given as:
Equality-Constrranee Constrained Optimization
15
à = -T(x) (VS(x) + Jx()
(6)
A = (X (TACO Kh(z))
The FL approach is particularly effective for handling nonlinear dynamics, making it well-suited for nonconvex constrained optimization. Numerical results in 15 72 highlight its strong performance
In this paper, we use term 'convergence' to refer to decay weaker than convergence to a local or global optimum, remains a meaningful guarantee in nonconvex constrained optimization. Extending the framework to incorporate saddle-pon point escape techniques and ensure convergence local optima is important direction for future work.
KKT gap zero. While this

---

in such settings. However, its theoretical properties remain less well understood. Existing analyses primarily focus on local stability [1], while global convergence and convergence rates are largely unexplored. Additionally, the connection between the FL. algorithm and existing optimization meth- ods is not well established. also remains unclear how to leverage the FI. approach to develop optimization and faster constrained optimization. In the following sections, we will systematically address these open problems.
novel techniques for
Remark (Scalability and Computational Complexity). Note that although (6) requires calculat- ing the matrix inversion of Jx())T()) (x)x)(())))aaaaa (x) Jx(()" constraints m instead of the dimension of the optimization variable z E IR.". In many practical set- tings, e.g. safe RL, the number of constraints is significantly smaller than the dimension of z. In such cases, the inversion is computationally inexpensive and can be performed efficiently. Moreover, even in cases where the number of constraints is large and the matrix inversion becomes computationally burdensome, we show that it is possible to approximate the inverse efficiently while still maintaining convergence guarantees. Specifically, in Appendix B.2 we present modified FL algorithm that incorporates a aepportional-ntegnnn (PI) controller to tolerate approximation error and still ensure convergence an optimal solution.
E Rmxxm the dimension scales with the number of
Remark 2 (Extensions of the FL approach). Although similar dynamics to 66) has also been pro- posed from other perspectives such as control barrier functions [., nonsmooth dynamics design 153 555, we believe that the feedback linearization perspective provides more general framework. In principle, beyond specifying a linear target of the form à = -yy, this approach allows us to leverage arbitrary stable controllers, such as PI controllers, e.g.
y - Ki So y(s)ds
for constraint enforcement, potentially offering new algorithmic behuviors or robustness benefits. In Appendix B.2 we present an example where the algorithm in 77) succeeds in converging to the optimal solution, even when only an inaccurate approximation of (x)=))(()J<<()))) able-while 4 fails to converge to an optimal solution. This highlights the robustness and broader applicability of the feedback linearization frumework.
is avail-
3
FL control method: Convergence and Relationship to SQP
3.1 Convergence Results
Section 2 introduces the FL method for equality-constrained optimization. The analyses in existing works mainly focus on the local stability, and little is known about the global convergence property. In this section, we establish global convergence rate first order KKT point (Contribution
The result relies on the following assumptions:
Assumption 1. There exists constant M such that f[) < M, [[]((=)] < M for all x; Assumption 2. The function fx) is lower-bounded, i.e. f(x) 2 smin for all x.
Assumption 3. There exists constant D such that (J)(=)J D2I for all x.
Note that Assumption 3 is similar to the assumption made in [5] that assumes that rank (J))))())) = m for all x, which is equivalent to Jn JA (w)) being invertible, thereby ensuring the regularity of the transformation in (6) and the existence of a well-defined feedback linearization. This assumption is also known as the lincar independence constraint qualification (LICQ. ef. 57.. see more discussion in Appendix(A] in optimization literature. Assumption [ jimplies that the functions f and g are Lipschitz. We would like to acknowledge that this assumption is relatively restrictive and is solely for analysis purpose In our numerical simulations we found that the algorithm is suitable for non- sanffrrnl-Liisshhie functions. We now state our result in terms of the convergence rate: [
Theorem 1. Let Assumption 7 2 and 3 hold and let the control gain K be a diagonal positive definite matrix, i.e., K = diag Ti=p where k; > 0. Then we have that the dynamic of the feedback linearization method (6) satisfies:
We note that if x² E D is known a priori for a compact domain D, potential approach for handling non con-uniformly Lipschitz functions f.g is to construct Lipschitz extensions f", such that their gradients and Jacobians match those of f.g 9 within D while remaining uniformly Lipschitz outside D (cf. (74).

---

1. For the set & := fo: h₂(x) 2 03. if z()) E &, then x(t) E fgr all 2 0. Similarly, if
x(0) E E, then x(t) E ε for all 2 O, further hy(x(t)) = (=)()) ie, h(x(t)) - with an exponential rate as t t00.
2. Define f(x)) = =(x) +
Amax
(MD) E 1=1 [,,(x)), then {(x(t)) is non non-incrennt con-increasing w.r.t. t.
Awin
3. Let X(t) == (J J [x(=)I()\Vf(((t) then we have that
(V)(((()(()) + + (x(((()) ((D)) X V -((=(0))((=(T)) - =(x(T)))
and that lim (A(t) - ((t)- =
4. (Asymptotic convergence and convergence rate) The above statements imply that,
infesegt KKfosgsrKKT-gapaattee (KTggap(a(t), max {
(
((e(v))-/mi + A
Σ. ExIb((<(0)1).
Amin
Amiin"
maxisism (x((x(0))
4x }} 0 (+)
further, we have that lim KKT gaz(T)t) X(t)) = 0, lim KKT- A(t)) = 0.
Statement in Theorem | implies that the algorithm can find an c-fist-order--KKT-oinn within time 22. We note that ensuring last-iterate convergence in nonconvex optimization is generally challenging. Hence, our analysis focuses on the best iterate, a widely adopted criterion in nonconvex optimization. However, in the setting where the optimization problem (T) is strongly convex, we are able to strengthen our convergence result to the last iterate convergence to the global optimal solution. Due to space limitations, we defer the detailed proof of Theorem the strongly convex setting to Appendix C The key step of the proof involves constructing the merit function f(x) in Statement 2. We also note that {(x) also serves as the exact penalty function in constrained optimization literature (cf. (27 84
1
[ as well as the result
3.2 Relationship with SQP
The FL. dynamics (6) provides concise and elegant formulation, prompting the question of whether certain optimization algorithms can be derived through its discretization. In this section, we establish fundamental connection between the continuous-time FL dynamics and the Sequential Quadratic Programming (SQP) algorithm (Contribution [ Specifically, we demonstrate that the forward-Fulle discretization (cf. g of (6) is equivalent to the SQP algorithm.
The state space continuous time dynamic for (6
4 ((x) Vf(x) ((=()(((())))))))))))))))))())))
-Kh()))) Ja (x)) (T())))))))))))))))) (UATTTTTTTT (MGTT(/VV))--X(())))))) Kh(z))).
Its forward- -ullr discretization scheme is
Te-1 = -1/((())))
(M(a))))) | (MM()))T)))))))( kh(x))))
We now consider the following SQP method, which is widely discussed in literature (cf.
57 12
58
rt+1 = arg min, vf(z)) (x 21) + (x xx) T((x)) (x xi) s.t. h(x;) + (x:) xZ) 0
(9)
We are now ready to state the main result of this section, which demonstrates the equivalence (8) and 9
Theorem 2. Under Assumption when K 1 I, the discretization of FL 8 is equivalent to the
SOP algorithm 9p.
The proof of Theorem 2 leverages the fact that 9 satisfies the relaxed Slater condition. The detailed proof is deferred to Appendix D
5

---

Remark 3 (Choice of T())). Theorem 2 provides insights into the selection of T(x) for the FL approach. Different choices of (x)) lead to different rypes of SQP algorithms. Here we discuss two specific types of (x)). First, T(x) is set as the inverse of the Hessian matrix, iee, T(x) = (V*)(x))~ then 2 corresponds to the Newton-type algorithm where the quadratic term in the objective function is given by (x 24) T T It), which is widely considered in literature (cf. 157 (2)). For this specific type Tx:),
its corresponding FL dynamics (6) the FL- Newton method. However, in the setting where the Hessian information is not available, another choice of (x) is simply setting it the identity matrix T(x) = I, which is considered in recent works such as (58). In this case, the objective function resembles a proximal operator (cf. (13] 60)), hence we name this Fl-pprxiiaa method. Due to space limit, we defer a more comprehensive overview of SQP to Appendix A We would also like to emphasize that Fl.-prxiimal belongs to the class of firstoorder methods as update only requires the first-order information fz),, Jj (z).
Remark 4 (Comparison with other first-order methods). To this end, we briefty compare FL- proximal methods with other first-order approaches, including Primal-Diie Gradient Descent (PDGD). Projected Gradient Descent (PGD). and the Augmented Lagrangian Method (ALM). with further details in Appendix A PDGD has been well studied [46 77 666, but is largely limited to convex settings and may fail in nonconvex problems [87 5V. PGD also struggles with nonconvexity, as projections onto nonconvex sets are often intractable. ALM can handle nonconvex constraints, but each iteration requires solving a potentially expensive nonconvex subproblem. In contrast, when the constraint dimension is small relative to that of x, SQP-based methods like ours often perform
better in practice 34 49
4 Extension to inequality constraints
The above sections primarily focus on the constrained optimization setting with equality constraints This section aims address the question of whether we can extend to setting with inequality constraints (Contribution 3), i.e.,
[
min, (x)
s.t. M(x) 00
(10)
The KKT conditions for the above problem are given by
fv)()) Ax= 0, h(x) 0, 1 ATh(z)= =0
(1I)
To measure convergence, we define the KKT-gapp of the state variable a and nonnegative control variable À follows:
KKT-gap(a A) == max [[v + + (x) Al, (AT((x) max, [x))+)++}
where [x)]+ max(h, 0).
We can still view the problem as control problem whose corresponding dynamics can be written
as
# -T(x)) (())) y = Mx), X 420 O.
(12)
However, the problem becomes more complicated because we require the non-negativity constraints unclear how to guaran-
2 and complementary slackness X h(xx()) = At first glance, tee these conditions through the control process. However, inspired by the relationship with SQP algorithms, we carefully design a more intricate FL. controller as follows:
FL for Ercqualliy-CCostrriiin Optimization
i = -Txx) (Vf(x) + (x)
(13.1)
A = arg min X (1)
(MAOTT Kh(z)))
(13.2)
Here we assume that the optimization problem in equation 13.2 admits finite solution. We would also like to point out that (3.2) takes the form of the solution of an optimization problem, resulting in non-smooth trajectory. A similar formulation of non-smooth ordinary differential equations (ODEs) has been explored in the context of differential variational inequalities (cf. (24
59 [4).

---

At first glance, it may not be immediately clear why the algorithm is structured as in 13 The derivation of (13) was inspired by the connection between the FL method and SQP in the equality- constrained setting. Hence, for the reverse-engineered its principles to derive continuous-tin counterpart, leading to the formulation of the FL method in (13). To ensure coherent and intuitive presentation, we begin by establishing its relationship with the SQP algorithm.
case, we first analyzed SQP and then
Relationship with the SQP algorithm The corresponding forward Euler discretization of 13 is given by
I++1 T (Vf(x)) JM( Ac)
(14)
A arg min J (M( A (M(G)))(a))VV(()))
- Kh(x:)))
We now consider the following SQP type of optimization method
I(+1 arg min, (z ((= 21) + 2/ mx) [xx) (x x1)
s.t. h(x)) + (z=) - m2) 5
(15)
The following theorem states the equivalence between (4) and 15
Theorem 3. When K = 1,,, if (15) is feasible then the discretization feedback linearization (14)
is equivalent to the SQP algorithm 15
Similar to the proof of Theorem 2 the proof of Theorem also leverages strong duality and KKT conditions. The detailed proof is deferred to Appendix D
Convergence Result Theorem 3 demonstrates the relationship between the FL. algorithm 13 and [15] Since SQP algorithms are known to be capable of converging to a KKT point (57), intuitively similar convergence can be established for our FL algorithm (13) which is the main focus of the following part.
We define the index set I(x) == {i (x) 0}. Our results rely on the following assumptions:
Assumption 4. Given the initial state x00) at = O, the optimization problem in (3.2) admits bounded solution ||Aloo I L for all a E, where & is defined by E {z|0 < h;(x) S hz())))) Vi [(x())))
quite complicated, there are some simplified versions that serve sufficient condition of Assumption 4 For example, if we start with a feasible x00), then & =
Although Assumption
and hence Assumption is automatically satisfied. Additionally, note that Assumption is another sufficient condition of Assumption (see Lemma in Appendix G. Notably Assumption 4 similar constraint qualification (MFCQ) considered in literature [
the iaagassaraa-FFromovv
and hold and let the control gain matrix K be diagonal matrix, i.e. K = diag k; }==1, where h, Then the learning dynamics 13 satisfies the following properties
Theorem 4. Let Assumption 7
dh;(x(t)) di
(t)), and hence the dynamic will converge to the feasible set.
Define f(x) = f(x(t)) + L L, then {(x(t)) is non-increasing w.rt t. Here [h,(x)]+
max 0})
3. The following inequality holds
f (IVS(=()) (ale())(())1 EiEzon (AA(()))((((()))) di S <((x(0)) ((x(T))
4. (Asymptotic convergence and convergence rate) The above statements imply that
inf KKT (KT))) & max {v Amia oStST
(F(=(0)) Smin L E (x((())).
ming n. (f(x(0)) Smin + (I. 11++ EiEI(EOO) (i((())) ((((0))))
(+)
Further we have that KKT-gap asymptotically converges 10 zero, i.e. lim (XT-)
7

---

Statement 4 in Theorem implies that the algorithm can find an e-first-order KKT-point within time z.. Similar to Theorem the key step of the proof is to construct the merit function in Statement (detailed proof deferred to Appendix E..
Momentum Acceleration for Constrained Optimization
we introduced the FL-proximal and FL-Newton algorithms. Generally, FL-Newton achieves faster convergence than FL-proximal due to its use of second-ords -oder information. However, scenarios where Hessian information unavailable, FL-proximal must be used instead, raising the question of whether its convergence can be accelerated. Given that momentum acceleration has been shown to improve convergence rates unconstrained optimization, a natural question arises: can version of the FL--prxiial algorithm, along with its corresponding discrete- time SQP formulation, achieve faster convergence? This section aims to address this question as part of Contribution 4
In Remark 3
Momentum acceleration is technique commonly used in optimization enhance convergence rates (cf. [63] [56] (20), see Appendix A for more detailed introduction about momentum acceleration). For unconstrained optimization. the discrete-timg momentum acceleration for gradient descent generally takes the form of
Wi 2, ((x; 21-1), I++1 w; f(w,)
(16)
Its corresponding continuous- analogue can be written as a second-orde ODE 63
2, = fz(z)
(17)
Inspired by the form of (16) and 17 for equality constrained optimization, we propose the follow- ing heuristic momentum acceleraatee momentum-acceleeatted discrete time SQP scheme
W1 24 B(x, 21-1). I++1 We nV f(w)) (ws)
A = - (J
(J,(w, (f(()) - , h((w)))
(18)
and continuous time FL scheme, which we name FL-momentum:
momentum Equality Constrained Optimization
i = (V/(z)+
=
(19)
(J, -
Kh(z))
Note that compared with FL-proximal 88. the difference in 18) is the addition of momentum
step W/ = It + 3(x, - It-1). Similarly we can propose the Fl-momentum scheme for inequality constraint case follows:
FL-momentum Inquality- Constrained Optimization
2, == (Vf(z)+ (x arg min Itt ( (J))))V(()-KKK))
(20)
The numerical simulation in Section 6 (Figure 2 suggests that momentum methods indeed accel- erate the convergence rate. We would also like to note that as far we know, the acceleration of SQP methods are generally achieved Newton or quasi-Newton methods, there's little work on exploring acceleration via momentum approaches, which makes our proposed momentum algorithm a novel contribution.
5.1 Convergence Analysis for FL-momentum: Nonconvex Case
In this section, we provide some convergence guarantees for the proposed algorithm. In particular, primarily focus on the convergence analysis for the continuous-time algorithm for equality con- strained optimization (19). It remains future work to establish the convergence for the discrete-tima algorithm (18) or the inequality eeequality-constrinne constrained algorithm (20).
We first define the following notation
X(z) == (V
(21)
8

---

Apart from Assumption 1 and we also make the following assumptions on f and h. Assumption 5. Both f(x), h(x) are three-times differentiable and the derivatives are bounded, thus, we know that there exist some constants L,,, L1, L2 such that
aa a(Ja(e) ((x)+ ( M(l)) S L2
[V2
oM(z) 8x
Li,
az
Assumption 6. We also assume that that there exists a constant H. such that [[(x)]] S H, Va, where (xx) = [h(x)" <=h((=)]
We are now ready to state our main result
Theorem 5. Assume that Assumption
7 and hhold. Let two positive constants a, 02 be such
that 02
4 Amax(K)
L2D
Li
01 2 0. We define the following merit function:
Awin
Amin (K)
(xx = =aaa((x) +
020
||(x)]| + arad(()) 1(x)) ||=||
2
+ (2))(()) + h(x) + (1))())) {(x) + a1 αx(x) dx
+ a1
h(x))
then for 2 (ai(Ly + 12) (M - H) + ((Amax(K)
1, we have that
a2
(xx(t), z(t)) non-increatee increascresing with respect to t. the following inequality holds
022min (K) |M(x(())1+ a1 Ja(((t)) (T((())) ((x(0),=(0)) (0( min ((z, 2)
8
3. We can bound the KKT-gap by
infostgr (KT)
[(r(0) Fiin
0(-L)
min AmAmi
and lime KKT (a)( 0, lime KKT
The detailed proof is provided in Appendix
Remark 5 (Limitation of the result). One limitation Theorem is that it establishes convergence but not acceleration over FL-proximal. However, when the constraint function h(x) is affine, the algorithm is equivalent to the momentumm-ccelerntt projected gradient method (see Appendix F.I. offering insight into its potential for accelerating optimization.
6 Numerical Verifications
For numerical validation, we consider logistic regression problem involving heterogeneous clients Many scenarios, such as federated learning and fair machine learning, require training common model in a distributed manner by utilizing data samples from diverse clients or distribu- tions. In practice. heterogeneity in local data distributions often results in uneven model performance across clients [48] 76). Since this outcome may be undesirable, reasonable objective in such set- tings to add constraints to ensure that the model's loss is comparable across all clients.
73
We formulate the above problem as a constrained optimization problem as follows: consider solving the logistic regression for clients. For each client {1,2,..., C), it is associated with its own
dataset D. {xx,, y:) b |Dol, where the label is y: 1-1, 1} and data feature is E Rd For each client c, its own logistic regression loss (0) is defined as
(e = D. EIED. log(1 + exp(-M A (Tz)))
where 0 is the parameter of the regression model. We further define the averaged regression loss R(()) as R(0)= = 1 fe)

---

As suggested in [73, 42), heterogeneity challenges can be addressed by introducing a proximity constraint that links the performance each individual client, Rg the average loss across all clients, R. This approach naturally formulates constrained learning problem [
ming R(0), 8.1. Rc( R(O) S 0, Vc (1,2..... C
(22)
where is small, fixed positive scalar.
Function Value over Iterations
KKT Gap over iterations
3.0
FL-proximal
10
FL-proximal
FL-momentum
10
FLmomartum
FL-Neutann
FL-Neutan
10
KKT-gap
2.0
10
10
10a
10
240
300
Iteration
Iteration
Figure 2: Result for Heterogeneous Logistic Regression
We solve the constrained optimization problem 22 FL-momentum algorithm Here data Yi is randomly generated from a Bernoulli distribution and x; is generated from a Gaussian distribution whose mean differs among different agents. The results of the numerical simulation presented in Figure 2 All algorithms converge to a first-order KKT point. FL-Newton converges fastest owing to its use of second-order (Hessian) information, while among first-order methods, FL- momentum outperforms FL- FL-proximal in convergence speed. More comprehensive numerical results are provided in Appendix B
by running the FL-proxxial, FL-Newton, and the number of clients C' 5 and [Del 200, the
7 Conclusion
In this paper, we study the theoretical foundations for solving constrained optimization problems from a control perspective via feedback linearization (FL). We established global convergence rates for squualty--onstrainee optimization, highlighted the relationship between FL and Sequential Quadratic Programming (SQP), and extended FL methods to handle inequality constraints. Fur- thermore, introduced momentum nomentuu-aacceleaatte FL algorithm, which empirically demonstrated faster convergence and provided rigorous convergence guarantees for its continuous-tim dynamics. Future directions include exploring the potential extension zeroth-order optimization settings and relaxing assumptions in the theoretical analysis.
Several limitations of our framework remain. First, our analysis ensures convergence of the best- found iterate to a first-order KKT point, but does not distinguish between local optima, global op- tima, and saddle points. Extending the framework to guarantee convergence to local optima via saddle-point escape mechanisms is an important direction for future work. Second, the applicabil- ity of feedback lincarization depends on structural assumptions such the Linear Independence Constraint Qualification (LICQ). Third, while FL-momentum empirically accelerates convergence, our current analysis yields the same O(1/VT) rate as its non-accelerated counterpart. Relaxing these assumptions and strengthening convergence guarantees remain promising directions for future research.
Acknowledgement
This work was supported in part by NSF ASCENT under Grant 2328241 and in part by the NSF AI Institute under Grant 21 1005.. Runyu Zhang was supported by the MIT Postdoctoral Fellowship Program for Engineering Excellence.
'It is not hard to verify that with probability one Equation 22 satisfies Assumption 2 and compact region, the function satisfies Assumption I (Lipschitz property) well
Also within
10

---

References
[1] Ahmed Allibhoy and Jorge Cortés. Control-barrier- cera---barrrrffucctionnn design of gradient flows for constrained nonlinear programming. IEEE Transactions Automatic Control, 2024.
[2] Javier Alonso-Mora, Stuart Baker, and Daniela Rus. Multi-robot formation control and object transport in dynamic environments via constrained optimization. The International Journal of Robotics Research, 2017.
[3] Wangpeng An, Haoqian Wang, Qingyun Sun, Jun Xu, Qionghai Dai, and Lei Zhang. A PID Controller Approach for Stochastic Optimization of Deep Networks. In 2018 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, Salt Lake City, UT, June 2018. IEEE.
[4] Neculai Andrei. Gradient Flow Algorithm for Unconstrained Optimization.
[5] David Applegate, Mateo Díaz, Haihao Lu, and Miles Lubin. Infeasibility detection with primal-dual hybrid gradient for large-scale linear programming. lion, 2024.
SIAM Journal on Optimiza-
[6] Sanjeev Arora, Noah Golowich, Nadav Cohen, and Wei Hu. A CONVERGENCE ANALYSIS OF GRADIENT DESCENT FOR DEEP LINEAR NEURAL NETWORKS. 2019.
[7] Uri M Ascher and Linda R Petzold. Computer methods for ordinary differential equations and differentiallalgebraiic equations. SIAM, 1998.
[8] Kendall Atkinson. An introduction numerical analysis. John wiley & sons, 1991.
[9] Amir Beck and Marc Teboulle. A fast iterative inverse problems. SIAM journal imaging sciences, 2009.
algorithm for linear
[10] Ernesto Birgin and José Mario Martínez. Practical augmented Lagrangian methods for constrained optimization. SIAM, 2014.
[11] Paul Boggs and Jon Tolle. Sequential Quadratic Programming. Acta Numerica, January 1995.
[12] Joseph-Frédéric Bonnans, Jean Charles Gilbert, Claude Lemaréchal, and Claudia Sagas- tizábal. Numerical optimization: theoretical and practical aspects. Springer Science & Busi- ness Media, 2006.
[13] Stephen Boyd. Convex optimization. Cambridge UP, 2004.
[14] M. Kanat Camlibel, Jong-Shi Pang, and Jinglai Shen. Lyapunov Stability of Complementarity and Extended Systems. SIAM Journal on Optimization, January 2007.
[15] Vito Cerone, Sophie M. Fosson, Simone Pirrera, and Diego Regruto. A new framework for constrained optimization via feedback control of lagrange multipliers. IEEE Transactions on Automatic Control, 2025.
[16] Xin Chen, Jorge I. Poveda, and Na Li. Model-Free Feedback Constrained Optimization Via Projected Primal-Dual Zeroth-Order Dynamics, June 2022.
[17] Yiting Chen, Liliaokeawawa Cothren, Jorge Cortés, and Emiliano Dall' Anese. Online regu- lation of dynamical systems to solutions constrained optimization problems. IEEE Control Systems Letters, 2023.
[18] Frank Curtis, Hao Jiang, and Daniel P. Robinson. An adaptive augmented Lagrangian method for large-scale constrained optimization. Mathematical Programming, August 2015.
[19] Frank E. Curtis, Tim Mitchell, and Michael L.. Overton. A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles. Optimization Methods and Software, January 2017.
[20] Alexandre d' Aspremont, Damien Scieur, and Adrien Taylor, Acceleration Methods, Founda- tions and Trends@ in Optimization, 2021.

---

[21] II Dikin. Iterative solution of problems of linear and quadratic programming. In Doklady Akademii Nauk. Russian Academy of Sciences, 1967.
[22] Dongsheng Ding and Mihailo R Jovanović. Global exponential stability of primal-dual gradi- ent flow dynamics based on the proximal augmented lagrangian. In 2019 American Control Conference (ACC). IEEE, 2019.
[23] Hermann W Dommel and William F'Tineyy Optimal power flow solutions. IEEE Transactions on power apparatus and systems, 1968.
[24] Paul Dupuis and Anna Nagurney. Dynamical systems and variational inequalities. Annals of Operations Research, 1993.
[25] Florian Dörfler, Zhiyu He, Giuseppe Belgioioso, Saverio Bolognani, John Lygeros, and Michael Muehlebach. Toward a Systems Theory of Algorithms. IEEE Control Systems Letters, 2024.
[26] Omer Elkabetz and Nadav Cohen. Continuous vs. Discrete Optimization Deep Neural Net- works, December 2021.
[27] Eremin. The penalty method convex programming. Cybernetics, 1967.
[28] Mahyar Fazlyab, Manfred Morari, and George J. Pappas. Safety Verification and Robustness Analysis of Neural Networks via Quadratic Constraints and Semidefinite Programming. IEEE Transactions on Automatic Control, January 2022.
[29] Guilherme França, Daniel P Robinson, and Rene Vidal. A dynamical systems perspective on nonsmooth constrained optimization. arXiv preprint arXiv: 1808. 04048, 2018.
[30] Daniel Garcia and Fengqi You. Supply chain design and optimization: Challenges and op- portunities. Computers & Chemical Engineering, 2015.
[31] Kunal Garg and Dimitra Panagou. Fixed-Time Stable Gradient Flows: Applications to Continuous-Tine Optimization. IEEE Transactions on Automatic Control, May 2021.
[32] Philip E. Gill and Daniel P. Robinson. A Globally Convergent Stabilized SQP Method. SIAM Journal on Optimization, January 2013.
[33] Philip E. Gill, Vyacheslav Kungurtsev, and Daniel P. Robinson. A stabilized SQP method: global convergence. IMA Journal of Numerical Analysis, January 2017.
[34] Nicholas IM Gould, Dominique Orban, and Philippe L Toint. Galahad, a library of thread-safe fortran 90 packages for large-scale nonlinear optimization. ACM Transactions on Mathemati- cal Software (TOMS), 2003.
[35] Revati Gunjal, Syed Shadab Nayyer, Sushama Wagh, and Navdeep Singh. Unified Control Framework: A Novel Perspective on Constrained Optimization, Optimization-based Control, and Parameter Estimation, July 2024.
[36] William W. Hager. Stabilized Sequential Quadratic Programming. In Jong-Shi Pang, editor, Computational Optimization. Springer US, Boston, MA, 1999.
[37] Shih-Ping Han. A globally convergent method for nonlinear programming. Journal of opti- mization theory and applications, 1977.
[38] Adrian Hauswirth, Saverio Bolognani, Gabriela Hug, and Florian Dörfler. Timescale Separa- tion in Autonomous Optimization. IEEE Transactions Automatic Control, February 2021.
[39] Adrian Hauswirth, Zhiyu Hc, Saverio Bolognani, Gabricla Hug, and Florian Dörfler. Opti- mization Algorithms as Robust Feedback Controllers, January 2024.
[40] Zhiyu He, Saverio Bolognani, Jianping He, Florian Dörfler, and Xinping Guan. Model-Free Nonlinear Feedback Optimization. IEEE Transactions on Automatic Control, July 2024.
12

---

[41] Michael A Henson and Dale E Seborg. Feedback linearizing control. In Nonlinear process control. Prentice-Hal Upper Saddle River, NJ, USA, 1997.
[42] Ignacio Hounie, Alejandro Ribeiro, and Luiz FO Chamon. Resilient constrained learning. Advances in Neural Information Processing Systems, 2024.
[43] Bin Hu and Laurent Lessard. Control Interpretations for First-Order Optimization Methods, March 2017.
[44] Bin Hu, Peter Seiler, and Anders Rantzer. A Unified Analysis of Stochastic Optimization Methods Using Jump System Theory and Quadratic Constraints. In Proceedings of the 2017 Conference Learning Theory. PMLR, June 2017.
[45] Alberto Isidori. Nonlinear control systems: an introduction. Springer, 1985.
[46] T Kose, Solutions of saddle value problems by differential equations. Econometrica, Journal of the Econometric Society, 1956.
[47] Laurent Lessard, Benjamin Recht, and Andrew Packard. Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints. SIAM Journal on Optimization, January 2016.
[48] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2020.
[49] Zhe Liu, Fahim Forouzanfar, and Yu Zhao. Comparison of SQP and AL algorithms for deter- ministic constrained production optimization of hydrocarbon reservoirs. Journal of Petroleum Science and Engineering, 2018.
[50] Steven H Low. Convex relaxation of optimal power flow part i: Formulations and equiva- lence. IEEE Transactions on Control of Network Systems, 2014.
[51] Michael Muehlebach and Michael Jordan. A dynamical systems perspective on nesterov ac- celeration. In International Conference on Machine Learming. PMLR, 2019.
[52] Michael Muchlebach and Michael I. Jordan. On constraints first-order optimization: view from non-smooth dynamical systems. Journal of Machine Learning Research, 2022.
[53] Michael Muchlebach and Michael I. Jordan. On Constraints in First-Order Optimization: A View from Non-Smooth Dynamical Systems, November 2022.
[54] Michael Muchlebach and Michael Jordan. Accelerated first-order optimization under nonlin- ear constraints. arXiv preprint arXiv:2302.00316, 2023.
[55] Michael Muchlebach and Michael I. Jordan. Accelerated First-Orde Optimization under Non- linear Constraints, January 2024.
[56] Y. Nesterov. A method for solving the convex programming problem with convergence rate o(1/k2), 1983.
[57] Jorge Nocedal and Stephen J. Wright. Numerical optimization. Springer series in operation research and financial engineering. Springer, New York, NY, second edition edition, 2006.
[58] Figen Oztoprak, Richard Byrd, and Jorge Nocedal. Constrained optimization in the presence of noise. SIAM Journal on Optimization, 2023.
[59] Jong-Shi Pang and David E. Stewart. Differential variational inequalities. Mathematical Pro- gramming, June 2008
[60] Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends in Optimiza- tion, 2014.
[61] David W Peterson. A review constraint qualifications in finteeddimensional spaces. Siam Review, 1973.
13

---

[62] Boris Polyak and Pavel Shcherbakov. Lyapunov Functions: An Optimization Theory Perspec- tive. PanC-apprsOOnieee, July 2017.
[63] B.T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, January 1964.
[64] Florian A Potra and Stephen J Wright. Interior-point methods. Journal of computational and applied mathematics, 2000.
[65] Michael JD Powell. Algorithms for nonlinear constraints that use lagrangian functions. Math- ematical programming, 1978.
[66] Guannan Qu and Na Li. On the exponential stability of primal-dual gradient dynamics. IEEE Control Systems Letters, 2018.
[67] Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via Stochas- tic Gradient Langevin Dynamics: nonasymptotic analysis. In Proceedings of the 2017 Con- ference on Learning Theory. PMLR, June 2017.
[68] Ivan Dario Jimenez Rodriguez, Aaron Ames. and Yisong Yue. LyaNet: A Lyapunov Frame- work for Training Neural ODEs. In Proceedings of the 39th International Conference on Machine Learning. PMLR, June 2022.
[69] Halsey Royden and Patrick Michael Fitzpatrick. Real analysis. China Machine Press, 2010.
[70] Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
[71] Katya Scheinberg, Donald Goldfarb, and Xi Bai. Fast First-Order Methods for Composite Convex Optimization with Backtracking. Foundations of Computational Mathematics, June 2014.
[72] Schropp and I. Singer. A dynamical systems approach to constrained minimization. Numer- ical Functional Analysis and Optimization, January 2000.
[73] Zebang Shen, Juan Cervino, Hamed Hassani, and Alejandro Ribeiro. AN AGNOSTIC AP- PROACH TO FEDERATED LEARNING WITH CLASS IMBALANCE. 2022.
[74] Elias M Stein. Singular integrals and differentiability properties of functions. Princeton uni- versity press, 1970.
[75] Weijie Su, Stephen Boyd, and Emmanuel J Candes. A differential equation for modeling nesterov's accelerated gradient method: Theory and insights. Journal of Machine Learning Research, 2016.
[76] Hao Wang, Zakhary Kaplan, Di Niu, and Baochun Li. Optimizing federated learning on non- iid data with reinforcement learning. In IEEE INFOCOM 2020-IEEE conference on computer communications. IEEE, 2020.
[77] Jing Wang and Nicola Elia. A control perspective for centralized and distributed convex op- timization. 2011 50th IEEE Conference on Decision and Control and European Control Conference, 2011.
[78] S. Wang. X.Q. Yang, and K.L. Teo. A Unified Gradient Flow Approach to Constrained Non- linear Optimization Problems. Computational Optimization and Applications, April 2003.
[79] Ashia C. Wilson, Benjamin Recht, and Michael I. Jordan. A Lyapunov Analysis of Momentum Methods in Optimization, March 2018
[80] Robert Wilson. A simplicial algorithm for concave programming. Ph. D. Dissertation, Graduate School of Bussiness Administration. 1963.
[81] Stephen Wright. Superlinear Convergence of a Stabilized SQP Method to a Degenerate Solution. 1998.
14

---

[82]
Zeke Xie, Issei Sato, and Masashi Sugiyama. A Diffusion Theory For Deep Learning Dynam- ies: Stochastic Gradient Descent Exponentially Favors Flat Minima, January 2021.
[83] Hiroshi Yamashita. A differential equation approach to nonlinear programming. Mathematical Programming, December 1980.
[84] Willard Zangwill. Non-linear programming via penalty functions. Management science, 1967.
[85] Xianlin Zeng, Jinlong Lei, and Jie Chen. Dynamical Primal-Dual Nesterov Accelerated Method and Its Application Network Optimization. IEEE Transactions on Automatic Con- trol, March 2023.
[86] Dongdong Zhang and Zhongwen Chen. Superlinear convergence of stabilized SQP-type method for nonlinear semidefinite programming. Journal of Applied Mathematics and Com- puting, October 2024.
[87] Jiawei Zhang and Zhi-Quan Luo. A proximal alternating direction method of multiplier for linearly constrained nonconvex minimization. SIAM Journal on Optimization, 2020.
[88] Changhong Zhao, Ufuk Topcu, Na Li, Li, and Steven Low. Power system dynamics as primal-dual algorithm for optimal load control. arXiv preprini arXiv:
2013.
[89] Limei Zhou, Yue Wu, Liwei Zhang, and Guang Zhang. Convergence analysis of differential equation approach for solving nonlinear programming problems. Applied Mathematics and Computation, January 2007.
15

---

A Related Literature
Control and Dynamical System Perspective for Optimization As mentioned in the introduc- tion, many papers analyze the performance continuous time optimization algorithms in different settings. For unconstrained cases, various continuous algorithms are studied such as gradient flow
26 6, 70, 31 4, stochastic gradient 66 82], Nesterov acceleration 77 79 51, and momentum
acceleration 63 62]. Notably, therreec tools, such as integral quadratic constraints (IQC)
[7 [4 Proportional Integral Derivative (PID) control [43], are used to bring interesting insights to the convergence property of optimization algorithms. However, for constrained cases, less work is known. One direction focuses on the dynamics of PDGD algorithm, which primarily focuses on the However, this approach is primarily limited to convex optimization. For nonconvex optimization, prior research has explored modifications of the gradient flow to accommodate equality constraints (see, e.g., [83 72 78)) as well as inequality constraints [1. [52 [54] The algorithms proposed in these works share similari- ties with the feedback linearization approach considered in [15] and this paper. Notably, [72 also highlighted connections to sequential quadratic programming (SQP). This paper advances the exist- ing literature in several key ways: i) while prior analyses [83 72 89 primarily focus on equality constraints, our results extend the algorithm to handle inequality constraints; ii) although the algo- rithms in these works resemble ours, we introduce a novel perspective based on feedback lineariza- tion; iii) previous convergence results are largely asymptotic, whereas we establish non-asymptotic convergence rates in terms of the KKT gap; and iv) We establish acceleration results through the momentum method, further contributing to the theoretical understanding of the algorithm. Notably, (54] also considers a momentum-basse bbsed acceleration for constrained optimization, however, both the continuous-time dynamics and the discretization approach are different, and thus resulting in distinct algorithms.
convex setting and establishing global convergence 46 22
83 15]
Another related line of research explores real-time optimization from a systems perspective [25] 38 39 40, [8]. These works primarily focus on scenarios where the optimizer (controller) interacts with a physical dynamical system (plant) and employ control-theoretic approaches, such as time- scale separation and the small-gain theorem, to design and analyze the performance of real-time interactive optimization schemes. In contrast, our setting differs in that the plant itself corresponds to the gradient dynamics of the Lagrangian, while the controller is represented by the dual variable A.
A series of works have also explored the properties learning and optimization in neural networks using tools from control and dynamical systems, including the proportional-interaa -eriivatve (PID) approach [3). quadratic constraints (28). and Lyapunov stability [68]. However, as these studies focus on different problem settings and techniques. they fall outside the scope of this paper.
Sequential Quadratic Programming (SQP) Originally proposed in 88 37 65 SQP serves as a powerful optimization algorithm for nonconvex constrained optimization. Generally speaking, the algorithm talks the following form (ef. (1])
1 It+1 arg min (f(zz) xι) + (x 10)x))) [.) 21)
x
S.I. M(xx) + (xx))(x -xx,,, vx) (5)
For faster convergence, matrix (xx)) is generally set as the Hessian matrix of the objective func- tion f. However, in the setting where only first order information is available, [99 has employed Goldfah
(BFGS) quasi-Newton Hessian approximations. In our FL- proximal algorithm, the matrix H (xx) as the scalar matrix (*,,) = I which also common choice when second-ordes information is unavailable [58]. We would also like to note that as far as we know, the acceleration of SQP methods are generally achieved via Newton or quwsi-Newton methods, there's s little work on exploring acceleration via momentum approaches, which makes our proposed momentum algorithm (18) (19 a novel contribution.
Note that our convergence results rely on the linear independent constraint qualification (LICQ) assumption (Assumption 3). In literature, there are works that seek to replace this restrictive as- sumption with milder conditions while still maintaining convergence properties, which generally known as the stabilized SQP (cf. [32 33 36 81 (6)). It is an interesting open question whether we can borrow insights from stabilized SQP to modify our FI. algorithms such that the LICQ assump- tion can be removed.
16

---

Other Constrained Optimization Algorithms Apart from SQP methods. there are also other op- timization algorithms for solving constrained optimization. Interior point method (IPM, cf. (2T664 is one of the most deployed algorithm for nonconvex constrained optimization. The algorithm in- volves navigating through the interior of the feasible region to reach an optimal solution. A common approach within IPMs involves the use of barrier functions to prevent iterates from approaching the boundary of the feasible region. By incorporating these barrier terms into the objective function, the algorithm ensures that each iteration remains within the feasible region's 's interior. Newton's 's method is then applied to solve the modified optimization problem, iteratively updating the solution estimate until convergence criteria are met. This methodology allows IPMs to efficiently handle large-scar optimization problems with numerous constraints. While both IPMs and SQP methods can ad- dress similar optimization challenges, they differ fundamentally in their approaches: IPMs focus maintaining iterates within the interior of the feasible region using barrier functions, whereas SQP methods iteratively solve quadratic approximations, often employing active-set strategies to handle constraints.
Augmented Lagrangian Methods (ALMs) (cf. nO (8) also known as the method of multipliers, another popular approach to solve large-scale constrained optimization problems by transform- ing them into a series of unconstrained problems. This transformation is achieved by augmenting the standard Lagrangian function with a penalty term that penalizes constraint violations, thereby facilitating convergence to the optimal solution.
Further, in convex optimization, proximal gradient methods (cf. class of optimization problems that take the form of
13
60
can be applied for wide
min f (ax) + g(x) x
where f(x) is a smooth convex function and g(x) is a convex but potentially nonsmooth, non- differentiable function. The proximal gradient method iteratively updates the solution estimate by:
T++1 prox ng
(xe
vf(x))),
where 7) is the stepsize and prox, denotes the proximal operator associated with g defined as:
prox og (x) = arg min ((()) + 20x 11 - x!|?
y
In particular, for convex constrained optimization, we can set g(x) as the indicator function:
{
O if (S)
g(z)
+o0 Otherwise
and in this setting, the proximal operator is equivalent to projection onto the feasible set h(()) (S) O. One limitation of the proximal gradient type of algorithms that hard to generalize to the nonconvex setting and that the projection onto the feasible set might be hard to compute the proximal operator for complex h(x)).
Accelcration Algorithms First-order acceleration methods, such as momentum and Nesterov Ac- celerated Gradient (NAG), are key advancements in gradient-based optimization. Momentum, in- troduced by [63]. enhances gradient descent by incorporating an extrapolation step that accelerates convergence. Specifically, the update rule w; (1+ 3)æ, Br,-1 extrapolates the current position Ze by considering the previous position æt-1, effectively predicting future point in the parameter space. This anticipatory step allows the optimization process to gain speed, particularly in regions where the objective function exhibits gentle slopes or low curvature. Nesterov's Accelerated Gra- dient (56] refines momentum by proposing a more refined extrapolation parameter 88 that possibly varies with iterations, enabling more precise adjustments and achieving faster convergence rates.
Note that momentum or Nesterov acceleration can also be applied to proximal gradient methods (cf. 717 i.e.
V/2 .z + Be(xx 21-1)
Jz+1 prox (Bi 17V fv))).
By integrating a momentum term, is able to accelerate convergence while maintaining the benefits of the proximal operator in handling nonsmooth g(z). We would also like to note that when g(x)) is chosen as the indicator function of h(x))= = Ar + gradient method recovers our momentum SQP scheme (88)
then the momentum accelerated proximal
17

---

B
More Numerical Simulations
B.1 Comparison with other first-order methods
Here we also compare the performance of our algorithm with existing first order methods such as primal dual gradient descent (PDGD) and Augmented Lagrangian method (ALM), as show in the figure below:
Function Value over Iterations
KKT Gap Iterations
Flpproximal
FLpprocimall
FL-momentum
FL-momertum
2.5
FLNoetton
FL-Novtan
PDGD
KKT-gap
10-1
1.5
10
10-
1.0
10-45
120
100
240
300
130
240
300
Iteration
Iteration
Figure 3: Running different constrained optimization algorithm for Problem
From the above figure, we note that our FL-proximal method performs comparably with existing methods such as PDGD and ALM. And expected, FL-momentum converges faster it lever- ages momentum acceleration and FL-Newton performs the best as it leverages the second order information.
B.2 Generalization of the FL algorithm using PI Control
As pointed out by Remark one drawback of the Fl--proximal is that it requires calculation of the inverse matrix of Ju Ja which might be inefficient when the number of constraints becomes larger. In reality, there are ways solve the inverse of Jn (z) approximately using heuristic methods or conjugate gradient.
Assume that instead of calculating (=)))) we approximate it with some matrix F(x) so (Ja()))a T)-1, For example, in the setting where Jn (x) is diagonal dominant, we can heuristically set F(::) be diagonal matrix where the diagonal element of F(x) is the inverse of the corresponding entry of (x) i.e.,
F(x) (daa{()(=))
(23)
Thus, is of great importance to study the algorithm's 's robustness towards the approximation error of F(x). We going in this section that it might be beneficial to consider PI controller for feedback linearization.
We begin by introducing the Approximated FL algorithm for equality constrained optimization as follows:
Approximated FL
à = + À = (1) -K((a)))))))))))))))
Note that the only difference of Approximated FI. comparing with Fl.-proximal that we replace (=)) with its approximation F(x).
As mentioned in Remark 2 apart from considering the feedback linearization with a proportional
controller, i.c. ÿ = - K y. we can also consider a more general algorithm variant where we also include the integral term and thus formulate the dynamics as
- KKpy - K; So ys)ds
18

---

Function Value iterations
KKT Gap Iterations
10P
2000
KKT-gap
10
Approsimated Approsimated Fi-P
2000
10-4
500
Approximated FL
10
Approsimated FL-21
0
1000
2000
S000
4000
$000
1000
2000
3000
4000
5000
Iteration
Iteration
Figure Comparison of Approximated FL Approximated FL-PI algorithm
This gives rises to the (Approximated) FL-PI algorithm as follows:
Approximated FL-PI
(Vf(z)+ Ax)
Fx) J=(T))(()V((())) K2n(x) + K, f" M(((s))ss)
To test the robustness the algorithm towards approximation error, we run both Approximated FL and Approximated FL-PI algorithm on a quadratic programming problem
min 2
Qx + cTx
2"
s.t.
Ar +
where m = 10, = 20, and A, c.b.c,Q1/ are all random matrices or vectors whose entries are generated following an i.i.d random Gaussian distribution.
demonstrates the algorithm performance for both Approximated FL and Approximated FL-- PI. Interestingly, in this setting, Approximated FL. fails to converge the optimal solution, whereas Approximated FL-PI is able to find the optimal solution. This indicates that is beneficial to include an additional integral controller term into the optimization algorithm.
B.3
More numerical examples: Optimal Power Flow
The Alternating Current Optimal Power Flow (AC OPF) problem a fundamental optimization task in power systems. Its goal is determine the most efficient operating conditions while satisfying system constraints. This involves optimizing the generation and distribution of electrical power minimize costs, losses, other objectives while ensuring that physical laws (such power flow equations) and operational limits are respected, thus it can be summarized as the following constrained optimization problem:
min, f(x), s.t. Ney(2)) 0, himey (x) 0,
(24)
where the objective function f(x represents the power generation cost and the equality constraints (xr( generally represents the physical law the power system, i.e., the power flow equations and Itineq includes operational limits in terms of voltage, power generation, transmission capacities etc. The optimization variable x generally consists voltage angles and magnitudes at each bus, and the real and reactive power injections at each generator (see (50) for detailed introduction AC OPF).
We solve the AC OPF problem 24 by running the FL-proximal algorithm, FL-Newton algorithm, and FL-momentum algorithm. Figure presents the numerical results for solving AC OPF on the IEEE-39 and IEEE-118 bus systems, respectively. In both cases, FL-Newton demonstrates the fastest convergence, which is expected given that it leverages second-order information (i.e., Hessian). Comparing Fl.-prxximal and Fl-momentum, both of which rely solely first-order information, Figure [ indicates that Fl-momentum accelerates the learning process and achieves faster convergence than FL-proximal for the IEEE-39 bus system. However, in the IEEE- 1IE--18 18 bus
19

---

KKT Gap over Iterations
KKT Gap Iterations
10
Fi-proximal
R-proximal
10
.mmmmntum
10P
n-momentum
Ei-Nevton
10-4
KKT-gap
10
KKT-gap
30
10
10
10'9
Iteration
Iteration
Figure 5: Result for AC OPF on
IEBE-39 bus (left) and 1EEE-118 bus (right) bus system
system, FL-proximal and FL-momentum exhibit similar convergence speeds, with their learning curves nearly overlapping. We hypothesize that this problem is ill-conditioned, limiting the effec- tiveness of momentum in accelerating the algorithm.
C Proof of Theorem I
Proof of Theorem Statement is obvious from the derivation Section 2 ÿ; -k thus (t( e (0) always keeps the same sign, since (t) = hę(x(f)), we have completed the proof for statement 1.
because we have that
For statement 2, we first derive the time derivative for f(x). For notational simplicity we define
P(x) T(x) (1)) (s)(=))()) -1 (x)T(xx) Note that P(x) 0 and that
(()))())))P(() - P(x).
f(x) Vf(x)
( " (MNO)T -K(())
-VV(() T())) (INGTTGT)) Kh(z)
S 1V (AATTTTTAAAAATTTTAATAAATAAAAATTT 11x0 Σ k:/h(())]
< --V((x) P(t)V\()))) + Amax (MD)²
Σ
ke (x))
Amiin
1=1
Further, from statement we have that
d(h((x)] kx)h(x))),
thus we have
d (f(x)
+
Amax M Dm 1b,(2))) Aratn
dt
Amax (MD)² Σ ki]hg(z)]
Amax (MD)² Σ ka)h((a))
< ((F()) +
Amin
Amia
(25)
(z)))())) is non-increasing, which completes the proof. ie.. ((x(t)) f(x(t)) + (MD)² Σ.
Note that ((x(t)) always differentiable with respect to é because based on Statement I, x(() will always stay in &, (or Eg) if the initialization z00) E &, (or Ef).
Now we prove statement Since f is lower bounded and that h(x) - asymptotically, we arrive at the conclusion that the trajectory of (x((t) is bounded. From (25 we have that
dé(x(t)) < -Vf((((t) P((()))))))) dt
20

---

(x((T) ((x(0)) - -- ff- ==
)P 1t
Further, it is not hard to verify that
P(z(t))V((((t) T(=((t) T(())) (I(=))(()((()))" " (
T(((() (VS((())) + Ja(Z()) X()),
so we have
(( [Vf(z(t)) + (M((()) X(1) -
[Vf(((() + (a((()) X s ((x(0)) {(x(T)).
Amin
We also have that
[|A(t) - =1 II (IITT)))T( Kh(x(t))l 1 D(|[K][[b(=)]] + as
Amin
which completes the proof of statement 3.
We now prove statement 4. From statement 3 we have that
2
inf [VVf(x(t)) + (((((())TX(())) x(t) &
Amin T (x((0) - [(x(TT)
T SIST
2 (xx(0)) -
Smin
Amax M²D2
In((((0))
T
Amin
Amin
f(x(0)) fmin
Amax MMaxxMDDD M2D2
[V/(((0) J(((() S
Σ [b((x(0)1)
SIST
Amin
Amin
2
Further, from statement
T [b,(x(t)) & h₂(x(0))e- - for all t 2
2
thus
inf (KT-g))(((t),X(t) =
max V + J. (z()))
(T(I)) max 1sism
{l]((((t))l<>)))
T SIST
SIST
2
f(x(0)) -
Smin
Amax M2D2
[,,(x(0)))
(M(o(0))e
S max
max Isism
Amin
Amin 2
Additionally, it is not hard to verify from statement 3 and statement that
lim [VS(((()) (n(())) X(1)]I 0, lim ||(x(())]lo> = 0, lim [/(()) - A((II too
1-0
+00000
thus
lim KKT (KT-gap(z(f), ((t)) = 0, lim KKT (KT-gap(z(t)), = 0. ++0
+
C.1 Global Convergence for Strongly Convex Settings
Under the assumption that the optimization problem T is strongly convex, we are able to establish the result that Fl--prxxilll converges to the global optimal solution with linear rate. We make the following assumption.
21

---

Assumption 7 (Strong Convexity). The objective function f(x) is strongly convex, i.e. for any y,y R",
f(x) - 1(0) 2 VS(u) (x v) + ~
2 yll².
The constraint function h(x) in takes the form of h(x) Ar + b where A E Rmxx b R", and full row rank.
Theorem 6. Under Assumption
we have that running FL-proximal (6) with T(x) = I) will give
f(z - f(x)) S E 24x 2n (f(x(())) - f(x")),
where x² is the global optimal solution of
Proof. We denote PA := AT (AAA A, P 1 PA. Then the key argument is that according
to strong convexity we have:
f(z) - f(x'){(++))))))))())) & Vf(x) (a x²) 1 15--2 x¹11
I (A(T r²) 111P
Ps (0 x*)
V(OPAVV) + VS(x) -x)).
- 2p
Further,
df
V5(x)) (V](=)+
dt
(Ar+ Vf(x) (AV (Ax + b)))
(Ja(x)
-V[(x) - AT AAA +b)
-V((z) - (AA")" A (Ax -Ar)
and thus we have
d(f(((t)) - f(x))) S - -
dt
which completes the proof
D Proof of Theorem 2 and 3
Proof of Theorem
2
We write out the lagrangian for the SQP problem 9
A(x, = fz()) (z =x) + (x- 2x- (
+ X (h(æz) + (xx (x)))
2n
Since the optimization problem (9) is convex and satisfies the relaxed Slater condition, the KKT
condition necessary and sufficient for global optimality. The KKT condition implies that
(xx)) " = x -T(x:))(V[())) + (n(x)) Since h(x))) + (z - 2²) = O. we have
0,L(x, A) 0
2=
M(x))) + Jn(xz) 21) = - (14=)))) Jn::)) 0 h(xx)
-== (J) _n(x))).
Thus we have that
(aa (f) -
It-1 = =I - (I)
This is exatcly 8 with K = 11'
22

---

Proof of Theorem
We write out the Lagrangian for the convex optimization problem (5
L(z, - Vxx))) (a 21) - (x Tx)) (x-
2n +)(h(z,) + (xx) - -x))
Since 13) feasible, it satisfies the relaxed Slater condition, thus we have that strong duality holds for 15 Also note that the optimization problem in 14
X = arg min
X20
2
A (4
is the dual problem of (15) (see Lemma 2 in Appendix G Thus the solution of X" in 14 serves as the Lagrangian dual variable of Then, we have that the is an optimal solution if and only if
x² arg min L(x, = (, =0
(s(m)) - Txx)) (x* 2.) (x)))) =x=
x' F2 - fxx) + (xx) (²),
thus we have that both (14 and [5] follow the update:
T++1 Iz = -): J x*),
which completes the proof.
E
Proof of Theorem
Before proving Theorem we first introduce the following lemma which plays important role in proof.
Lemma 1. Solving À in equivalent to solving the following equations:
Jn A
Ky 8,
A
(26)
2 0, 0
Proof. The proof is simply writing out the KKT condition of the optimization problem.
We now ready to prove Theorem
Proof of Theorem We first prove statement 1.
ý Jn (x) = (MOT)))))
Ky (Lemma] 1
thus
= s -k
and hence we have proven statement 1.
Now we prove statement
f(x) = v = - -
=
V (ACTI(())((E + A
23

---

(() + +
--V/(<) + [ (IT((z) + (KU)z + >x (Ku)z
I - -VV +
[V](=++ (x)) ( + (Kylz + 1 E
k:yl
1E7
- - Vf(=)+
+ (x) AITT(() + (Ky)zz + Σ
kgyl
VEI
Here we abbreviate I(x) (1)4,(*) 0,15i5 m) as I. From the proof of Statement we have that for i E I
[1(=)= kky 3/ S k.
Thus, combining the inequalities together, we have that d (f(x)
L DieT 2 (VV ( + AZo (Ky)z= + L Σ VET.
kiyl >
k;
dt
IET
- Vf(=)+ (M(
( X (Ky)z- < (y/ 0 for I³).
(27)
Further, we have that the function on the righthand side is equivalent to
f(z)
+ h4(2) (t)+LLh(()) f(x) + Σ [1((z)))+ l(z).
SET
We would also like to note that given f(x) and h(x) are differentiable and Lipschitz, we have that {(x) is an absolute continuous function (cf. 669 and almost everywhere differentiable. Hence,
given that de(x(() < holds almost everywhere, we have that ((((t) is non-increasing with respect which completes the proof.
We now prove Statement 3. From 27 we have that
{(x) S -IV[[(x) + (x) A + >To(KK)))
- ( +
E (((())((((()) dt s - (EI(E)
which completes the proof. Note that here leverage the fact that ((x) is absolute continuous.
We now prove Statement From Statement 3 we have that
inf S1ST
[V](((() + (
Σ A:A:(0)s(((O))
< T (e(x(0)) e(x(()))
2
(EI(0))
2 $ T [(x(0)) - funi + L Σ hs(x(0))
vEI(x(())
Thus we have that there exists I < t" < such that
[Vf(((t)) + JK((()))))
Σ ((d(tt))((((("))) s T
2 [(x(0)) - Smin +
Σ na(a(0))
iEZ(=(()))
SEI(=(0))
Thus
+
2 Aminn
f(x(0)) funn + L Σ hc(z(0))
(EI(w(()
2 Σ (x((t))((((t')) < T
f(x(()) fman + Σ h₂(2(0))
(EI(Z())))
iEI(x(())
24

---

Since t* 2 2. we have that
S (,(x(0))e "2
ha(x(1"))
Σ X(())((((()) S Le
miny R
Σ hx(x(0)).
(ET(Z(())
(EZ(Z(0)
Thus have that
IA(t)) (((()))())
E +(1))
E (((()(((((1)))
SEI(a(("))
(EI(a(()))
f(x(0)) fmin Σ
bx(2(0)) + Le
ming K
Σ
h₂(a(0))
min; A T
(EI(a(())
iEI(x(0)
2 [(x(()) - mmi (I+1) Σ bx(x(0))
min; K;
(EI(a(0)
Thus we have that
inf KKT (KT-)aa(=xt)) S KKT-gap(=<(), 051ST
max
[Vff(x(t*) (x((tt)) X(t*),, A(t ((("))((()))))))))) m/xx()))))))))))((()))))
[(x(0)) - -min + L Ar(o(())
< max
A T
#EI(z(01)
Σ Inter(o) )
2 f(x(0)) - fmin + (L+ iET(a(0))
min; Ke T
Further, from statement 3 we get that
lim
[Vf(=(t))+ + MMCO)))))) 0,
++000
lim - Σ (A(tt)((<(tt) 1++
iEI(z)
From statement we get that
lim [h<()(t))]+
lim
Σ ((tt),(I((t)
++0
(EI(a)
Additionally,
lim ((())(((0) +++00
lim Σ (A((1))((())) + lim Σ ((()),12(()
min, K;
iEZ(x))
IEI(a)
thus we have that
lim XKT-gap(>(())A(() = 0, t
which completes the proof.
F Proof of Theorem S
We first define the following notations: denote P(x) = 1 -A Note that we have P(z) ?
25

---

Proof of Theorem 5
From 19
we have that
df
f((x)
(28)
dt
d' f(())
d(V (V)))) 2)
dt²
dt
V1/10)) VV) Ja(x) (I
= [=] --v(x))) +
(29)
Thus by combining
28 + 29
we have
d
dt (af(x) + Vi(x)
()(x))--V(0)) ((=)()) + +X(x) (K())
(30)
Similarly we have
d3)1(x)11 M(x) Mx)) Jn(r)2 dt
(31)
d [1())11 Jn dt
dt2
=n( Mo)
Σ
=4(x)) T (1))
= -xJ - -M)))) M()) { + (x)
+ Σ
21((x)
(211(z)=
1=1
(32)
By the definition of we know that
+ A) = KK(z),
thus we have
h(x) (n())= = JT(xx)) (x(x)= ah(x)) Ja(x)²= - Mx) (K(()) + ad(()) (2);(x)=
dt
(33)
Let (xx) [h(x)
(2), then Σ [h) H()) Thus by combining
31 (33 we have
d
h()))l
+ M(=) Jn(+)2) = 2 Jn()) T - h(x) Kh(x) + 2 H(x)
(34)
dt 2
Another set ODE we will use
dd(x) h(x) A(x) (n)))++((()))
ax(z)
(35)
dt
Ox
d dl
Oi(x) dr 2
Jn())= M(x)T
3 (Ja(z)
TT
2 A(x)
[n(x)
dx
i [1(x)) 07(x)
Ox(x)
+ T
+ h(x))))
(-az -
- - ()
Ox.
dr
(8x22) 8x
i [h(x)T(())
+
h(x))
T
aT(T)) Ja(x)= - KM(x))
8x.
26

---

ah(x)) Dx(x)
dr h(xx) --xx)= 0x(x))
Ox(x)
P(x)V/(<)) h(x)
dx Jn (Ja())((())))))) (h(x))
-K((()))))))))))
dr.
(36)
Thus by combining a x 35
(36
we get
d (a((x) + M(x)
[X(x)
(z(z) h(2)
Zx
① (J((T) A(x)
(b(a) axt
h((x))
T
2 - ((x) Kk((x)
8x
07(x) P(z)Vf(2) - h(x) O7(x)
- -
(n) ( (37)
dx
dx
Further we also have
d>11=11 di2
= (
(38)
Finally, we combine 01
30 + 01
x
02
38)
we get
d
a2a |+x)||++++ a ta22)))))) h(x))))(()))))>))))))()))))))))) +a)))))))()))))))))))))) ((x)) dr a1af())+
a7(x)
h(o) =+ara\(x)"1(
dl
((x.2)
a [()))))>(x)+ +
@x(z) T Ex
h(z)
T
01 f(x) + ta
+ [2Jn(x)) Jn(x) + aa Hz)((x)
2 - =a||2//2
dx
-ayh(z) 2Ma ((()) 8x
2X
d5
-2 (M) Kh(z)
- -VVFf() P(x))(() (ah(a)TT(((e
(39)
Note that for 02 2 4a
Amnna (K)
0
0A(x)
we have
Amin (K)
Amio (K)
that
07(x)
-a2((x)
(ogh(x)TK((ee) Kh(x) azh(x)
Jn (J)()))(())))))))) y K'kK())) - a2h(() -22)((()) Kh(w)
dx
a
P(a))
Ox(x)
P(t)
a1 @x(x)
Amin h(xx)())))))()))))))))))))))(( Kh(x)
1
dx
2 dr
1
< a2h(x) [K(x))
Thus substituting the above equation into Part I of 39 we get
Part I 5
02 1((x))
Kh(#)
a1 V[= - = P(x)
(M, PPOV Kh(x)
4
- ||||||
2
||||| [J][a || 21(x) Kh(x) -
1P( -
Then, substitute the above inequality to 39 and by setting
Amax (K)
L
a2 2 a1 x
Amin (K) LzD +
Amin (K)
27

---

we get
de(x,22) x) dt
1 +
2 ((mmx(K))3))
01 01(Ls + L2) + a2 + H) +
|| a||=11
Q1 02
02 h(x) Kh(x) 4 vf(x) T ((z)f(x)
Thus by setting
2(Amax (K)D2)
a1 I2) + az(M22 + H)
a1 +
+
02
we get
dl de 2)
2 - |=1
-( - 50,
02 8
41
(40)
thus ((x(t), z()) non-increasing with respect to t, which proves Statement 1.
For Statement 2, note that
P(x)VS(>) = (s(x) + (x)) ((()),
thus from 40 we have
d2 (M)))) + {()0))00000))0000)),,,,)),,,<<,0 =(0) ((x(T),((T)) (()(())))))) 1(1011 8
ag a2nmin (K) [n(x(t))
4 + ((x(0),))) - Emin
which completes the proof.
We now prove Statement 3. Note that
02 Amin
01 (K) * 4 (KT-gap(xxMxx)1 S
(2) |1(x(())))= @1 [V](((())+ + (M00)
min
8
thus we have
KKT-Bap(((()x<<(())) ((x(0))=(0)) - Emin
1=
min Aaimia (K) (K)
and thus gives
((z(0), 200)
Emin
inf 0SEST
KKT XKT- <
O(
azdmin (K)
21 T VT
min
and that
X(x(t))) lim KKT--ap(xx())) (a))))))((((((((())) O. +00
further, given that lim X(x(1)) A(t) 0. we have
lim KKT (()))))(=(t)A(() = O, +++00
which completes the proof.
F1 Intuition: optimization with affine constraints
The main goal of this section to provide intuition for why FL-momentum able to accelerate convergence compared with Fl.-proximal. We will focus on the setting where the constraints are affine functions and show that in this setting, the Fl.-proximal method corresponds the projected gradient descent, and the FL-momentum corresponds to its momentum--cccelrratee version.
28

---

The problem that we consider is
min f(z)
(41)
8.1. Ax + b=
In this setting, the forward Euler discretization for FL-proximal is given by
T ==== (V)
77
A1-A((AA") (1-= (AAA)
(z) - vz
(42)
where Proj is the projection onto the hyperplane Ar b 0. Note that the above scheme is equivalent to projected gradient descent. Further, the forward Euler discretization for FL-momentum given by
Wi It + 3(x; I(-1)
d = (Ja(wa))n((w))") (J](w,)V(()) -(ww))
THE Wy ( (A (A (Vf(w:)) (Aw++b)
-A- (
Proj( fw:)).
Note that this is exactly the momentum-accellrated projected gradient descent (cf. 20
G Auxiliaries
Lemma 2. The following two problems are the dual problem for each other.
min V (x xz) (x TT) x1)
2n
(43)
s.l. M(x)) + (xx))(( - xz) 50
X20 f (a (=( - - (((a)))
min
(44)
Proof. We write the lagrangian for 443
1
L(x,A)= = (f(x)) (x x²) +
(x 22)
T(z:)- + (h(z)) + (xx)((x Iz))
27)
Note that strong duality holds for
(43) Thus solving
is equivalent to solving
min max v((x)) (x 2x)) +
(x xτ) xx) TT(x)) -1) ( + JI zz))
A20
21
1
max min Vf(Iz) (x - -xe)= + (a- xx) TT(e) -z x.) + T (h(,,) + J/ x2))
X20
F
2n
Note that
min ff(z)) (x 2x) [( 21)
Txx xz) T (h(2)) + Ja.))( -xz))
- + (2))))
substituting this to the lagrangian we get
max max L(x, A)
X20
29

---

((=)) JA (V)())))) 71 A) / (o(z
max X20
+ AT - -
(oN())))( T [I(()))(+:\(\((>)) (
(1(x)))
1/
max X20
1 M
min A A20 2
The proof is thus completed by strong duality
Lemma 3. Assumption along with Assumption 7 is sufficient condition of Assumption
Proof. From Lemma we have that
A ( X -
Ami (onca((())) $ X (n()))))))<
(I) Ky) s (=
Ky)ll
( Ky)l
Amax V + ||Amax (K))(=(0)) (1)mx(())M(000))
Amiu (Jr (z)
Amin
Amin (J
From Assumption and Assumption
we have that
Amin (Ju(T) & D2, [[C]( S M
And thus
|a|| Amax M + ( (K)A(<(0))] D2.
Amin
Hence, Assumption satisfied by setting L = Amaa M² 4 Amarr ((((()))) D
Amin
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not move the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], [NA].
[NA] means either that the question is Not Applicable for that particular paper the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA),
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evalu- ation. While "[Yes] " generally preferable to "[No] " perfectly acceptable to answer "[No] provided proper justification is given (c.g., "error bars are not reported because it would be too
30

---

computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Check- list",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately summarize the main theoretical and empirical contributions. They clearly reflect the scope and significance of the work and include relevant assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper includes multiple remarks discussing limitations, assumptions, etc.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate" "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
31

---

If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results are accompanied by clearly stated assumptions and complete proofs, either in the main text or the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Ye]]
Justification: Although primarily theoretical and foundational, this work includes compre- hensive details of the experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (c.g., g., in the case of large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
also be provided via
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
32

---

(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results way produce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: An anonymized version of the code and data is included in the supplemental material,
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips...
ssiicnolicciss))o for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips cc/ wtcc/pullcc/ae hemoliccguides.. htt///ddd///odeuumbm Code/ubmiiiieeoe wwweeeSuumiiiiiiiiii ssionPolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper specifies numerical details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
33

---

Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: This paper considers deterministic algorithms there's error bars, but measure the performance with standard criteria in literature.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of eπrоrs is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Ys]]
Justification: This work is primarily theoretical, and the numerical simulations are simple and lightweight. They were run on standard personal computer and do not require spe- cialized hardware or significant computational resources, so detailed compute reporting is not necessary.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't i make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research adheres to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
34

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Although this work is primarily theoretical and foundational, we conduct experiments on representative applications and discuss how the theoretical insights could inform broader practical use.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact spe- eific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work does not involve models or datasets with high risk of misuse that would require safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
of the model, for example by
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
35

---

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: This paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, patpheetsspppaerwiiiii wwwppswithhooeeooooppp... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of a dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors are encouraged reach out the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and sereenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
36

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to our LIM policy hnttps://neuriie ccc/Conferencce///255//////////e for what should or should not be described.
37


Reviews:
[{"review_id": "v0vp4KneXf", "reviewer": "Reviewer_fiZS", "summary": "The paper discusses methods for solving non-convex constrained optimization using a feedback linearization approach. Specifically, the optimization process is considered as a dynamical system, where the vector field is the first derivative of the Lagrangian, and the Lagrange multipliers are inputs to the system.   The feedback linearization approach allows the authors to formulate the control law in terms of the gradients and Jacobian matrix of the objective and constraints. Then, the stability of the closed-loop system implies convergence of the states to an equilibrium point with respect to the KKT-gap. Finally, discretization of the continuous-time system shows the equivalence of the feedback linearization approach to SQP algorithms. The second part of the paper generalizes the results to inequality constrained problems.", "questions": "I would have liked to see an analysis of how the proposed approach compares with traditional methods. The results are interesting from a theoretical perspective, but is this a better approach in general and will this line of research ultimately have a significant practical benefit?", "limitations": "no limitations acknowledged.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "In general, the paper is well-structured and the contribution is sufficient. However, there are several comments.\n\nFirst, the global  convergence terminology for the paper is unusual. The authors consider the KKT-gap as a convergence metric, which should be addressed more carefully. For example, Theorems 1 and 4 show that the KKT gap goes to zero. However, the first-order KKT conditions (the KKT gap is equal $0$) are a necessary condition for optimality, so the equilibrium point $x*$ may not be a solution to the optimization problem.  Furthermore, the convergence of $x(t)$ to any equilibrium point is not guaranteed.  Thus, I would recommend adding a definition of global convergence as used in the theorems. Also, for clarity of the results, the paper should discuss the relationship between the global convergence and the convergence of $x(t)$ to the optimal solution $x*$.\n\nSecond, Lyapunov functions are required to be positive-definite everywhere except the equilibrium point, where the function must be equal $0$. However, the proposed Lyapunov functions, $l$, in theorems (Thm. 1 and 4) do not satisfy the condition. These conditions, along with non-increasing with respect to time of the Lyapunov function, guarantee the stability property of the system. Authors should avoid calling this function a Lyapunov function or modify it to satisfy the conditions.\n\nThird,  the relationship with SQP methods is interesting. However, the assumptions required for Thms. 2 and 3 are missing. For example, Thm. 3 leverages strong duality. However, the optimization in Eqn. (15) is not necessarily convex, because of non-convexity of $f$ and $h$.\n\nFinally, there are minor formatting issues. Specifically, equations in Property 3 in Theorems 3 and 4 should be formatted more accurately. For example, Theorem 3 the norm symbol in the equation is missing. The equation in Theorem~4 (Property 3) is hard to understand without additional spaces between symbols.", "was_revised": true, "final_justification": "The authors have prepared careful responses to reviewer questions. The paper is well-written and the results interesting. I have no significant concerns.", "rebuttal": "We sincerely thank the reviewer for carefully reading our paper and providing constructive feedback, as well as for their positive assessment of our work! Below, we address the technical questions raised by the reviewer.\n\n1. **The “global convergence” terminology**: *The reviewer points out that the use of “global convergence” in the paper is unconventional, as the theorems only show that the KKT gap converges to zero, which is a necessary but not sufficient condition for optimality.*\n\n **Response**: We thank the reviewer for this important observation. Indeed in the paper we focus on convergence of the KKT gap to zero from any initial condition. We agree that this does not imply convergence to a global or local optimum, especially in the non-convex setting. Our goal is to establish convergence to a first-order KKT point, which, although an easier task than converging to the global optimal, is still challenging in the non-convex constrained optimization settings. We will revise the manuscript to clarify that our convergence criterion is based on KKT gap and acknowledge its limitations. The following discussion will be added:  \n>“In this paper, we use the term ‘convergence’ to refer to the decay of the KKT gap to zero. While this is weaker than convergence to a local or global optimum, it remains a meaningful guarantee in nonconvex constrained optimization. Extending the framework to incorporate saddle-point escape techniques and ensure convergence to local optima is an important direction for future work.”\n\n2. **The “Lyapunov function” terminology**: *The reviewer points out that the functions labeled as Lyapunov functions in Theorems 1 and 4 do not meet the standard definition (i.e., being positive-definite and zero only at the equilibrium)*.\n\n**Response**: We agree with the reviewer that the proposed function does not meet the strict criteria of a Lyapunov function. To avoid confusion, we will revise the terminology and refer to it as a \"merit function\", which is more appropriate in the context of optimization. We sincerely thank the reviewer for this helpful suggestion, which improves both the clarity and rigor of the paper.\n\n3. **Assumptions for Theorems 2 and 3**: *The reviewer notes that some assumptions for Theorems 2 and 3 are missing. The reviewer also questions whether strong duality still holds in Theorem 3 since the objective function $f$ and constraints $h$ can be nonconvex.*\n\n**Response**: We thank the reviewer for the valuable feedback. For Theorem 2, we acknowledge that it relies on Assumption 3, which ensures the invertibility of $J_h(x)J_h(x)^\\top$. This condition guarantees the validity of the feedback linearization and, accordingly, we will revise the manuscript to clearly state this requirement in the theorem statement.\n\nRegarding Theorem 3, we would like to clarify that the only necessary assumption is the feasibility of Equation (15), as noted in the paper. While it is true that $f$ and $h$ may be non-convex in the original problem, Theorem 3 concerns convex quadratic programming subproblems (Equation (14) and (15)), which are solved at each iteration. Under the relaxed Slater condition, strong duality holds for this subproblem even when the overall problem is non-convex. We have revised the text to emphasize this distinction and ensure the assumptions are stated with better clarity. We are grateful to the reviewer for prompting this important clarification and we would be happy to elaborate further if needed.\n\n4. **Comparison with existing traditional methods and practical benefits**: *The reviewer finds the theoretical framework interesting but asks whether the proposed approach offers practical benefits.*\n\n\n**Response**: We thank the reviewer for finding the theoretical framework interesting and for raising the questions. With regard to comparison with existing methods, due to the space limits, we have deferred a more thorough discussion and comparison in the appendix. We would like to kindly point the reviewer to **Remark 4** in the main text for a summary of the comparison, and **Appendix A** for a detailed comparison with existing first order methods, as well as Appendix **B.1** for corresponding numerical results to compare these methods. Here we briefly quote Remark 4 for the convenience of the reviewer:\n\n>“We briefly compare FL-proximal methods with other first-order approaches, including Primal-Dual Gradient Descent (PDGD), Projected Gradient Descent (PGD), and the Augmented Lagrangian Method (ALM). PDGD has been well studied (Kose 1956, Wang and Elia 2011, Qu and Li 2018), but is largely limited to convex settings and may fail in nonconvex problems (Zhang and Luo 2020, Applegate et al. 2024). PGD also struggles with nonconvexity, as projections onto nonconvex sets are often intractable. ALM can handle nonconvex constraints, but each iteration requires solving a potentially expensive nonconvex subproblem. In contrast, when the constraint dimension is small relative to that of x, SQP-based methods like ours often perform better in practice (​​Gould et al. 2003, Liu et al. 2018)”\n\n\nWith regard to the practical benefits of the proposed theoretical framework, we would like to emphasize that, while our perspective provides an alternative—but mathematically equivalent—formulation of the first-order KKT conditions through the lens of control and dynamical systems, it offers a fresh viewpoint that can inspire novel algorithmic designs with potentially advantageous properties. For example, our momentum-accelerated method is directly motivated by momentum mechanisms in dynamical systems. Furthermore, as discussed in **Remark 2 (Extensions of the FL approach)** and **Appendix B.2**, our framework opens up additional opportunities for control-inspired enhancements. In particular, beyond using purely proportional control, one could employ **Proportional-Integral (PI) control** to improve robustness against system misspecification or approximation errors in the computation of the inverse of $J_h(x)J_h(x)^\\top$. To the best of our knowledge, such PI-inspired approaches have not been explored in the optimization literature, and we believe this also illustrates the broader potential of our methodology.\n\nWe hope the above discussion helps clarify the reviewer’s questions, and we would be happy to engage in further discussion if needed.\n\n5. **Formatting issues.**\n\n**Response**: We sincerely thank the reviewer for reading through the technical sections of the papers carefully and thoroughly! We’ve corrected the formatting issues raised by the reviewer in the revised version of the paper.\n\n6. **No limitation acknowledged.**\n\n**Response**: We thank the reviewer for pointing out the absence of an explicit summary of limitations in the conclusion section. We sincerely apologize for this oversight. That said, we would like to respectfully note that several parts of the main text (such as Remark 5 and Lines 147 and 245–249) do discuss the assumptions and limitations of our results. In response to the reviewer’s suggestion, we will add the following dedicated paragraph in the conclusion of the revised manuscript to more thoroughly summarize these limitations:\n>“Several limitations of our framework remain. First, our analysis ensures convergence of the best-found iterate to a first-order KKT point, but does not distinguish between local optima, global optima, and saddle points. Extending the framework to guarantee convergence to local optima via saddle-point escape mechanisms is an important direction for future work. Second, the applicability of feedback linearization depends on structural assumptions such as the Linear Independence Constraint Qualification (LICQ). Third, while FL-momentum empirically accelerates convergence, our current analysis yields the same $O(1/\\sqrt{T})$ rate as its non-accelerated counterpart. Relaxing these assumptions and strengthening convergence guarantees remain promising directions for future research.”\n\nWe sincerely thank the reviewer once again for the thoughtful and careful reading of our paper, and for recognizing its contributions. We greatly appreciate the time and effort dedicated to understanding our work in depth. We are encouraged by the feedback and would be happy to engage in further discussions or clarifications."}, {"review_id": "VbmeFbei2y", "reviewer": "Reviewer_9MiY", "summary": "This paper builds upon the idea proposed in [15], which uses feedback linearization as a continuous analog to solve constrained optimization problems. The authors establish a sublinear convergence rate for both equality- and inequality-constrained cases. They also draw a connection between the feedback linearization framework and sequential quadratic programming (SQP), showing that an Euler discretization of the feedback linearization setup is equivalent to SQP.", "questions": "What are some concrete application scenarios where this method would be particularly useful?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The convergence results contribute meaningfully to the existing literature on continuous-time optimization methods.\n- The connection between feedback linearization and SQP is insightful. The design of the linear dynamics sheds light on the behavior of different SQP variants with different quadratic regularization terms.\n- The inclusion of a momentum-accelerated version of the algorithm, along with a partial convergence result, is a nice addition.\n\nWeaknesses:\n- The relevance of this work to the NeurIPS audience is unclear. While the paper is well-suited for optimization, control, or applied mathematics venues, its connection to machine learning is limited.\n- The momentum-accelerated variant has the same convergence rate as the original algorithm and performs worse empirically than the FL-Newton variant, making its practical value unclear.\n- The paper presents only one numerical example, which limits the empirical validation of the approach.", "was_revised": true, "final_justification": "The response has addressed my concerns to a large extent. Overall, I believe this paper offers a valuable—albeit somewhat narrow—contribution, and I therefore recommend acceptance.", "rebuttal": "We sincerely thank the reviewer for the comments and feedback as well as the recognition of our theoretical contributions, the connection to SQP, and the momentum-based extension. Below we address the comments and concerns of the reviewer one by one:\n\n1. **Relevance of the work to NeurIPS audiences**: *“While the paper is well-suited for optimization, control, or applied mathematics venues, its connection to machine learning is limited.”*\n\n**Response**: We sincerely thank the reviewer for the feedback. While our work is indeed grounded in control and optimization theory, we believe it contributes to a methodological area that continues to play a central role in machine learning. Optimization lies at the heart of nearly all ML methods: the training of models, from logistic regression to deep neural networks, relies on solving high-dimensional optimization problems. As emphasized in Bottou, Curtis, and Nocedal (2018, Optimization Methods and Software), many of the field’s most significant advances have been enabled by progress in optimization algorithms themselves.\n\nMore specifically, constrained optimization has become increasingly central to several emerging areas in machine learning. In safety-critical domains, such as robotics (Alonso-Mora et al., IJRA 2017)) and power grids operations (Dommel and Tinney, PAS 2007), etc., machine learning models must satisfy hard safety or operational constraints at training and inference time. The need for optimization methods that offer convergence guarantees under constraints is critical in such settings, where violations can lead to safety breaches or system failure. In addition, fairness-aware machine learning imposes constraints to ensure statistical parity or mitigate disparate treatment. For instance, Agarwal et al. (ICML 2018) cast fair classification as a constrained optimization problem, showing how algorithmic tools can effectively balance accuracy and fairness criteria. \n\nOur contribution builds directly on this growing demand for principled constrained optimization methods in machine learning. By offering a scalable algorithm with theoretical convergence guarantees under general constraints, our work contributes to the foundational tools needed to support these applications. We will revise the manuscript to better highlight these connections and we appreciate the opportunity to clarify the relevance of our work. We hope this discussion addresses the reviewer’s concern, and we would be happy to further elaborate if helpful.\n\n2. **Momentum acceleration rate**: *“The momentum-accelerated variant has the same convergence rate as the original algorithm and performs worse empirically than the FL-Newton variant, making its practical value unclear.”*\n\n**Response**: We thank the reviewer for raising this point. Regarding the reviewer’s observation about FL-momentum and FL-proximal sharing the same theoretical convergence rate, we note that FL-momentum is designed for nonconvex constrained optimization, where standard first-order methods—including momentum-based variants—are typically limited to a rate of $O(1/\\sqrt{T})$. This is the rate we establish for FL-momentum as well. The theoretical result is not aimed at demonstrating acceleration, but rather at establishing the algorithm’s validity through rigorous analysis of its stability and convergence. Furthermore, although our primary focus is on the nonconvex setting, we also show in **Remark 5** and **Appendix F.1** that FL-momentum recovers the behavior of classical **momentum-accelerated projected gradient methods** in the convex case. This connection provides additional theoretical justification for our approach.\n\nRegarding the empirical comparison with FL-Newton, we agree that FL-Newton typically converges faster, as it leverages second-order (Hessian) information. In contrast, FL-momentum is a purely first-order method, which is expected to offer greater computational efficiency at the cost of slightly slower convergence than second-order methods. Our objective is not to outperform FL-Newton, but to show that FL-momentum provides a clear acceleration over FL-proximal, our first-order baseline. We include FL-Newton as a reference to illustrate that FL-momentum can achieve competitive performance while avoiding the computational overhead associated with second-order methods. We will clarify this distinction in the revised manuscript.\n\n3. **Lack of numerical examples**: *“The paper presents only one numerical example, which limits the empirical validation of the approach.”*\n\n**Response**: We acknowledge the reviewer’s concern about the limited number of numerical examples. We would like to kindly point out that apart from the numerical simulations in Section 6, additional numerical experiments are also presented in **Appendix B**. In particular, Appendix B.1 compares our approach with other first-order constrained optimization algorithms and Appendix B.3 provides an additional numerical example on the **optimal power flow** problem, an important real-world constrained optimization problem. Due to space constraints, we have moved the full details to the appendix, but we have highlighted the setting and key results in the main text.\n\n4. **Practical applications**: *“What are some concrete application scenarios where this method would be particularly useful?”*\n\n**Response**: We thank the reviewer for this question. As mentioned in response to the reviewer’s Concern 1, many important problems—such as safe or fair learning, heterogeneous federated learning, and optimal power flow—naturally fall into the category of constrained optimization, where enforcing constraints on model behavior or distributional fairness is critical.  Our approach is well-suited for these settings. To demonstrate its applicability, we have empirically evaluated our method in two representative domains: heterogeneous machine learning (see **Section 6**) and optimal power flow (see **Appendix B.3**).\n\nOnce again, we sincerely thank the reviewer for the constructive feedback and for recognizing the contributions of our work. We hope that our responses have sufficiently addressed the concerns raised by the reviewer. We remain open to further discussion and would be glad to elaborate further if needed."}, {"review_id": "z5c2IeW59t", "reviewer": "Reviewer_SLKh", "summary": "This paper introduces a method for solving constrained optimization problems using feedback linearization (FL), a concept from control theory. The authors establish global convergence rates for this method in equality-constrained problems and reveal a strong connection between the FL approach and the SQP algorithm. They then extend the method to handle inequality constraints and propose a momentum-accelerated FL algorithm that demonstrates faster convergence in practice. The theoretical claims are supported by proofs in the appendices and validated through numerical simulations on a logistic regression problem with fairness constraints.", "questions": "1. In nonlinear control, the feedback linearization control law may not be globally valid. How's this issue resolved?\n\n2. The feedback linearization process often involves complex coordinate transformations and deriving the feedback control law. What is the computational complexity of the FL implementation?\n\n3. How does this approach compare against other solvers? What are potential applications?", "limitations": "The author(s) haven't discussed their limitations in the conclusion section.\n\n1. The theoretical guarantees rely on strong conditions, such as the global boundedness of the objective and constraint function gradients (Assumption 1) and the universal invertibility of the $J_h(x)J_h(x)^\\top$ matrix (Assumption 3). These assumptions may not hold in all practical scenarios.\n\n2. While the FL-momentum method shows empirical speed-up, the theoretical analysis does not formally prove an accelerated convergence rate for the general nonconvex case; it provides the same $O(1/\\sqrt{T})$ rate as the non-accelerated method.\n\n3. The method's reliance on inverting an $m \\times m$ matrix, where $m$ is the number of constraints, could be a bottleneck for problems with a very large number of constraints.\n\n4. The analysis guarantees convergence for the best-found solution over time, which is a weaker guarantee than ensuring the final solution is optimal.\n\nI'd like to change my rating if the author(s) would like to comment on the limitations above,", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis is a solid theoretical paper that bridges the fields of nonlinear control theory and optimization. The most interesting contribution is the link established between feedback linearization and SQP, which provides a fresh perspective on a classic optimization algorithm and lends credibility to the FL approach. The development of a momentum-accelerated version is a practical and novel addition. \n\nWeaknesses:\n\nDespite its strengths, the paper has some limitations, as summarized below. Also, there is a missing 'n' in the paper title, and there might be more typos and mistakes in the main paper.", "was_revised": true, "final_justification": "The pathway demonstrated by the author(s) to the global last-iterate guarantee seems promising and I hope this could be seriously addressed in the final manuscript. I've updated my score to 4.", "rebuttal": "We sincerely thank the reviewer for the constructive feedback and valuable comments! We also appreciate the careful reading that brought several typos to our attention. We will carefully proofread the manuscript and correct these issues in the revised version. Below, we address the reviewer’s technical comments one by one.\n\n1. **Globally validness of feedback linearization**: *“In nonlinear control, the feedback linearization control law may not be globally valid. How's this issue resolved?”*\n\n**Response**: We appreciate the reviewer’s question. In case there’s any potential ambiguity, we would like to briefly clarify that FL is a nonlinear control technique that transforms a nonlinear system into an equivalent linear system via an exact change of variables. Unlike standard linearization, which relies on first-order approximations valid only near a nominal point, FL yields an exact transformation that can, under suitable conditions, hold over the entire domain of interest. However, this global validity depends on structural properties of the system, specifically the regularity of the transformation. If these conditions are not met, the transformation may be only locally valid or ill-defined.\n\nIn our setting, Assumption 3 (LICQ) ensures these conditions are met: it guarantees that the constraint Jacobian has full row rank throughout the feasible domain, thereby ensuring the regularity of the transformation and the existence of a well-defined feedback linearization. We will revise the manuscript to clarify this connection and make the role of LICQ in ensuring validity more explicit.\n\n2. **Computational complexity**: *In Question 2 and Limitation 3, the reviewer questions about the computational complexity of feedback linearization, as it involves computation of the inverse of a $m\\times m$ matrix.*\n\n**Response**: We thank the reviewer for this valuable question. We acknowledge that computing the inverse of an $m \\times m$ matrix is an inherent part of the feedback linearization (FL) approach. However, as noted in **Remark 1**, the method is particularly well-suited for settings where the number of constraints is significantly smaller than the dimension of the variable $x$—a common scenario in many practical applications, such as fair or safe learning, where only a small number of fairness or safety constraints need to be enforced.\n\nMoreover, even in cases where the number of constraints is large and the matrix inversion becomes computationally burdensome, we show that it is possible to **approximate** the inverse efficiently while still maintaining convergence guarantees. Specifically, in **Appendix B.2**, we present a modified FL algorithm that incorporates a **Proportional-Integral (PI) controller** to tolerate approximation error and still ensure convergence to an optimal solution. In the revised manuscript, we will add a brief discussion of this extension in Remark 1 to clarify this potential for reducing computational complexity.\n\n3. **Comparison with existing methods and practical applications**: *“How does this approach compare against other solvers? What are potential applications?”*\n\n**Response**: With regard to comparison with existing methods, due to space limits, we have deferred a more thorough discussion and comparison in the appendix. We would like to kindly point the reviewer to **Remark 4** in the main text for a summary of the comparison, and **Appendix A** for a detailed comparison with existing first order methods, as well as **Appendix B.1** for corresponding numerical results to compare these methods. Here we quote Remark 4 for the convenience of the reviewer:\n\n>“We briefly compare FL-proximal methods with other first-order approaches, including Primal-Dual Gradient Descent (PDGD), Projected Gradient Descent (PGD), and the Augmented Lagrangian Method (ALM). PDGD has been well studied (Kose 1956, Wang and Elia 2011, Qu and Li 2018), but is largely limited to convex settings and may fail in nonconvex problems (Zhang and Luo 2020, Applegate et al. 2024). PGD also struggles with nonconvexity, as projections onto nonconvex sets are often intractable. ALM can handle nonconvex constraints, but each iteration requires solving a potentially expensive nonconvex subproblem. In contrast, when the constraint dimension is small relative to that of x, SQP-based methods like ours often perform better in practice (​​Gould et al. 2003, Liu et al. 2018)”\n\nWith regard to potential applications, we would like to highlight that many important problems—such as fair learning, safe learning, heterogeneous federated learning, and optimal power flow—naturally fall into the category of constrained optimization. Our algorithm is well-suited for these settings and we have empirically evaluated the performance of our method in two representative domains: heterogeneous machine learning (see **Section 6**) and optimal power flow (see **Appendix B.3**).\n\nWe hope the above explanation addresses the reviewer's concerns, and we would be glad to provide further clarification or engage in any additional discussions.\n\n4. **Restrictive Assumptions**: *The reviewer found the Assumption 1 (Boundedness of Gradient) and Assumption 3 (LICQ) relatively restricted and may not hold in practical applications.*\n\n**Response**: We thank the reviewer for raising this important concern. As noted in Footnote 1, Assumption 1 only needs to hold within a compact region where the optimization variables reside during the algorithm’s execution. This is a commonly adopted assumption in the optimization literature (cf. Nocedal and Wright, Numerical Optimization). \n\nRegarding Assumption 3 (LICQ), we acknowledge that it is a strong condition; however, it is a standard assumption in constrained optimization. We provide further context and justification in Appendix A, under the paragraph labeled “SQP.” Moreover, in the inequality-constrained setting, our main convergence results rely on Assumption 4, which is strictly weaker than LICQ. We clarify this relationship in the discussion following Assumption 4 and provide a formal proof in Appendix G. While we recognize that these assumptions may not hold universally in practice, we include a dedicated discussion in Appendix A to highlight their limitations and the potential for extensions beyond the current setting. \n\n5. **Convergence Rate for momentum acceleration**: *“While the FL-momentum method shows empirical speed-up, the theoretical analysis does not formally prove an accelerated convergence rate for the general nonconvex case; it provides the same $O(1/\\sqrt{T})$ rate as the non-accelerated method.”*\n\n**Response**:  We thank the reviewer for this helpful observation. We would like to note that the feedback-linearization-based algorithms in this paper are primarily designed for nonconvex constrained optimization, where standard first-order methods—including momentum-based variants—are typically limited to a rate of $O(1/\\sqrt{T})$. This is the rate we establish for FL-momentum as well. The theoretical result is not aimed at demonstrating acceleration, but rather at establishing the algorithm’s validity through rigorous analysis of its stability and convergence. Furthermore, although our primary focus is on the nonconvex setting, we also show in **Remark 5** and **Appendix F.1** that FL-momentum recovers the behavior of classical **momentum-accelerated projected gradient** methods in the convex case. This connection provides additional theoretical justification for our approach.\n\n6. **Best-iterate instead of last-iterate convergence**: *“The analysis guarantees convergence for the best-found solution over time, which is a weaker guarantee than ensuring the final solution is optimal.”*\n\n**Response**: We fully acknowledge that our convergence analysis guarantees only the best-found solution over time, rather than optimality of the final iterate. This limitation is explicitly noted in our current manuscript (see Lines 162–164). We would also like to point out that convergence guarantees based on the best-found iterate are widely adopted in research on non-convex optimization (cf. Eq. (2) in Danilova et al., “Recent Theoretical Advances in Non-Convex Optimization,” 2022). Ensuring the optimality of the final iterate is substantially more challenging in non-convex landscapes, where saddle points may block convergence of the last iterate. We are grateful for the reviewer’s feedback and would be happy to explore potential extensions that could strengthen guarantees for the last iterate, particularly in convex settings, in future work.\n\n7. **Lack of limitation statement in conclusion section.**\n\n**Response**: We thank the reviewer for highlighting the absence of an explicit summary of limitations in the conclusion section, and we sincerely apologize for this oversight. As noted in our responses above, several parts of the main text (e.g. Remark 5 and Lines 147, 162–164 and 245–249) already discuss the assumptions and limitations of our results. In response to the reviewer’s suggestion, we will add the following dedicated paragraph in the conclusion of the revised manuscript to more thoroughly summarize these limitations:\n\n>“Several limitations of our framework remain. First, our analysis ensures convergence of the best-found iterate to a first-order KKT point, but does not distinguish between local optima, global optima, and saddle points. Extending the framework to guarantee convergence to local optima via saddle-point escape mechanisms is an important direction for future work. Second, the applicability of feedback linearization depends on structural assumptions such as the Linear Independence Constraint Qualification (LICQ). Third, while FL-momentum empirically accelerates convergence, our current analysis yields the same $O(1/\\sqrt{T})$ rate as its non-accelerated counterpart. Relaxing these assumptions and strengthening convergence guarantees remain promising directions for future research.”"}]

Author rebuttal:

Meta-review: This paper builds on the idea of reference [15], which first proposed applying feedback linearization (FL) to constrained optimization, and substantially extends it in both scope and depth. The authors show that modeling optimization dynamics in continuous time and applying FL leads to convergence to first-order KKT points. A key contribution is establishing the connection between FL and Sequential Quadratic Programming (SQP), where discretization of the FL dynamics recovers SQP updates. The framework is extended from equality- to inequality-constrained problems, and a momentum-accelerated variant is proposed with theoretical guarantees and supporting experiments. Overall, the paper bridges control and optimization, providing new perspectives on classical constrained methods.

Reviewers agreed the work is rigorous and well-structured, highlighting the SQP connection and momentum-based extension as notable contributions. Concerns centered on restrictive conditions (bounded gradients, LICQ), limited empirical validation, and unclear ML relevance. In rebuttal, the authors constructively clarified terminology (introducing “merit function”), explicitly added missing assumptions, emphasized broader ML applications such as safe/fair learning, federated optimization, and power flow, and acknowledged limitations directly in the conclusion. They also pointed to additional experiments in the appendix and clarified the role of FL-momentum relative to FL-Newton.

Overall, this work provides further novel perspective on constrained optimization through the lens of feedback linearization. While empirical validation remains modest and the assumptions limit generality, the methodological contribution and relevance to constrained ML applications justify acceptance.
Decision: Accept (poster)