Title: Time-Reversed Dissipation Induces Duality Between Minimizing Gradient Norm and Function Value
Venue: NeurIPS 2023
Accepted: yes
Keywords: Convex Optimization

Acceleration

First-Order methods
Abstract: In convex optimization, first-order optimization methods efficiently minimizing function values have been a central subject study since Nesterov's seminal work of 1983. Recently, however, Kim and Fessler's OGM-G and Lee et al.'s FISTA-G have been presented as alternatives that efficiently minimize the gradient magnitude instead. In this paper, we present H-duality, which represents a surprising one-to-one correspondence between methods efficiently minimizing function values and methods efficiently minimizing gradient magnitude. In continuous-time formulations, H-duality corresponds to reversing the time dependence of the dissipation/friction term. To the best of our knowledge, H-duality is different from Lagrange/Fenchel duality and is distinct from any previously known duality or symmetry relations. Using H-duality, we obtain a clearer understanding of the symmetry between Nesterov's method and OGM-G, derive a new class of methods efficiently reducing gradient magnitudes of smooth convex functions, and find a new composite minimization method that is simpler and faster than FISTA-G.

Time-Reversed Dissipation Induces Duality Between Minimizing Gradient Norm and Function Value

Abstract

In convex optimization, first-order optimization methods efficiently minimizing function values have been a central subject study since Nesterov's seminal work of 1983. Recently, however, Kim and Fessler's OGM-G and Lee et al.'s FISTA-G have been presented as alternatives that efficiently minimize the gradient magnitude instead. In this paper, we present H-duality, which represents a surprising one-to-one correspondence between methods efficiently minimizing function values and methods efficiently minimizing gradient magnitude. In continuous-time formulations, H-duality corresponds to reversing the time dependence of the dissipation/friction term. To the best of our knowledge, H-duality is different from Lagrange/Fenchel duality and is distinct from any previously known duality or symmetry relations. Using H-duality, we obtain a clearer understanding of the symmetry between Nesterov's method and OGM-G, derive a new class of methods efficiently reducing gradient magnitudes of smooth convex functions, and find a new composite minimization method that is simpler and faster than FISTA-G.

1 Introduction

Since Nesterov's seminal work of 1983 [37], accelerated first-order optimization methods that efficiently reduce function values have been central to the theory and practice of large-scale optimization and machine learning. In 2012, however, Nesterov initiated the study of first-order methods that efficiently reduce gradient magnitudes of convex functions [41]. In convex optimization, making the function value exactly optimal is equivalent to making the gradient exactly zero, but reducing the function-value suboptimality below a threshold is not equivalent to reducing the gradient magnitude below a threshold. This line of research showed that accelerated methods for reducing function values, such as Nesterov's FGM [37], the more modern OGM [26], and the accelerated composite optimization method FISTA [11] are not optimal for reducing gradient magnitude, and new optimal alternatives, such as OGM-G [29] and FISTA-G [31], were presented.

These new accelerated methods for reducing gradient magnitudes are understood far less than those for minimizing function values. However, an interesting observation of symmetry, described in Section 2, was made between these two types of methods, and it was conjectured that this symmetry might be a key to understanding the acceleration mechanism for efficiently reducing gradient magnitude.

Contribution. We present a surprising one-to-one correspondence between methods efficiently minimizing function values and methods efficiently minimizing gradient magnitude. We call this correspondence H-duality and formally establish a duality theory in both discrete- and continuoustime dynamics. Using H-duality, we obtain a clearer understanding of the symmetry between FGM/OGM and OGM-G, derive a new class of methods efficiently reducing gradient magnitudes, and find a new composite minimization method that is simpler and faster than FISTA-G, the prior state-of-the-art in efficiently reducing gradient magnitude in the composite minimization setup.

1.1 Preliminaries and Notation

Given f : R d → R , write f ⋆ = inf x ∈ R d f ( x ) ∈ ( -∞ , ∞ ) for the minimum value and x ⋆ ∈ argmin x ∈ R n f ( x ) for a minimizer, if one exists. Throughout this paper, we assume f ⋆ = -∞ , but we do not always assume a minimizer x ⋆ exists. Given a differentiable f : R d → R and a pre-specified value of L > 0 , we define the notation

̸

for x, y ∈ R d . A differentiable function f : R d → R is convex if the convexity inequality [ x, y ] ≤ 0 holds for all x, y ∈ R d . For L > 0 , a function f : R d → R is L -smooth convex if it is differentiable and the cocoercivity inequality J x, y K ≤ 0 holds for all x, y ∈ R d [42]. If f has a minimizer x ⋆ , then J x, ⋆ K = J x, x ⋆ K , but the notation J x, ⋆ K is well defined even when a minimizer x ⋆ does not exist. If f is L -smooth convex, then J x, ⋆ K ≤ 0 holds for all x ∈ R d [42].

Throughout this paper, we consider the duality between the following two problems.

(P1) Efficiently reduce f ( x N ) -f ⋆ assuming x ⋆ exists and ∥ x 0 -x ⋆ ∥ ≤ R .

(P2) Efficiently reduce 1 2 L ∥∇ f ( y N ) ∥ 2 assuming f ⋆ > -∞ and f ( y 0 ) -f ⋆ ≤ R .

Here, R ∈ (0 , ∞ ) is a parameter, x 0 and y 0 denote initial points of methods for (P1) and (P2), and x N and y N denote outputs of methods for (P1) and (P2).

Finally, the standard gradient descent (GD) with stepsize h is

1.2 Prior works

Classically, the goal of optimization methods is to reduce the function value efficiently. In the smooth convex setup, Nesterov's fast gradient method (FGM) [37] achieves an accelerated O (1 /N 2 ) -rate, and the optimized gradient method (OGM) [26] improves this rate by a factor of 2 , which is, in fact, exactly optimal [18].

On the other hand, Nesterov initiated the study of methods for reducing the gradient magnitude of convex functions [41] as such methods help us understand non-convex optimization better and design faster non-convex machine learning methods. For smooth convex functions, (GD) achieves a O (( f ( x 0 ) -f ⋆ ) /N ) -rate on the squared gradient magnitude [34, Proposition 3.3.1], while (OGM-G) achieves an accelerated O (( f ( x 0 ) -f ⋆ ) /N 2 ) -rate [29], which matches a lower bound and is therefore optimal [35, 36]. Interestingly, (OGM) and (OGM-G) exhibit an interesting hint of symmetry, as we detail in Section 2, and the goal of this work is to derive a more general duality principle from this observation.

In the composite optimization setup, iterative shrinkage-thresholding algorithm (ISTA) [13, 45, 16, 14] achieves a O ( ∥ x 0 -x ⋆ ∥ 2 /N ) -rate on function-value suboptimality, while the fast iterative shrinkage-thresholding algorithm (FISTA) [11] achieves an accelerated O ( ∥ x 0 -x ⋆ ∥ /N 2 ) -rate. On the squared gradient mapping norm, FISTA-G achieves O (( F ( x 0 ) -F ⋆ ) /N 2 ) -rate [31], which is optimal [35, 36]. Analysis of an accelerated method often uses the estimate sequence technique [38, 8, 39, 9, 40, 32] or a Lyapunov analysis [37, 11, 50, 10, 53, 1, 5, 6, 7, 43]. In this work, we focus on the Lyapunov analysis technique, as it is simpler and more amenable to a continuous-time view.

The notion of duality is fundamental in many branches of mathematics, including optimization. Lagrange duality [46, 47, 12], Wolfe duality [57, 15, 49, 33], and Fenchel-Rockacheller duality [20, 46] are related (arguably equivalent) notions that consider a pairing of primal and dual optimization problems. The recent gauge duality [21, 22, 3, 58] and radial duality [24, 23] are alternative notions of duality for optimization problems. Attouch-Théra duality [4, 48] generalizes Fenchel-Rockacheller to the setup of monotone inclusion problems. In this work, we present H-duality, which is a notion of duality for optimization algorithms , and it is, to the best of our knowledge, distinct from any previously known duality or symmetry relations.

2 H-duality

In this section, we will introduce H-duality, state the main H-duality theorem, and provide applications. Let N ≥ 1 be a pre-specified iteration count. Let { h k,i } 0 ≤ i<k ≤ N be an array of (scalar) stepsizes and identify it with a lower triangular matrix H ∈ R N × N via H k +1 ,i +1 = h k +1 ,i if 0 ≤ i ≤ k ≤ N -1 and H k,i = 0 otherwise. An N -step Fixed Step First Order Method (FSFOM) with H is

for any initial point x 0 ∈ R d and differentiable f . For H ∈ R N × N , define its anti-transpose H A ∈ R N × N with H A i,j = H N -j +1 ,N -i +1 for i, j = 1 , . . . , N . We call [FSFOM with H A ] the H-dual of [FSFOM with H ].

2.1 Symmetry between OGM and OGM-G

Let f be an L -smooth convex function. Define the notation z + = z -1 L ∇ f ( z ) for z ∈ R d . The accelerated methods OGM [19, 26] and OGM-G [29] are

for k = 0 , . . . , N -1 , where { θ i } N i =0 are defined as θ 0 = 1 , θ 2 i +1 -θ i +1 = θ 2 i for 0 ≤ i ≤ N -2 , and θ 2 N -θ N = 2 θ 2 N -1 . 1 (OGM) and (OGM-G) are two representative accelerated methods for the setups (P1) and (P2), respectively. As a surface-level symmetry, the methods both access the { θ i } N i =0 sequence, but (OGM-G) does so in a reversed ordering [29]. There turns out to be a deeperlevel symmetry: (OGM) and (OGM-G) are H-duals of each other, i.e., H A OGM = H OGM-G. The proof structures of (OGM) and (OGM-G) also exhibit symmetry. We can analyze (OGM) with the Lyapunov function

for -1 ≤ k ≤ N with { u i } N i =0 = (2 θ 2 0 , . . . , 2 θ 2 N -1 , θ 2 N ) and u -1 = 0 . Since J · , · K ≤ 0 and { u i } N i =0 is a positive monotonically increasing sequence, {U k } N k = -1 is dissipative, i.e., U N ≤ U N -1 ≤ · · · ≤ U 0 ≤ U -1 . So

where z = N ∑ i =0 u i -u i -1 L ∇ f ( x i ) . The justification of ( · ) is the main technical challenge of this analysis, and it is provided in Appendix B.2. Dividing both sides by θ 2 N , we conclude the rate

Likewise, we can analyze (OGM-G) with the Lyapunov function

for 0 ≤ k ≤ N with { v i } N i =0 = ( 1 θ 2 N , 1 2 θ 2 N -1 , . . . , 1 2 θ 2 0 ) . Similarly, {V k } N k =0 is dissipative, so

Again, the justification of ( ◦ ) is the main technical challenge of this analysis, and it is provided in Appendix B.2. The crucial observations are (i) u i = 1 /v N -i for 0 ≤ i ≤ N and (ii) the convergence rates share the identical factor 1 /θ 2 N = 1 /u N = v 0 . Interestingly, a similar symmetry relation holds between method pairs [ (OBL-F ♭ ) , (OBL-G ♭ ) ] [44] and [ (GD) , (GD) ] , which we discuss later in Section 2.4.

2.2 H-duality theorem

The symmetry observed in Section 2.1 is, in fact, not a coincidence. Suppose we have N -step FSFOMs with H and H A . We denote their iterates as { x i } N i =0 and { y i } N i =0 . For clarity, note that { u i } N i =0 are free variables and can be appropriately chosen for the convergence rate analysis. For the FSFOM with H , define {U k } N k = -1 with the general form (2) with u -1 = 0 . If 0 = u -1 ≤ u 0 ≤ u 1 ≤ · · · ≤ u N , then {U k } N k = -1 is monotonically nonincreasing (dissipative). Assume we can show

To clarify, since { x i } N i =0 lies within span { x 0 , ∇ f ( x 0 ) , . . . , ∇ f ( x N ) } , the U N depends on ( x 0 , x ⋆ , {∇ f ( x i ) } N i =0 , { u i } N i =0 , H ) . If (C1) holds, the FSFOM with H exhibits the convergence rate

For the FSFOM with H A , define {V k } N k =0 with the general form (3). Also, note that { v i } N i =0 are free variables and can be appropriately chosen for the convergence rate analysis. If 0 ≤ v 0 ≤ v 1 ≤ · · · ≤ v N , then {V k } N k =0 is monotonically nonincreasing (dissipative). Assume we can show

To clarify, since { y i } N i =0 lies within span { y 0 , ∇ f ( y 0 ) , . . . , ∇ f ( y N ) } , the V N depends on ( y 0 , {∇ f ( y i ) } N i =0 , f ⋆ , { v i } N i =0 , H A ) . If (C2) holds, the FSFOM with H A exhibits the convergence rate

We now state our main H-duality theorem, which establishes a correspondence between the two types of bounds for the FSFOMs induced by H and H A .

Theorem 1. Consider sequences of positive real numbers { u i } N i =0 and { v i } N i =0 related through v i = 1 u N -i for i = 0 , . . . , N . Let H ∈ R N × N be lower triangular. Then,

Theorem 1 provides a sufficient condition that ensures an FSFOM with H with a convergence guarantee on ( f ( x N ) -f ⋆ ) can be H-dualized to obtain an FSFOM with H A with a convergence guarantee on ∥∇ f ( y N ) ∥ 2 . To the best of our knowledge, this is the first result establishing a symmetrical relationship between (P1) and (P2). Section 2.3 provides a proof outline of Theorem 1.

2.3 Proof outline of Theorem 1

Define

Expanding U and V reveals that all function value terms are eliminated and only quadratic terms of {∇ f ( x i ) } N i =0 and {∇ f ( y i ) } N i =0 remain. Now, (C1) and (C2) are equivalent to the conditions

respectively. Next, define g x = [ ∇ f ( x 0 ) |∇ f ( x 1 ) | . . . |∇ f ( x N ) ] ∈ R d × ( N +1) and g y = [ ∇ f ( y 0 ) |∇ f ( y 1 ) | . . . |∇ f ( y N ) ] ∈ R d × ( N +1) . Weshowthat there is S ( H,u ) and T ( H A , v ) ∈ S N +1 such that

Next, we find an explicit invertible matrix M ( u ) ∈ R ( N +1) × ( N +1) such that S ( H,u ) = M ( u ) ⊺ T ( H A , v ) M ( u ) . Therefore,

with g y = g x M ( u ) ⊺ and we conclude the proof. This technique of considering the quadratic forms of Lyapunov functions as a trace of matrices is inspired by the ideas from the Performance Estimation Problem (PEP) literature [19, 55]. The full proof is given in Appendix A.

2.4 Verifying conditions for H-duality theorem

In this section, we illustrate how to verify conditions (C1) and (C2) through examples. Detailed calculations are deferred to Appendices B.1 and B.2.

Example 1. For (OGM) and (OGM-G), the choice

leads to

Therefore, (C1) and (C2) hold.

Example 2. Again, define z + = z -1 L ∇ f ( z ) for z ∈ R d . Consider the FSFSOMs [44]

and

where y + -1 = y 0 , x + -1 = x 0 and γ = √ N ( N +1) / 2 . It turns out that (OBL-F ♭ ) and (OBL-G ♭ ) are H-duals of each other. The choice

leads to

where u -1 = 0 . Since U and V are expressed as a sum of squares, (C1) and (C2) hold.

Example 3. Interestingly, (GD) is a self-dual FSFOM in the H-dual sense. For the case h = 1 , the choice

leads to

for some { s ij } and { t ij } stated precisely in Appendix B.2. V ≥ 0 can be established by showing that the { t ij } forms a diagonally dominant and hence positive semidefinite matrix [29]. U ≥ 0 can be established with a more elaborate argument [19], but that is not necessary; V ≥ 0 implies (C2), and, by Theorem 1, this implies (C1).

2.5 Applications of the H-duality theorem

A family of gradient reduction methods. Parameterized families of accelerated FSFOMs for reducing function values have been presented throughout the extensive prior literature. Such families generalize Nesterov's method and elucidate the essential algorithmic component that enables acceleration. For reducing gradient magnitude, however, there are only four accelerated FSFOMs (OGM-G), (OBL-G ♭ ), and M-OGM-G [60], and [17, Lemma 2.6]. Here, we construct a simple parameterized family of accelerated FSFOMs for reducing gradient magnitude by H-dualizing an accelerated FSFOM family for reducing function values.

Let { t i } N i =0 and { T i } N i =0 be sequences positive real numbers satisfying t 2 i ≤ 2 T i = 2 ∑ i j =0 t j for 0 ≤ i ≤ N -1 and t 2 N ≤ T N = ∑ N j =0 t j . Consider a family of FSFOMs

for k = 0 , 1 , . . . , N -1 , where x + -1 = x 0 . This family coincides with the GOGM of [28], and it exhibits the rate [28, Theorem 5]

which can be established from (2) with u i = T i for 0 ≤ i ≤ N .

Corollary 1. The H-dual of (6) is

for k = 0 , . . . , N -1 , where y + -1 = y 0 , and it exhibits the rate

Proof outline. By Theorem 1, (C2) holds with v i = 1 /T N -i for 0 ≤ i ≤ N . We then use (5).

When T i = t 2 i for 0 ≤ i ≤ N , the FSFOM (6) reduces to Nestrov's FGM [37] and its H-dual is, to the best of our knowledge, a new method without a name. If t 2 i = 2 T i for 0 ≤ i ≤ N -1 and t 2 N = T N , (6) reduces to (OGM) and its H-dual is (OGM-G). If t i = i +1 for 0 ≤ i ≤ N -1 and t N = √ N ( N +1) / 2 , (6) reduces to (OBL-F ♭ ) and its H-dual is (OBL-G ♭ ).

Gradient magnitude rate of (GD) . For gradient descent (GD) with stepsize h , the H matrix is the identity matrix scaled by h , and the H-dual is (GD) itself, i.e., (GD) is self-dual. For 0 < h ≤ 1 , the rate f ( x N ) -f ⋆ ≤ 1 2 Nh +1 L 2 ∥ x 0 -x ⋆ ∥ 2 , originally due to [19], can be established from (2) with { u i } N i =0 = ( . . . , (2 Nh +1)( i +1) 2 N -i , . . . , 2 Nh +1 ) . Applying Theorem 1 leads to the following.

Corollary 2. Consider (GD) with 0 < h ≤ 1 applied to an L -smooth convex f . For N ≥ 1 ,

To the best of our knowledge, Corollary 2 is the tightest rate on gradient magnitude for (GD) for the general step size 0 < h < 1 , and it matches [53, Theorem 3] for h = 1 .

Resolving conjectures of A ⋆ -optimality of (OGM-G) and (OBL-F ♭ ) . The prior work of [44] defines the notion of A ⋆ -optimality, a certain restricted sense of optimality of FSFOMs, and shows that (OGM) and (OBL-F ♭ ) are A ⋆ -optimal under a certain set of relaxed inequalities. On the other hand, A ⋆ -optimality of (OGM-G) and (OBL-G ♭ ) are presented as conjectures. Combining Theorem 1 and the A ⋆ -optimality of (OGM) and (OBL-F ♭ ) resolves these conjectures; (OGM-G) and (OBL-G ♭ ) are A ⋆ -optimal.

2.6 Intuition behind energy functions: Lagrangian Formulation

Ome might ask where the energy functions (2) and (3) came from. In this section, we provide an intuitive explanation of these energy functions using the QCQP and its Lagrangian. Consider an FSFOM (1) given with a lower triangular matrix H ∈ R N × N with function f , resulting in the sequence { x i } N i =0 . To analyze the convergence rate of the function value, we formulate the following optimization problem:

However, this optimization problem is not solvable, as f is a functional variable. To address this, [19, 55] demonstrated its equivalence to a QCQP:

where x -1 : = x ⋆ . To clarify, the optimization variables are {∇ f ( x i ) , f ( x i ) } N i =0 , f ⋆ , x 0 , and x ⋆ since { x i } N i =0 lies within span { x 0 , ∇ f ( x 0 ) , . . . , ∇ f ( x N ) } . We consider a relaxed optimization problem as [26]:

Now, consider the Lagrangian function and the convex dual.

where { a i } N -1 i =0 , { b i } N i =0 , and α are dual variables which are nonnegative. Considering a -1 = 0 and a N = 1 , the infimum of L 1 equals -∞ unless b i = a i -a i -1 for 0 ≤ i ≤ N . Therefore, by introducing u N = L 2 α and u i = a i u N for 0 ≤ i ≤ N , the convex dual problem can be simplified as follows:

If the above two constraints holds for { u i } N i =0 , we have f ( x N ) -f ⋆ ≤ L 2 u N R 2 . This understanding motivates the introduction of (2) and (C1). We can perform a similar analysis on the gradient norm minimization problem with the relaxed optimization problem as follows:

Finally, we note that although (2) and (3) both originate from the relaxed optimization problems, they have been commonly employed to achieve the convergence analysis. The function value convergence rate of OGM[26], FGM[37], G-OGM[28], GD[19], OBL-F ♭ [44] can be proved by using (2) with appropriate { u i } N i =0 . The gradient norm convergence rate of OGM-G[29], OBL-G ♭ [44], M-OGMG[60], and [17, Lemma 2.6] can be proved by using (3) with appropriate { v i } N i =0 . We also note that recent works [2, 25] do not employ (2) to achieve the convergence rate, particularly for gradient descent with varying step sizes.

3 H-duality in continuous time

We now establish a continuous-time analog of the H-duality theorem. As the continuous-time result and, especially, its proof is much simpler than its discrete-time counterpart, the results of this section serve as a vehicle to convey the key ideas more clearly. Let T > 0 be a pre-specified terminal time. Let H ( t, s ) be an appropriately integrable 2 real-valued kernel with domain { ( t, s ) | 0 < s < t < T } . We define a Continuous-time Fixed Step First Order Method (C-FSFOM) with H as

for any initial point x 0 ∈ R d and differentiable f . Note, the Euler discretization of C-FSFOMs (7) corresponds to FSFOMs (1). The notion of C-FSFOMs has been considered previously in [30].

Given a kernel H ( t, s ) , analogously define its anti-transpose as H A ( t, s ) = H ( T -s, T -t ) . We call [C-FSFOM with H A ] the H-dual of [C-FSFOM with H ]. In the special case H ( t, s ) = e γ ( s ) -γ ( t ) for some function γ ( · ) , the C-FSFOMs with H and its H-dual have the form

Interestingly, friction terms with γ ′ have time-reversed dependence between the H-duals, and this is why we refer to this phenomenon as time-reversed dissipation.

3.1 Continuous-time H-duality theorem

For the C-FSFOM with H , define the energy function

for t ∈ [0 , T ] with differentiable u : (0 , T ) → R . If u ′ ( · ) ≥ 0 , then {U ( t ) } t ∈ [0 ,T ] is dissipative. Assume we can show

Then, the C-FSFOM with H exhibits the convergence rate

For the C-FSFOM with H A , define the energy function

for t ∈ [0 , T ] with differentiable v : (0 , T ) → R . If v ′ ( · ) ≥ 0 , then {V ( t ) } t ∈ [0 ,T ] is dissipative. Assume we can show

Then, the C-FSFOM with H A exhibits the convergence rate

Theorem 2 (informal) . Consider differentiable functions u, v : (0 , T ) → R related through v ( t ) = 1 u ( T -t ) for t ∈ [0 , T ] . Assume certain regularity conditions (specified in Appendix C.2). Then,

The formal statement of 2 and its proof are given in Appendix C.2. Loosely speaking, we can consider Theorem 2 as the limit of Theorem 1 with N →∞ .

3.2 Verifying conditions for H-duality theorem

As an illustrative example, consider the case H ( t, s ) = s r t r for r ≥ 3 which corresponds to an ODE studied in the prior work [50, 51]. For the C-FSFOM with H , the choice u ( t ) = t 2 2( r -1) for the dissipative energy function {U ( t ) } T t =0 of (8) leads to

For the C-FSFOM with H A , the choice v ( t ) = 1 u ( T -t ) = 2( r -1) ( T -t ) 2 for the dissipative energy function {V ( t ) } T t =0 of (9) leads to

Since the right-hand sides are expressed as sums/integrals of squares, they are nonnegative, so (C3) and (C4) hold. (By Theorem 2, verifying (C3) implies (C4) and vice versa.) The detailed calculations are provided in Appendix C.1.

3.3 Applications of continuous-time H-duality theorem

The C-FSFOM (7) with H ( t, s ) = Cp 2 s 2 p -1 t p +1 recovers

an ODE considered in [56]. The rate f ( X ( T )) -f ⋆ ≤ 1 2 CT p ∥ X (0) -x ⋆ ∥ 2 can be established from (8) with u ( t ) = Ct p . The C-FSFOM with H A can be expressed as the ODE

By Theorem 2, using (9) with v ( t ) = 1 C ( T -t ) p leads to the rate

Note that the continuous-time models of (OGM) and (OGM-G), considered in [51], are special cases of this setup with p = 2 and C = 1 / 2 . The detailed derivation and well-definedness of the ODE are presented in Appendix C.3.

4 New method efficiently reducing gradient mapping norm: (SFG)

In this section, we introduce a novel algorithm obtained using the insights of Theorem 1. Consider minimizing F ( x ) := f ( x ) + g ( x ) , where f : R d → R is L -smooth convex with 0 < L < ∞ and g : R d → R ∪{∞} is a closed convex proper function. Write F ⋆ = inf x ∈ R n F ( x ) for the minimum value. For α > 0 , define the α -proximal gradient step as

Consider FSFOMs defined by a lower triangular matrix H = { h k,i } 0 ≤ i<k ≤ N as follows:

When g = 0 , this reduces to (1). FISTA [11], FISTA-G [31] and GFPGM [27] are instances of this FSFOM with α = 1 . In this section, we present a new method for efficiently reducing the gradient mapping norm. This method is faster than the prior state-of-the-art FISTA-G [31] by a constant factor of 5 . 28 while having substantially simpler coefficients.

Theorem 3. Consider the method

for k = 0 , . . . , N -2 , where y ⊕ , 4 = y 0 . This method exhibits the rate

We call this method Super FISTA-G (SFG), and in Appendix D.3, we present a further general parameterized family (SFG-family). To derive (SFG-family), we start with the parameterized family GFPGM [27], which exhibits an accelerated rate on function values, and expresses it as FSFOMs with H . We then obtain the FSFOMs with H A + C , where C is a lower triangular matrix satisfying certain constraints. We find that the appropriate H-dual for the composite setup is given by this H A + C , rather than H A . We provide the proof of Theorem 3 in Appendix D.2.

(SFG) is an instance of (SFG-family) with simple rational coefficients. Among the family, the optimal choice has complicated coefficients, but its rate has a leading coefficient of 46 , which is slightly smaller than the 50 of (SFG). We provide the details Appendix D.4.

5 Conclusion

In this work, we defined the notion of H-duality and formally established that the H-dual of an optimization method designed to efficiently reduce function values is another method that efficiently reduces gradient magnitude. For optimization algorithms, the notion of equivalence, whether informal or formal [59], is intuitive and standard. For optimization problems, the notion of equivalence is also standard, but the beauty of convex optimization is arguably derived from the elegant duality of optimization problems. In fact, there are many notions of duality for spaces, problems, operators, functions, sets, etc. However, the notion of duality for algorithms is something we, the authors, are unfamiliar with within the context of optimization, applied mathematics, and computer science. In our view, the significance of this work is establishing the first instance of a duality of algorithms. The idea that an optimization algorithm is an abstract mathematical object that we can take the dual of opens the door to many interesting questions. In particular, exploring for what type of algorithms the H-dual or a similar notion of duality makes sense is an interesting direction for future work.

Acknowledgments and Disclosure of Funding

We thank Soonwon Choi, G. Bruno De Luca, and Eva Silverstein for sharing their insights as physicists. We thank Jaewook J. Suh for reviewing the manuscript and providing valuable feedback. We are grateful to Jungbin Kim and Jeongwhan Lee for their valuable discussions at the outset of this work. J.K. and E.K.R. were supported by the National the Samsung Science and Technology Foundation (Project Number SSTF-BA2101-02). C.P. acknowledges support from the Xianhong Wu Fellowship, the Korea Foundation for Advanced Studies, and the Siebel Scholarship. A.O. acknowledges support from the MIT-IBM Watson AI Lab grant 027397-00196, Trustworthy AI grant 029436-00103, and the MIT-DSTA grant 031017-00016. Finally, we also thank anonymous reviewers for giving thoughtful comments.

References

K. Ahn and S. Sra. From Nesterov's estimate sequence to Riemannian acceleration. COLT , 2020.

J. M. Altschuler and P. A. Parrilo. Acceleration by stepsize hedging II: Silver stepsize schedule for smooth convex optimization. arXiv:2309.16530 , 2023.

A. Y. Aravkin, J. V. Burke, D. Drusvyatskiy, M. P. Friedlander, and K. J. MacPhee. Foundations of gauge and perspective duality. SIAM Journal on Optimization , 28(3):2406-2434, 2018.

H. Attouch and M. Théra. A general duality principle for the sum of two operators. Journal of Convex Analysis , 3:1-24, 01 1996.

J.-F. Aujol and C. Dossal. Optimal rate of convergence of an ODE associated to the fast gradient descent schemes for b > 0 . HAL Archives Ouvertes , 2017.

J.-F. Aujol, C. Dossal, G. Fort, and É. Moulines. Rates of convergence of perturbed FISTA-based algorithms. HAL Archives Ouvertes , 2019.

J.-F. Aujol, C. Dossal, and A. Rondepierre. Optimal convergence rates for Nesterov acceleration. SIAM Journal on Optimization , 29(4):3131-3153, 2019.

A. Auslender and M. Teboulle. Interior gradient and proximal methods for convex and conic optimization. SIAM Journal on Optimization , 16(3):697-725, 2006.

M. Baes. Estimate sequence methods: extensions and approximations. 2009.

N. Bansal and A. Gupta. Potential-function proofs for gradient methods. Theory of Computing , 15(4):1-32, 2019.

A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences , 2(1):183-202, 2009.

S. Boyd and L. Vandenberghe. Convex optimization . Cambridge university press, 2004.

R. E. Bruck. On the weak convergence of an ergodic iteration for the solution of variational inequalities for monotone operators in Hilbert space. Journal of Mathematical Analysis and Applications , 61(1):159-164, 1977.

P. L. Combettes and V. R. Wajs. Signal recovery by proximal forward-backward splitting. Multiscale Modeling and Simulation , 4(4):1168-1200, 2005.

G. B. Dantzig, E. Eisenberg, and R. W. Cottle. Symmetric dual nonlinear programs. Pacific Journal of Mathematics , 15(3):809-812, 1965.

I. Daubechies, M. Defrise, and C. De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics , 57(11):1413-1457, 2004.

J. Diakonikolas and P. Wang. Potential function-based framework for making the gradients small in convex and min-max optimization. SIAM Journal on Optimization , 32(3):1668-1697, 2021.

Y. Drori. The exact information-based complexity of smooth convex minimization. Journal of Complexity , 39:1-16, 2017.

Y. Drori and M. Teboulle. Performance of first-order methods for smooth convex minimization: a novel approach. Mathematical Programming , 145(1):451-482, 2014.

W. Fenchel. On conjugate convex functions. Canadian Journal of Mathematics , 1(1):73-77, 1949.

R. M. Freund. Dual gauge programs, with applications to quadratic programming and the minimum-norm problem. Mathematical Programming , 38(1):47-67, 1987.

M. P. Friedlander, I. Macê do, and T. K. Pong. Gauge optimization and duality. SIAM Journal on Optimization , 24(4):1999-2022, 2014.

B. Grimmer. Radial duality part II: Applications and algorithms. arXiv:2104.11185 , 2021.

B. Grimmer. Radial duality part I: Foundations. arXiv:2104.11179 , 2022.

B. Grimmer. Provably faster gradient descent via long steps. arXiv:2307.06324 , 2023.

D. Kim and J. A. Fessler. Optimized first-order methods for smooth convex minimization. Mathematical Programming , 159(1):81-107, 2016.

D. Kim and J. A. Fessler. Another look at the fast iterative shrinkage/thresholding algorithm (FISTA). SIAM Journal on Optimization , 28(1):223-250, 2018.

D. Kim and J. A. Fessler. Generalizing the optimized gradient method for smooth convex minimization. SIAM Journal on Optimization , 28(2):1920-1950, 2018.

D. Kim and J. A. Fessler. Optimizing the efficiency of first-order methods for decreasing the gradient of smooth convex functions. Journal of Optimization Theory and Applications , 188(1):192-219, 2021.

J. Kim and I. Yang. Unifying nesterov's accelerated gradient methods for convex and strongly convex objective functions: From continuous-time dynamics to discrete-time algorithms. ICML , 2023.

J. Lee, C. Park, and E. K. Ryu. A geometric structure of acceleration and its role in making gradients small fast. NeurIPS , 2021.

B. Li, M. Coutiño, and G. B. Giannakis. Revisit of estimate sequence for accelerated gradient methods. ICASSP , 2020.

O. L. Mangasarian and J. Ponstein. Minmax and duality in nonlinear programming. Journal of Mathematical Analysis and Applications , 11:504-518, 1965.

A. S. Nemirovski. Optimization II: Numerical methods for nonlinear continuous optimization., 1999.

A. S. Nemirovsky. On optimality of Krylov's information when solving linear operator equations. Journal of Complexity , 7(2):121-130, 1991.

A. S. Nemirovsky. Information-based complexity of linear operator equations. Journal of Complexity , 8(2):153-175, 1992.

Y. Nesterov. A method for unconstrained convex minimization problem with the rate of convergence O (1 /k 2 ) . Proceedings of the USSR Academy of Sciences , 269:543-547, 1983.

Y. Nesterov. Smooth minimization of non-smooth functions. Mathematical Programming , 103(1):127-152, 2005.

Y. Nesterov. Accelerating the cubic regularization of Newton's method on convex problems. Mathematical Programming , 112(1):159-181, 2008.

Y. Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization , 22(2):341-362, 2012.

Y. Nesterov. How to make the gradients small. Optima. Mathematical Optimization Society Newsletter , (88):10-11, 2012.

Y. Nesterov. Lectures on Convex Optimization . Springer, 2nd edition, 2018.

√

C. Park, J. Park, and E. K. Ryu. Factor2 acceleration of accelerated gradient methods. Applied Mathematics and Optimization , 2023.

C. Park and E. K. Ryu. Optimal first-order algorithms as a function of inequalities. arXiv:2110.11035 , 2021.

G. B. Passty. Ergodic convergence to a zero of the sum of monotone operators in Hilbert space. Journal of Mathematical Analysis and Applications , 72(2):383-390, 1979.

R. T. Rockafellar. Convex Analysis . Princeton university press, 1970.

R. T. Rockafellar. Conjugate Duality and Optimization . CBMS-NSF Regional Conference Series in Applied Mathematics. Society for Industrial and Applied Mathematics, 1974.

E. K. Ryu and W. Yin. Large-Scale Convex Optimization . Cambrige University Press, 2022.

J. Stoer. Duality in nonlinear programming and the minimax theorem. Numerische Mathematik , 5(1):371-379, 1963.

W. Su, S. Boyd, and E. Candes. A differential equation for modeling Nesterov's accelerated gradient method: Theory and insights. NeurIPS , 2014.

J. J. Suh, G. Roh, and E. K. Ryu. Continuous-time analysis of accelerated gradient methods via conservation laws in dilated coordinate systems. ICML , 2022.

A. B. Taylor. Convex interpolation and performance estimation of first-order methods for convex optimization. 2017.

A. B. Taylor and F. Bach. Stochastic first-order methods: non-asymptotic and computer-aided analyses via potential functions. COLT , 2019.

A. B. Taylor, J. M. Hendrickx, and F. Glineur. Exact worst-case performance of first-order methods for composite convex optimization. SIAM Journal on Optimization , 27(3):1283-1313, 2017.

A. B. Taylor, J. M. Hendrickx, and F. Glineur. Smooth strongly convex interpolation and exact worst-case performance of first-order methods. Mathematical Programming , 161(1-2):307-345, 2017.

A. Wibisono, A. C. Wilson, and M. I. Jordan. A variational perspective on accelerated methods in optimization. Proceedings of the National Academy of Sciences of the United States of America , 113(47):E7351--E7358, 2016.

P. Wolfe. A duality theorem for non-linear programming. Quarterly of applied mathematics , 19(3):239-244, 1961.

S. Yamanaka and N. Yamashita. Duality of optimization problems with gauge functions. Optimization , pages 1-31, 2022.

S. Zhao, L. Lessard, and M. Udell. An automatic system to detect equivalence between iterative algorithms. arXiv:2105.04684 , 2021.

K. Zhou, L. Tian, A. M.-C. So, and J. Cheng. Practical schemes for finding near-stationary points of convex finite-sums. AISTATS , 2022.

A Proof of Theorem 1

Reformulate (C1) and (C2) into U and V . In this paragraph, we will show

and

Recall the definition of U and V .

First we calculate U N -u N ( f ( x N ) -f ⋆ ) .

Note that all function value terms are deleted at ◦ . Therefore,

and

Since ( x 0 -x i ) , ( x i -x i +1 ) ∈ span {∇ f ( x 0 ) , . . . , ∇ f ( x N ) } , the value of U is independent with x 0 , x ⋆ . Thus the only term that depends on x 0 and x ⋆ is L 2 ∥ ∥ ∥ x 0 -x ⋆ + ∑ N i =0 u i -u i -1 L ∇ f ( x i ) ∥ ∥ ∥ 2 . Next, since x 0 , x ⋆ can have any value, we can take x 0 -x ⋆ = u i -u i -1 L ∇ f ( x i ) . Thus it gives the fact that (C1) is equivalent to [ U ≥ 0 , ( ∀∇ f ( x 0 ) , . . . , ∇ f ( x N ) ∈ R d ) ] .

Now we calculate V -1 ∥∇ f ( y ) ∥ 2 .

Note that all function values are deleted at ( ◦ ) . By the calculation result,

(C2) is equivalent to [ V ≥ 0 , ( ∀∇ f ( y 0 ) , . . . , ∇ f ( y N ) ∈ R d ) ] . To establish Theorem 1, demonstrating

would suffice.

Transforming U and V into a trace. Define

In this paragraph, we convert the U of (11) and the V of (12) into the trace of symmetric matrices. The key idea is: For each ⟨ a, b ⟩ term where a, b ∈ span {∇ f ( x 0 ) , . . . , ∇ f ( x N ) } , we can write a = g x a , b = g x b for some a , b ∈ R ( N +1) × 1 . Then

Also note that ( x 0 -x i ) , ( x i -x i +1 ) ∈ span {∇ f ( x 0 ) , . . . , ∇ f ( x N ) } and ( y i -y N ) , ( y i -y i +1 ) ∈ span {∇ f ( y 0 ) , . . . , ∇ f ( y N ) } . By using this technique, we observe that there exists S ( H,u ) , T ( H A , v ) that satisfy

From now, we specifically calculate S ( H,u ) and T ( H A , v ) . Denote { e i } N i =0 ∈ R ( N +1) × 1 as a unit vector which ( i +1) -th component is 1 , e -1 = e N +1 = 0 and define H as

Then, by the definition of g x and H , we have

Therefore, we can express (11) with H , { u i } N i =0 , g x and { e i } N i =0 using (17) and (18) as

Using (16) induces (11) = Tr( g x S ( H,u ) g ⊺ x ) where

Similarly, we calculate (12). Define H A as a anti-transpose matrix of H :

Then, by the definition of g y and H A , we have

Therefore, we can express (12) with H A , { v i } N i =0 , g and { e i } N i =0 using (20) and (21) as (12)

We can write (12) = Tr ( g y T ( H A , v ) g ⊺ y ) where

T ( H A , v )

Finding auxiliary matrix M ( u ) that gives the relation between S ( H,u ) and T ( H A , v ) . We can show that there exists an invertible M ( u ) ∈ R ( N +1) × ( N +1) such that

If we assume the above equation,

with g y = g x M ( u ) ⊺ . Since M ( u ) is invertible,

Also, note that

By combining (24) and (25), we obtain

To sum up, we obtain (15)

which concludes the proof.

Explicit form of M ( u ) and justification of (23) . Explicit form of M ( u ) is

Now, we express M ( u ) = ∑ 0 ≤ i,j ≤ N m ij ( u ) e i e ⊺ j , S ( H,u ) = ∑ 0 ≤ i,j ≤ N s ij e i e ⊺ j , and T ( H A , v ) = ∑ 0 ≤ i,j ≤ N t ij e i e ⊺ j . Calculating M ⊺ ( u ) T ( H A , v ) M ( u ) gives

Thus it is enough to show that ∑ i,j s ij e i e ⊺ j and ∑ i,j t ij f i ( u ) f j ( u ) ⊺ are the same under the basis transformation f i ( u ) = ∑ k m ik ( u ) e k . From here, we briefly write f i instead f i ( u ) , and f -1 = 0 . Note that u i ( e i -e i +1 ) = ( f N -i -f N -i -1 ) , 0 ≤ i ≤ N by definition of M ( u ) . Therefore, we have

Therefore, we can rewrite (19) as follows:

Similarly, by using

we can rewrite (22) as follows:

For the final step, we compare (28) and (27) term-by-term, by showing X 1 = X 2 for X = A , B , C , D .

A 1 = A 2 comes directly.

B 1 = B 2 comes from changing the summation index i → N -i and j → N -j .

C 1 = C 2 comes from the expansion of summation.

D 1 = D 2 comes from f 0 = u N e N .

Therefore, which concludes the proof.

Remark 1. We can interpret M as a basis transformation, where

Remark 2. To clarify, the quantifier [ ∀∇ f ( x 0 ) , . . . , ∇ f ( x N )] in (C1) means ∇ f ( x 0 ) , . . . , ∇ f ( x N ) can be any arbitrary vectors in R d . This is different from [ ∇ f ( x 0 ) , . . . , ∇ f ( x N ) be gradient of some f : R d → R ] . The same is true for (C2).

B Omitted calculation of Section 2

B.1 Calculation of H matrices

OGMand OGM-G. [26, Proposition 3] provides a recursive formula for (OGM) as follows:

If k > i ,

Thus H OGM can be calculated as

Recursive formula [29] of (OGM-G) is as following:

If k > i ,

Thus H OGM-G can be calculated as

which gives H OGM-G = H A OGM .

Gradient Descent. For (GD), H ( i +1 , k +1) = h i +1 ,k = hδ i +1 ,k +1 , where δ i,j is the Kronecker delta. Therefore, H GD = H A GD .

OBL-F ♭ and OBL-G ♭ . Recall γ = √ N ( N +1) 2 . We obtain the recursive formula of the H matrix of (OBL-F ♭ ).

By using the above formula, we obtain

Similarly, we achieve the following recursive formula of the H matrix of (OBL-G ♭ ).

By using the above recursive formula, we obtain

Thus H OBL-F ♭ = H A OBL-G ♭ .

B.2 Calculation of energy functions

Calculation of U and V with H matrix In this paragraph, we calculate U and V . Recall (13) and (14).

First, we put x k +1 -x k = -1 L k ∑ i =0 h k +1 ,i ∇ f ( x i ) to (11). We have

By arranging, we obtain where

.

Recall that we defined u -1 = 0 . Next, put y k +1 -y k = -1 L k ∑ i =0 h k +1 ,i ∇ f ( y i ) to (12). We have

By arranging, we obtain where

Now we calculate { s ij } and { t ij } for [ (OGM) , (OGM-G) ] [ (OBL-F ♭ ) , (OBL-G ♭ ) ] and [ (GD) , (GD) ] .

B.2.1 Calculation of energy function of (OGM) and (OGM-G)

We will show s ij = 0 and t ij = 0 for all i, j . By the definition of { u i } N i = -1 and { θ i } N i = -1 ,

Therefore, we have

.

̸

Now we claim that s ij = 0 when j = i . In the case that i = j +1 , we have

̸

We show s ij = 0 for j = i with induction on i , i.e., proving

First we prove (36) for i = j +2 .

Next, assume (36) for i = i 0 . When i = i 0 +1 ,

where the second equality comes from the induction hypothesis, and the third equality comes from (30). In sum, we proved s ij = 0 for every i and j , which implies U = 0 .

Next, we will claim that t ij = 0 for all i, j . Firstly, explicit formula of H OGM-G ( k +1 , i +1) first. When k > i ,

To calculate { t i,j } , it is enough to deal with the sum N -1 ∑ l = i h l +1 ,j , which can be expressed as

By inserting (37) in (35), [ t ij = 0 , ∀ i, j ] is obtained, which implies V = 0 . (37) and (35) are also stated in [29, Lemma 6.1].

B.2.2 Calculation of energy function of (OBL-F ♭ ) and (OBL-G ♭ )

First we calculate { s ij } for (OBL-F ♭ ). Recall u i = ( i +1)( i +2) 2 for 0 ≤ i ≤ N -1 and u N = γ 2 + γ where γ = √ N ( N +1) / 2 . When j = i ,

̸

Now we claim that s ij = 0 when j = i . In the case j = i -1 , we have

̸

We show s ij = 0 for j = i with induction on i , i.e., proving

(38) holds when i = j +2 since

.

Assume (38) for i = i 0 . For i = i 0 +1 ,

Next, we calculate { t ij } for (OBL-G ♭ ). We need to deal with the sum N -1 ∑ l = k h l +1 ,i , which can be expressed as

By combining v 0 = 1 γ 2 + γ , v i = 1 ( N -i +1)( N -i +2) for 1 ≤ i ≤ N and (35), we obtain

Therefore,

B.2.3 Calculation of energy function of GD

We calculate { t ij } first. Recall that { v i } N i =0 = ( 1 2 N +1 , . . . , N + i (2 N +1)( N -i +1) , . . . ) and h i +1 ,k = δ i,k to (35), and making { t ij } symmetric gives us

̸

̸

We can verify that the matrix { t ij } 0 ≤ i,j ≤ N is diagonally dominant: t ii = | ∑ j = i t ij | . Therefore, ∑ 0 ≤ i,j ≤ N -1 t ij L ⟨∇ f ( y i ) , ∇ f ( y j ) ⟩ ≥ 0 for any {∇ f ( y i ) } N i =0 . This proof is essentially the same as the proof in [29], but we repeat it here with our notation for the sake of completeness.

Next, we prove that (GD) with h = 1 and { u i } N i =0 = ( . . . , (2 N +1)( i +1) 2 N -i , . . . , 2 N +1 ) satisfies (C1), by showing more general statement:

Note that (39) gives

Later in the proof of Corollary 2, we will utilize the equation (39). The result (39) is proved in [19, Theorem 3.1], and we give the proof outline here.

In order to demonstrate (39), we will directly expand the expression U N -u N ( f ( x N ) -f ⋆ ) , instead of employing U as a intermediary step. Define { u ′ i } N i =0 : = ( . . . , (2 N +1)( i +1) 2 N -i , . . . , 2 N +1 ) . Then

where g = [ ∇ f ( x 0 ) | . . . |∇ f ( x N ) | L ( x 0 -x ⋆ )] ∈ R d × ( N +2) and S ∈ S N +2

Now we will show S ⪰ 0 to obtain [ U N -u N ( f ( x N ) -f ⋆ ) ≥ 0 , ∀ g ] , which is (C1). By using Sylvester's Criterion, S 0 ≻ 0 follows. S 1 ≻ 0 follows from the fact that S 1 expressed by the sum of positive semi-definite matrices zz ⊺ . Since the convex sum of two positive semi-definite matrices is also positive semi-definite, S ′ = hS 0 +(1 -h ) S 1 ≻ 0 .

Next, we argue that det S = 0 . Indeed, take τ = (1 , . . . , -(2 Nh +1)) ⊺ to show Sτ = 0 , which gives det S = 0 . Note that the determinant of S can also be expressed by

We have shown that S ′ ≻ 0 , (41) implies 1 2 -λ ⊺ ( S ′ ) -1 λ = 0 , which is the Schur complement of the matrix S . By a well-known lemma on the Schur complement, we conclude S ⪰ 0 .

B.3 Omitted proof in Section 2.5

B.3.1 Omitted calculation of Corollary 1

Here, we will give the general formulation of H-dual FSFOM of

Proposition 1. The H-dual of (42) is

where

̸

for k = 0 , . . . , N -1 and ( β N , γ N ) is any value that β N + γ N = 0 . 3

Proof. The H matrix { h k,i } 0 ≤ i<k ≤ N satisfies

Therefore,

where δ k,i is a Kronecker Delta function. Similarly, H matrix of (43) { g k,i } 0 ≤ i<k ≤ N satisfies

Thus g k +1 ,i = h N -i,N -1 -k .

Now we derive the H-dual of (6) by applying Proposition 1. Note that

Next, define β N and γ N as a same form of { β i , γ i } N -1 i =0 with any t N +1 > 0 . Note that

By applying the formula at Proposition 1, we obtain

and

3 Here, note that FSFOM (43) is independent with the any choice of β N , γ N since y 1 = y 0 + ( β ′ 0 + γ ′ 0 ) ( y + 0 -y 0 ) = y 0 +( β N -1 + γ N -1 ) ( y + 0 -y 0 ) .

B.3.2 Proof of Corollary 2

First, recall (39) when 0 < h ≤ 1 : [

Additionally, observe that the H matrix of (GD) is diag ( h, . . . , h ) , which gives the fact that the H-dual of (GD) is itself.

Next, use Theorem 1 to obtain

By using the same argument with (5), (44) gives

In addition, we can achieve the convergence rate of the gradient norm under the initial condition of ∥ x 0 -x ⋆ ∥ 2 :

and

The first inequality comes from (45) and the second inequality comes from (40).

B.3.3 Proof of A ⋆ -optimality of (OGM-G) and (OBL-G ♭ )

Definition of A ⋆ -optimal FSFOM. For the given inequality sets L , A ⋆ -optimal FSFOM with respect to [ L , P1 ] is defined as a FSFOM which H matrix is the solution of following minimax problem:

Similarly, A ⋆ -optimal FSFOM with respect to [ L , ( P 2)] is specified with its H matrix, which is the solution of the following minimax problem:

Here R N × N L is the set of lower triangular matrices. Next we denote the inner maximize problem of (46) and (47) as P 1 ( L , H, R ) and P 2 ( L , H, R ) , respectively. For a more rigorous definition of A ⋆ -optimal FSFOM, refer [44].

Remark. We discuss the minimax problems (46) and (47) and their interpretation. Specifically, we consider the meaning of the maximization problem in these minimax problems, which can be thought of as calculating the worst-case performance for a fixed FSFOM with H . In other words, the minimization problem in (46) and (47) can be interpreted as determining the optimal value of H that minimizes the worst-case performance.

Prior works. The prior works for A ⋆ -optimality are summarized as follows. Consider the following sets of inequalities.

and

(OGM) is A ⋆ -optimal with respect to [ L F , (P1) ] [26]. Furthermore, (OGM) is also A ⋆ -optimal under [ L exact , (P1) ] which implies that (OGM) is the exact optimal FSFOM with respect to (P1). In addition, the A ⋆ -optimality of (OGM-G) with respect to [ L G , (P2) ] was presented as a conjecture in [29], (OBL-F ♭ ) is A ⋆ -optimal with respect to [ L F ′ , ( P 1)] [44, Theorem 4], and the A ⋆ -optimality of (OBL-G ♭ ) with respect to [ L G ′ , (P2) ] was presented as a conjecture [44, Conjecture 8]. 4

In the remaining parts, we give the proof of the following two theorems.

Theorem 4. (OGM-G) is A ⋆ -optimal with respect to [ L G , (P2) ] .

Theorem 5. (OBL-G ♭ ) is is A ⋆ -optimal with respect to [ L G ′ , (P2) ] .

Proof of A ⋆ -optimality of (OGM-G) We provide an alternative formulation of P 2 ( L G , H, R ) by using the methodology called PEP [19, 52]

Next, define F , G , { e i } N i =0 and { x i } N i =0 as

e i ∈ R N +1 is a unit vector which ( i +1) -th component is 1 , and

By using the definition of F , G , { e i } N i =0 and { y i } N i =0 , P 2 ( L G , H, R ) can be converted into

where

Moreover, under the condition d ≥ N +2 , we can take the Cholesky factorization of G to recover the triplet { ( y i , f ( y i ) , ∇ f ( y i )) } N i =0 . 5 Thus (48) and (49) are equivalent. The next step is calculating the Lagrangian of (49) and deriving the Lagrangian dual problem of it. In [54], they argued about the strong duality of (49).

.

where

If [ X = 0 and T ⪰ 0] is false, we can choose F and G that makes the value of Lagrangian to be -∞ . Thus the convex dual problem of (49) is

For the constraint X = 0 , λ i = δ i -δ i -1 for 1 ≤ i ≤ N -1 , τ = δ N and -δ 0 + λ 0 + δ N = 0 . By substituting v i +1 = δ i for 0 ≤ i ≤ N -1 and v 0 = δ N , (50) becomes

Therefore if strong duality holds, P 2 ( L G , H, R ) becomes

We can apply a similar argument for P 1 ( L F , H, R ) . To begin with, define { f i } N i = -1 as a unit vector of length N +2 which ( i +2) -component is 1 . Additionally, define { x i } N i =0 , { C i } N -1 i =0 and { D i } N i =0 as follows:

Then, if the strong duality holds, the problem

is equivalent to

̸

Fact 1. Assume h i +1 ,i = 0 for 0 ≤ i ≤ N -1 . Then the strong duality holds between (48) and (49).

Denote the dual variables of each constraints as { δ i } N -1 i =0 , { λ i } N -1 i =0 , δ N and τ . Then Lagrangian becomes

where

Therefore, we obtain

if and only if

For the next step, we claim that the optimal value of (56) and

̸

are same, i.e., consider the case when all u i are positive is enough. To prove that, assume there exists 0 = u 0 = · · · = u k , 0 < u k +1 ≤ · · · ≤ u N and H that satisfy S ′ ⪰ 0 . Next, observe that the f k f ⊺ k component of S ′ is 0 but f k +1 f ⊺ k component of S ′ is u k +1 h k +1 ,k = 0 , which makes S ′ ⪰ 0 impossible.

When the optimal value of (52) is 0 , it implies { ( v 0 , . . . , v N ) | T ⪰ 0 , 0 < v 0 ≤ · · · ≤ v N } is an empty set. Therefore, { ( u 0 , . . . , u N ) | S ′ ⪰ 0 , 0 < u 0 ≤ · · · ≤ u N } is also empty set. Since (59) and (56) have the same optimal value, the optimal value of (56) is 0 .

If the optimal value of (52) is positive, the optimal values of (56) and (52) are the same since (57) and (58) are equivalent.

By using Schur's Complement, where

Hence (55) is equivalent to

Now we will prove the following proposition.

Proposition 2. Consider a matrix H = { h i,j } 0 ≤ j<i ≤ N and H A . If h i +1 ,i = 0 for 0 ≤ i ≤ N -1 and treating the solution of infeasible maximize problem as 0 , the optimal values of P 1 ( L F , H, R ) and P 2 ( L G , H A , R ) are same.

̸

Proof. To simplify the analysis, we can consider { f i } N i =0 as a length N +1 unit vector, as all terms with f -1 can be eliminated by using S ′ instead of S . With this simplification, both S ′ and T belong to S N +1 .

Next, let 0 < u 0 ≤ u 1 ≤ ·· · ≤ u N and v i = 1 u N -i for 0 ≤ i ≤ N , noting that 0 < v 0 ≤ · · · ≤ v N . It is important to observe that S ′ and T can be expressed in terms of S ( H,u ) and T ( H A , v ) , respectively. Furthermore, in the proof of Theorem 1 (A), we proved that S ( H,u ) = M ( u ) ⊺ T ( H A , v ) M ( u ) for some invertible M ( u ) . Thus,

Proof of Theorem 4. H OGM is the solution of (46) since (OGM) is A ⋆ -optimal with respect to L F and (P1). Additionally, if

̸

holds, H OGM is the solution to the following problem due to the strong duality.

̸

Applying Proposition 2 and using the fact H OGM-G = H A OGM provides that H OGM-G is the solution of

̸

̸

holds, the optimal solution of (47) is the H OGM-G, which proves the A ⋆ -optimality of (OGM-G) with respect to L G and (P2). The proof of (60) and (63) uses the continuity argument with H and please refer [44, Claim 4].

Remark of Proof of A ⋆ -optimality of (OGM-G) . We proved that (OGM-G) is A ⋆ -optimal if we use the subset of cocoercivity inequalities. Therefore, it is still open whether (OGM-G) is optimal or not among the L -smooth convex function's gradient minimization method.

Proof of A ⋆ -optimality of (OBL-G ♭ ) . We provide the proof that the H matrix of (OBL-G ♭ ) is the solution of

To begin with, bring to mind the A ⋆ -optimality of (OBL-F ♭ ): (OBL-F ♭ ) is A ⋆ -optimal with respect to the [ L F ′ , P 1] , i.e., the H matrix of (OBL-F ♭ ) is the solution of

To prove the A ⋆ -optimality of (OBL-G ♭ ), we use the A ⋆ -optimality of (OBL-F ♭ ).

Under the assumption of strong duality, we could change P 1 ( L F' , H, R ) into the following SDP:

Finally, if where

Here we used the same notation with (53). Each 1 2 L f i f ⊺ i term is subtracted from the original S since we consider the inequality J x i , x ⋆ K -1 2 L ∥∇ f ( x i ) ∥ 2 instead of J x i , x ⋆ K for L F ′ . Moreover, (66) is equivalent to

where

Similarly, under the assumption of strong duality, P 2 ( L G' , H, R ) is equivalent to

where

Now we will prove the following proposition.

Proposition 3. Consider a matrix H = { h k,i } 0 ≤ i<k ≤ N and H A . If h i +1 ,i = 0 for 0 ≤ i ≤ N -1 and treating the solution of infeasible maximize problem as 0 , the optimal values of P 1 ( L F' , H, R ) and P 2 ( L G' , H A , R ) are same.

̸

Proof. The proof structure is the same as the proof of Proposition 2. First consider { f i } N i =0 as length N +1 , which gives S ′ 1 , T 1 ∈ S N +1 . Furthermore, in the proof of Appendix A, we proved that

The other steps are the same as the proof of Proposition 2.

Proof of Theorem 5. H OGM is the solution of (46) since (OGM) is A ⋆ -optimal with respect to L F and (P1). Additionally, if

̸

holds, H OGM is the solution to the following problem due to the strong duality.

̸

Applying Proposition 2 and using the fact H OGM-G = H A OGM provides that H OGM-G is the solution of

̸

̸

holds, the optimal solution of (47) is the H OGM-G, which proves the A ⋆ -optimality of (OGM-G) with respect to L G and (P2). The proof of (69) and (72) uses the continuity argument with H and please refer [44, Claim 4].

Therefore,

Finally, if

C Omitted parts in Section 3

C.1 Omitted calculations in Section 3.1

Strictly speaking, (7) is not a differential equation but rather a diffeo-integral equation. However, if H is separable, i.e., when H ( t, s ) = e β ( s ) -γ ( t ) for some β, γ : [0 , T ) → R , then (7) can be reformulated as an ODE.

Assume the process (7) is well-defined. We can alternatively write (7) as

By multiplying e γ ( t ) each side and differentiating, we obtain

The H-dual of (73) is

Under the well-definedness, by multiplying e -β ( T -t ) each side and differentiating, we obtain

When β ( s ) = γ ( t ) = r log t , two ODEs become

which are introduced in Section 3.2.

For the calculations of energy functions of (74) and (75), refer [51, Section 3.1, Section 4.2]. They proved that

holds for (74) and

holds for (75). After arranging terms, we obtain the results in Section 3.2. 6

C.2 Proof of Theorem 2

To begin with, we give a formal version of Theorem 2. Consider two following conditions.

where A 1 and A 2 are family of vectors which makes Fubini's Theorem can be applied and will be defined later in this section.

and

Theorem 6 (Formal version of Theorem 2) . Assume the C-FSFOMs (7) with H and H A are welldefined in the sense that solutions to the diffeo-integral equations exist. Consider differentiable functions u, v : (0 , T ) → R that v ( t ) = 1 u ( T -t ) for t ∈ [0 , T ] and

(i) lim s → 0 u ( t ) = 0

(iii) f is L -smooth and convex.

Then the following holds.

Calculations of energy functions via transformation Frist of all, we calculate U ( T ) -u ( T )( f ( X ( T )) -f ⋆ ) .

We used parts of integration and u (0): = lim s → 0 u ( s ) = 0 . Since X (0) -x ⋆ can have any value, (C3) is equivalent to

holds for any {∇ f ( X ( s )) } s ∈ [0 ,T ] .

Now define a transformation { f s ∈ R d } s ∈ [0 ,T ] →{ g s ∈ R d } s ∈ [0 ,T ] as

and { g s ∈ R d } s ∈ [0 ,T ] →{ f s ∈ R d } s ∈ [0 ,T ] .

One can show that the above two transformations are in the inverse relationship. Next, we calculate U ( T ) . Define f s : = ∇ f ( X ( s )) . Under the transformation (77), we can find a simple expression of

(76).

We used Fubini's Theorem at ( ◦ ) . Next we calculate V ( T ) .

( ◦ ) comes from parts of integration and the assumption

To clarify, u ′ ( T -s ) = u ′ ( z ) | z = T -s . Now briefly write g s : = ∇ f ( Y ( T -s )) . Then

We use Fubini's Theorem at ( ◦ ) . Finally, using (78), we obtain

Proof of Theorem 2 Define A 1 and A 2 as follows.

A 1 = {{ f s } ∈ | Analysis in the previous paragraph holds }

A 2 = {{ g s } s ∈ [0 ,T ] | Analysis in the previous paragraph holds }

Now we prove Theorem 2. We have shown that (C3 ′ ) is equivalent to (76) ≥ 0 for all {∇ f ( X ( s )) } s ∈ [0 ,T ] ∈ A 1 . By definition of A 1 , it is equivalent to

for any { g s } s ∈ [0 ,T ] ∈ A 2 . Moreover, by (81), it is also equivalent to

for any { g s } s ∈ [0 ,T ] ∈ A 2 , which is (C4 ′ ).

C.3 Omitted parts in Section 3.3

Regularity of (10) at t = -T . To begin with, note that ODE (10) can be expressed as

for t ∈ (0 , T ) . Since right hand sides are Lipschitz continuous with respect to W and Y in any closed interval [0 , s ] ∈ [0 , T ) , solution ( Y, W ) uniquely exists that satisfies above ODE with initial condition ( Y (0) , W (0)) = ( y 0 , 0) . Next, we give the proof of regularity of ODE (10) at terminal time T in the following order. The proof structure is based on the regularity proof in [51]:

(i) sup t ∈ [0 ,T ) ∥ ∥ ∥ ˙ Y ( t ) ∥ ∥ ∥ is bounded

(ii) Y ( t ) can be continuously extended to T

(iii) lim t → T -∥ ∥ ∥ ˙ Y ( t ) ∥ ∥ ∥ = 0

(iv) lim t → T -˙ Y ( t ) ( T -t ) p -1 = Cp ∇ f ( Y ( T )) .

By multiplying ˙ Y ( s ) and integrating from 0 to t , we obtain

and integration by parts gives us

1

˙

∥

∥

2

2(

t

)

-

∥

∥

p

2

-

Y

(

t

)

-

∥

∥

T

2

T

Define Ψ( t ): [0 , T ) → R as

Since

Ψ( t ) is a nonincreasing function. Thus

and the right hand side of (83) is constant, which implies M := sup t ∈ [0 ,T ) ∥ ∥ ∥ ˙ Y ( t ) ∥ ∥ ∥ < ∞ .

Step (ii): Y ( t ) can be continuously extended to T . We can prove Y ( t ) is uniformly continuous due to the following analysis.

Since a uniformly continuous function g : D → R d can be extended continuously to D , Y : [0 , T ) → R d can be extended to [0 , T ] .

Step (iii): lim t → T -∥ ∥ ∥ ˙ Y ( t ) ∥ ∥ ∥ = 0 . We first prove the limit lim t → T -∥ ∥ ∥ ˙ Y ( t ) ∥ ∥ ∥ = 0 exists. From C.3, we know lim t → T -Y ( t ) exists and by continuity of f , lim t → T -f ( Y ( t )) also exists. Moreover, Ψ( t ) is non-increasing and

thus lim t → T -Ψ( t ) exists. Therefore, lim t → T -∥ ˙ Y ( t ) ∥ 2 ( T -t ) p -2 exists, which implies lim t → T -∥ ∥ ∥ ˙ Y ( t ) ∥ ∥ ∥ = 0 when p > 2 . For the case p = 2 , lim t → T -∥ ∥ ∥ ˙ Y ( t ) ∥ ∥ ∥ exists, and lim t → T -∥ ∥ ∥ ˙ Y ( t ) ∥ ∥ ∥ = 0 follows since we have

is bounded below. Thus the value of integration is finite.

Step (iv): lim t → T -˙ Y ( t ) ( T -t ) p -1 = Cp ∇ f ( Y ( T )) . The (7) form of process Y is

1

p

-

2

∥

∥

∥

˙

Y

(0)

∥

∥

∥

2

+

∫

0

t

(

T

2

p

-

-

s

)

1

p

-

1

∥

∥

∥

˙

Y

(

s

)

∥

∥

∥

2

ds

+

2

Cp

(

f

(

Y

(

t

))

-

f

(

Y

(0))) = 0

.

By dividing ( T -t ) p -1 each side, we obtain

By the result of C.3, we can apply L'Hopital's rule, which gives

Also, since ∇ f is L -l=Lipshitz, ∥∇ f ( Y ( t )) -∇ f ( Y ( T )) ∥ ≤ L ∥ Y ( t ) -Y ( T ) ∥ . Therefore,

for any β < p .

Applying Theorem 2 to (10) . For the case p = 2 , refer [51]. Now we consider the case p > 2 , with u ( t ) = Ct p and v ( t ) = 1 C ( T -t ) p . First, we verify conditions (i) and (ii) in Theorem 6. (i) holds since u (0) = 0 , and (ii) holds since

( ◦ ) uses L'Hospital's rule and ( · ) uses the limit results at the previous paragraph.

To prove

we carefully check that for any L -smooth convex function f , {∇ f ( Y ( s )) } s ∈ [0 ,T ] ∈ A 2 .

Verification of (80) . Fubini's Theorem is used for

The above function is continuous in 0 ≤ s, a ≤ T due to lim t → T -˙ Y ( t ) ( T -t ) p -1 = Cp ∇ f ( Y ( T )) and (84) with β = 2 < p .

Verification of (79) . First,

holds from (iv). Thus sup b ∈ (0 ,T ] ∥ ∥ ∥ Y ( T -b ) -Y ( T ) b p ∥ ∥ ∥ < ∞ . Now we will show

Observe and

By using the boundness of Y ( T -b ) -Y ( T ) b p , we obtain the desired result. Next,

which gives

For the final step, recall that Fubini's Theorem is used for

To prove that the above function is continuous, take any 0 < ϵ < p -2 2 . Then

D Omitted calculation in Section 4

D.1 Preliminaries

General proximal step and composite FSFOM For a > 0 , we newly define a prox-grad step with step size 1 L 1 α as follows:

First, we will provide a generalized version of prox-grad inequality; Prox-grad inequality is originally used to prove the convergence of FISTA as below [11]:

Proposition 4 (Generalized version of prox-grad inequality) . For every x, y ∈ R d , the following holds:

Proof. The optimality condition of (85) provides

In addition, L -smoothness of f and convexity inequality of f and g give

Summing the above inequalities provides the generalized version of prox-grad inequality.

Two problem settings for the composite optimization problem. For the composite optimization problem, we consider the following two problems.

(P1 ′ ) Efficiently reduce F ( x ⊕ ,α N ) -F ⋆ assuming x ⋆ exists and ∥ x 0 -x ⋆ ∥ ≤ R .

Note that when g = 0 , (P1 ′ ) and (P2 ′ ) collapse to (P1) and (P2), respectively.

Parameterized FSFOM that reduces the composite function value : GFPGM [27] First define x ⋆ : = argmin x ∈ R d F ( x ) . We recall the composite function minimization FSFOMs with x ⊕ ,α i by a lower triangular matrix { h k,i } 0 ≤ i<k ≤ N as follows:

In the case g = 0 , (87) collapses to (1) as α ( z -z ⊕ ,α ) = ( z -z + ) = 1 L ∇ f ( z ) . The iterations of GFGPM are defined as

where x ⊕ , 1 -1 = x 0 and [ t i > 0 , T i = ∑ i j =0 t j ≤ t 2 i for 0 ≤ i ≤ N ] . (88) reduces the composite function value as follows:

We can reconstruct the convergence proof of [27, Theorem 3.3] via our energy function scheme, which is defined as

for k = -1 , . . . , N . The convergence rate result (89) is proved by using that {U k ] N k = -1 is dissipative, and

Here, we can make a crucial observation.

Moreover, we can express the recursive formula of the H matrix of (88) as

As we used a parameterized family in the convex function (see Section 2.5), we will use these convergence results to construct the method for minimizing min v ∈ ∂F ( y ⊕ ,α N ) ∥ v ∥ 2 .

Relationship between minimizing α 2 2 L ∥ ∥ y ⊕ ,α N -y N ∥ ∥ 2 and minimizing min v ∈ ∂F ( y ⊕ ,α N ) ∥ v ∥ 2 . Note that Lα ( y ⊕ ,α N -y N ) + ∇ f ( y N ) + u = 0 , u ∈ ∂g ( y ⊕ ,α N ) , which gives

( · ) is triangle inequality and ( ◦ ) comes from the L -smoothness of f .

D.2 Proof of Theorem 3

In this section, we give the proof outline of Theorem 3 and discuss the construction of matrix C .

Proof outline of Theorem 3. We begin by proposing a parameterized family reduces 1 2 L ∥ ∥ y ⊕ ,α N -y N ∥ ∥ 2 under a fixed value of a > 0 . To construct this family, take { t i } N i =0 , T i = ∑ i j =0 t j that satisfies (96). We define the H matrix of the family as ( 1 α H 0 + 1 α 2 C ) A , where H 0 has the same formulation as the H matrix of GFGPM (88) and C follows the recursive formula (97). We refer to this family as (SFG-family).

We can prove that (SFG-family) exhibits the convergence rate

which is motivated by the observation ( ⋄ ). In parallel with ( ⋄ ), we consider the energy function

for k = 0 , . . . , N , which J y N , ⋆ K changed into L y -1 , y 0 M α , all other J · , · K terms changed into L · , · M α and v i = 1 T N -i .

(94) follows from the fact that {V i } N i =0 is dissipative and

Detailed justification is given in Appendix D.3.

For the next step, we show that (SFG) is an instance of (SFG-family), under the choice of

The detailed derivation is given in Appendix D.4.

Finally, combining the convergence result

and (93) gives

where e -1 = 0 , e i is a unit vector which ( i + 1) -th component is 1 and R N +1 and H 1 : = [ 0 0 H 1 0 ] . If we take H 1 that makes T ⊕ ⪰ 0 , V N ≥ αL 2 ∥ y ⊕ ,α N -y N ∥ 2 follows. Next we observe 1 α 2 A 3 + 1 α B 3 = T ( H 1 , v ) for v i = 1 T N -i which is defined as (22). By expanding (91) and defining g ′ : = L [ x 0 -x ⊕ ,α 0 | . . . | x N -x ⊕ ,α N ] ,

follows where

Here, u i = T i , and S ( H 0 , u ) is given by the formulation (19). Using (91), we obtain

Now recall matrix M ( u ) (26) and the result of Theorem 1:

By substituting H 1 = 1 H A + X and using (95), the result is

Next, consider A 4 as a function of the lower triangular matrix X . The key observation is that if we choose X appropriately, all non-diagonal terms of C 3 can be eliminated. With this choice of X , we have

Note that T ⊕ contains only diagonal terms. Therefore, T ⊕ ⪰ 0 is equivalent to all coefficients of e i e ⊺ i being nonnegative, which can be formulated using t i and T i .

Remark. In the proof of Theorem 1 ( B 2 and B 1 at (27) and (28)), we have shown that

Observe that ( e i -e i +1 )( e 0 + · · · + e i ) ⊺ is a lower triangular matrix and X A is a strictly lower triangular matrix, i.e., all diagonal component is 0 . Thus it is worth noting that A 4 cannot induce any diagonal term e i e ⊺ i , choosing matrix C as the optimal one.

D.3 Parameterized family reduces the gradient mapping norm

FSFOM that reduces 1 2 L ∥ ∥ y ⊕ ,α N -y N ∥ ∥ 2 in the composite minimization. We propose the parameterized family that reduces 1 2 L ∥ ∥ y ⊕ ,α N -y N ∥ ∥ 2 . For fixed a > 0 , take { t i } N i =0 , T i = ∑ i j =0 t j that satisfies the following conditions:

where we define ∑ k -2 i =0 1 T i = 0 for k = 1 case. We define an lower triangular matrix C = { c k,i } 0 ≤ i<k ≤ N as

Matrix C has a crucial role in the correspondence between composite function value minimization and composite function gradient norm minimization with H → ( 1 α H + 1 α 2 C ) A .

Proposition 5 (SFG family) . Assume that we choose H 0 matrix of (88), and define C as (97). Consider an FSFOM (87) which H matrix is ( 1 α H 0 + 1 α 2 C ) A . Such FSFOM can be expressed as

and y ⊕ ,α -1 = y 0 . Then it exhibits the convergence rate

Proof of Proposition 5. To start, we give the claim about matrix C .

Claim 1. C : = [ 0 0 C 0 ] satisfies the following equality.

where { e i } N i =0 is a any basis of R N +1 , e N +1 , f -1 are zero vectors and { f i } N i =0 are another basis of R N +1 which is defined as

The proof of Claim 1 will be provided after the proof of Proposition 5. To begin with, for the FSFOM with matrix ( 1 α H 0 + 1 α 2 C ) A , define the energy function

for k = 0 , . . . , N . {V k } N k =0 is dissipative since L · , · M α ≤ 0 . This energy function is inspired by (3). Note that if we can prove

then

Thus, it is enough to show (99). Defining g i : = Lα ( y i -y ⊕ ,α i ) for 0 ≤ i ≤ N gives

Plugging the above equalities provides

By the same procedure as the proof of Theorem 1, we obtain 2 Lα 2 ( V N -αL 2 ∥ ∥ y ⊕ ,α N -y N ∥ ∥ 2 ) = Tr( g ⊺ g T ⊕ ) where

And consider the vector e i ∈ R ( N +1) × 1 which is same with (29). In the proof of Theorem 1 ( (27) , (28) ) , we have shown that

and

under the above transformation. Therefore, T ⊕ can be expressed as

where

and

To calculate A , expand the energy function (91). Then we obtain

Combining above results and T N e N = f 0 , we achieve

and the condition (96) makes each coefficient of e i e ⊺ i nonnegative, which gives T ⪰ 0 . To achieve the iteration formula (SFG-family), consider the FSFOM (87) with 1 α H 0 + 1 α 2 C , which can be

Moreover, by Claim 1,

expressed as

For last step, We make an observation that Proposition 1 can be applied when all z + terms changed into z ⊕ ,α . Proposition 1 gives the H-dual.

Proof of Claim 1. The left hand side of (98) is

Now we calculate the right-hand side of (98). By plugging

where

Now we claim that a i = ∑ N k = i +1 c k,i f N -k for i = 0 , . . . , N -1 . Note that

Coefficients of e i +1 are coincides since

c

1

,

0

T

1

=

t

1

,

Now assume

By multiplying the above equation recursively, we obtain

for 1 ≤ j ≤ N -1 . which implies a i = ∑ N k = i +1 c k,i f N -k .

To prove (100), expand and obtain

Above equation holds due to the recursive formula of C (97).

D.4 Instances of SFG family

Derivation of (SFG) Here, we will show that (SFG) is the instance of SFG family, under the choice of

First, t i = i +2 2 for i ≥ 1 and t 0 = 3 2 . Also (96) holds since

Plugging above values into (92) and (97), we obtain

and

Other terms come directly, and

.

To sum up, the matrix { g k,i } 0 ≤ i<k ≤ N : = 1 4 H 0 + 1 16 C satisfies the following recursive formula.

Note that

Therefore, the FSFOM with { g k,i } 0 ≤ i<k ≤ N can be expressed as

where x ⊕ , 4 -1 = x 0 . To obtain the H-dual, we apply Proposition 1.

where k = 0 , . . . , N -2 and y ⊕ , 4 -1 = y 0 .

Fastest method among the SFG family via a numerical choice of a In this section, we give simple form of SFG, when all inequality conditions in (96) holds as equalities. { g k,i } 0 ≤ i<k ≤ N becomes

and for k > 0 ,

Hence FSFOM with 1 α H 0 + 1 α 2 C is

where T -1 = 2 a -1 α 2 . By using Proposition 1, we obtain H-dual.

Since all equality holds at (96), the above FSFOM achieves the fastest rate among the SFG family under fixed α . 7 Now we optimize a to achieve the fastest convergence rate. Combine (93) and the result of Proposition 5 to obtain

To achieve the tightest convergence guarantee, we solve the following optimization problem under the fixed N .

Denote the solution of the above optimization problem as R ( α, N ) . Since R ( α, N ) depends on a and cannot achieve a closed-form solution, we numerically choose a and observe an asymptotic behavior of R ( α, N ) . By choosing α = 3 . 8 , asymptotic rate of R (3 . 8 , N ) is about 46 N 2 .

E Broader Impacts

Our work focuses on the theoretical aspects of convex optimization algorithms. There are no negative social impacts that we anticipate from our theoretical results.

F Limitations

Our analysis concerns L -smooth convex functions. Although this assumption is standard in optimization theory, many functions that arise in machine learning practice are neither smooth nor convex.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces novel notion of H-duality, which shows unexpected connection between time-reversed coefficients of methods and optimality w.r.t. gradient norm or function value.\n\nSTRENGTHS:\nVery inspiring work, introduced notion is quite curious. I think that such an idea will be interesting to most of readers. The contribution of this paper is primarily theoretical which could make it \"intermediate\" for those readers who work in practical field of ML, but I believe that introduced notions can help theorists to develop the theory of optimisation taking into account more important details, which will impact on practice in the end. Especially such abstract works can affect theory the most, if the introduced notion is really helpful and fundamental, which can be confirmed only by the time, but I see it necessary to present such ideas to community to \"test\" them. Idea seems to be novel: my experience doesn't tell me attempts to introduced such a parallels in methods constructions in terms of duality notion.\n\nWEAKNESSES:\nI do not see significant weaknesses. Proof presented in the paper are correct, but assumption are expressful enough to make proofs simple in cases where they are satisfied. In the best case, more detailed development of these concepts should lead to simplification and weakening of assumptions by showing from when do they arise in many cases.\n\nQUESTIONS:\nNotion is based on phrases like \"efficiently reduce function values/gradient magnitude\", which is not very clear in part \"efficiently\". Any method which optimises function value optimises gradient norm as well for convex functions, so word \"efficiently\" requires formalisation. I guess, it is connected with optimality w.r.t. different measures of convergence. If it's possible to develop it better, it would be good.\n\nLIMITATIONS:\nEverything is okay", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work defines H-duality, which is a one-to-one correspondence between methods that minimize function values and methods that minimize gradient magnitudes, under the assumption that the objective function is convex and $L$-smooth. It is proved in both discrete and continuous time dynamics that, when one method satisfies a convergence condition, the convergence of its H-duality correspondence can also be guaranteed. Furthermore, a new class of methods is derived by applying H-duality to a known class of FSFOMs, and the H-duality convergence theorem can be used to establish the convergence of the new methods.\n\nSTRENGTHS:\n1. To the best of my knowledge, the concept of H-duality proposed in this work is novel, and the theorems which establish the equivalence between convergence conditions of two algorithms and their H-duality are also novel.\n\n2. This work demonstrates that the proposed H-duality can be useful, by using it to establish a symmetry property between pairs of methods, to prove the convergence of them, and to derive new methods.\n\n3. The paper is in general well written, and the sketch of proof helps readers more easily understand the techniques in the proof.\n\nWEAKNESSES:\n1. It should be noted that this work only applies to $L$-smooth and convex objective functions, and the value of $L$ is required be known as a prior, since it is used in the methods.\n\n2. If I understand correctly, the number of iterations $N$ is fixed in advance, which could also limit the applicability of this work. (details in question 1)\n\nQUESTIONS:\n1. In practice a method is usually terminated when a certain stopping criterion is reached, and the total number of iterations $N$ is often not known or fixed in advance. However, in this work, $N$ is required to be known and fixed. Could the authors please discuss about this difference? How would it affect the applicability of the conclusions in this work?\n\n2. In section 2.3, it is not obvious to me why $U \\geq 0$ is equivalent to (C1), because there is an additional term $- \\frac{L}{2} \\|\\|x_\\star -x_0 + \\frac{1}{L} \\sum_{i=0}^N (u_i - u_{i-1}) \\nabla f(x_i) \\|\\|^2$, which is not in (C1). Could the authors please explain in a high level how to bridge this gap?\n\nLIMITATIONS:\nAs far as I can see, there is no potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper studies the duality for optimization algorithms. As claimed by the authors, the notion they present is distinct form any previously known duality or symmetry relations. In this work, the authors present a one-to-one correspondence between methods efficiently minimizing function values and methods efficiently minimizing gradient magnitude, which is called H-duality. Using H-duality, a clearer understanding of the symmetry between FGM/OGM and OGM-G and a new class of methods efficiently reducing gradient magnitudes can be obtained. The authors claim that their new composite minimization method achieve better results than state-of-the-art methods in efficiently reducing gradient magnitude in the composite minimization setup.

STRENGTHS:
1. This paper presents a novel notion of duality.
2. The theory of H-duality seems to be sound.

WEAKNESSES:
1. The relevance to NeurIPS is a concern. Actually, this paper studies the duality for optimization algorithms, which is an important problem in Optimization and Control. I am not sure whether NeurIPS is the right place.
2. In the Section 4 “New method efficiently reducing gradient mapping norm: (SFG)”, the authors claim that their method (SFG) is faster than FISTA-G by a constant factor of 5.28 while having substantially simpler coefficients. In the paper which presented FISTA-G, two graph were given to describe different methods’ effectiveness on reducing the gradient magnitude. But the result of experiments can’t be found in the paper, which make the comparison between SFG and FISTA-G is lack of persuasiveness. Could you add the result of experiments to describe the comparison between SFG and FISTA-G?

QUESTIONS:
Please see my question in the "Weakness" section.

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows an intriguing symmetry in convex optimization between first-order methods that efficiently minimizes the function value and those that efficiently minimizes the squared gradient norm. In particular, they show that if a specific Lyapunov function upper bounds the function value suboptimality for Algorithm A, then a corresponding Lyapunov function would upper bound the squared gradient norm for Algorithm B, which is the \"dual\" of Algorithm A. This duality relation is further extended to continuous-time dynamics. By using this new tool, the authors propose a new parameterized family of methods for reducing the squared gradient norm, as well as a new composite minimization method that improves the state-of-the-art by a constant factor.\n\nSTRENGTHS:\n- I find the results in this paper interesting and novel. At the surface, accelerated methods for reducing the function value seem quite distinct from those for reducing the gradient norm, and the mechanism for the latter is somehow mysterious. Surprisingly, the authors uncover a one-to-one correspondence between these two families of algorithms. This new concept of H-duality could help us better understand OGM-G type algorithms and design better and practical ones for minimizing the gradient norm. \n- Moreover, the paper is very well-written. While the underlying proofs require heavy notations and tedious calculations, the authors manage to make the main theorems quite accessible and clearly explain the high-level ideas.\n\nWEAKNESSES:\nOverall, I don't find any major issues with this paper. While this might be too much to ask, I feel that I still don't have a good intuition for H-duality theorem or why it makes sense to reverse the combining coefficients, except that the algebra indicates so. Still, I believe this could be the first step towards some more fundamental understanding.\n\nQUESTIONS:\n- Just a clarifying question, it seems to me that the H-duality is only useful when the algorithm can be analyzed using the particular Lyapunov function in (2). On the other hand, many algorithms, such as the original Nesterov's accelerated method, may not conform to such a Lyapunov function, since they do not rely on the cocoercivity inequaltiy. Is it possible to generalize (2) to a broader family of Lyapunov functions?\n- By definition, it is necessary to know the time horizon $N$ to form the H-dual of a given algorithm. Is it possible to establish a duality relation between two anytime algorithms? More concretely, can we use H-duality theorem to obtain an anytime method for reducing the gradient norm?\n\nA minor typo: in (1) it should be $\\nabla f(x_i)$ instead of $\\nabla f(x_k)$.\n\nLIMITATIONS:\nThe authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the concept of H-duality, which refers to the duality between optimization algorithms. The notion involves two sets of algorithms: 1) those that efficiently reduce the function value, and 2) those that efficiently reduce the gradient magnitude. The authors create the H matrix, which contains the coefficients of previous gradient evaluations at time step t, for all fixed step size first-order methods. They define the anti-transpose of H, denoted as H^A, as the H-dual of H.\n\nIn the main theorem, the authors demonstrate that if there exists an algorithm that reduces the function value using matrix H, then there is another algorithm that minimizes the gradient magnitude using H^A. This duality property is applicable to both discrete time and continuous time first-order optimization algorithms.\n\nLeveraging this duality property for composite objectives, the authors utilize a proximal-based first-order method that minimizes the function value. Subsequently, they propose a novel algorithm that minimizes the gradient value.\n\nSTRENGTHS:\nThe introduction of duality between algorithms is a concept presented to the optimization community. This idea not only aids in establishing equivalence between algorithms but also proves valuable in conducting convergence analysis for these algorithms.\n\nWEAKNESSES:\nThe proposed duality, while introducing a novel perspective, has certain limitations as it does not encompass a broader class of algorithms and loss functions commonly used in machine learning. For instance, it requires the convergence of the last iterate and overlooks convergence based on average iterates. Additionally, it assumes the objective to be convex and L-smooth, which may not always hold in practical Machine Learning scenarios. As a result, the paper primarily pertains to optimization theory rather than providing specific optimization techniques tailored to machine learning applications, especially for stochastic algorithms frequently used in this field.\n\nThe paper lacks a clear explanation of how discovering this equivalence between algorithms can facilitate the development of better optimization techniques. For instance, in cases where the objective is convex and the gradient vanishes (\\nabla f* = 0), it is evident that achieving convergence based on function values can easily translate to convergence based on gradient magnitudes.\n\nAnother aspect that remains unclear in the paper is whether algorithms that are H-dual to each other exhibit similar convergence rates. For instance, in the gradient descent (GD) example, it is observed that the gradient norm can attain a rate of O(1/N^2) under certain conditions. Consequently, it is ambiguous whether two dual algorithms have identical convergence rates or if one may achieve a better rate. In the former case, it becomes necessary to conduct separate convergence analyses for each algorithm to account for potential differences in their rates.\n\nQUESTIONS:\nlook at the weaknesses\n\nLIMITATIONS:\nHave been addressed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}