SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jaeyeon Kim
kjy011102@snu.ac.kr
ernestryu@snu.ac.kr
In convex optimization, first-order optimization methods efficiently minimizing function values have been a central subject study since Nesterov's seminal work of 1983. Recently, however, Kim and Fessler's OGM-G and Lee et al.'s FISTA-G have been presented as alternatives that efficiently minimize the gradient magnitude instead. In this paper, we present H-duality, which represents a surprising one-to-one correspondence between methods efficiently minimizing function values and methods efficiently minimizing gradient magnitude. In continuous-time formulations, H-duality corresponds to reversing the time dependence of the dissipation/friction term. To the best of our knowledge, H-duality is different from Lagrange/Fenchel duality and is distinct from any previously known duality or symmetry relations. Using H-duality, we obtain a clearer understanding of the symmetry between Nesterov's method and OGM-G, derive a new class of methods efficiently reducing gradient magnitudes of smooth convex functions, and find a new composite minimization method that is simpler and faster than FISTA-G.
Since Nesterov's seminal work of 1983 [37], accelerated first-order optimization methods that efficiently reduce function values have been central to the theory and practice of large-scale optimization and machine learning. In 2012, however, Nesterov initiated the study of first-order methods that efficiently reduce gradient magnitudes of convex functions [41]. In convex optimization, making the function value exactly optimal is equivalent to making the gradient exactly zero, but reducing the function-value suboptimality below a threshold is not equivalent to reducing the gradient magnitude below a threshold. This line of research showed that accelerated methods for reducing function values, such as Nesterov's FGM [37], the more modern OGM [26], and the accelerated composite optimization method FISTA [11] are not optimal for reducing gradient magnitude, and new optimal alternatives, such as OGM-G [29] and FISTA-G [31], were presented.
These new accelerated methods for reducing gradient magnitudes are understood far less than those for minimizing function values. However, an interesting observation of symmetry, described in Section 2, was made between these two types of methods, and it was conjectured that this symmetry might be a key to understanding the acceleration mechanism for efficiently reducing gradient magnitude.
Contribution. We present a surprising one-to-one correspondence between methods efficiently minimizing function values and methods efficiently minimizing gradient magnitude. We call this correspondence H-duality and formally establish a duality theory in both discrete- and continuous-time dynamics. Using H-duality, we obtain a clearer understanding of the symmetry between FGM/OGM and OGM-G, derive a new class of methods efficiently reducing gradient magnitudes, and find a new composite minimization method that is simpler and faster than FISTA-G, the prior state-of-the-art in efficiently reducing gradient magnitude in the composite minimization setup.
1.1 Preliminaries and Notation
Given  $f\colon \mathbb{R}^d\to \mathbb{R}$ , write  $f_{\star} = \inf_{x\in \mathbb{R}^{d}}f(x)\in (-\infty ,\infty)$  for the minimum value and  $x_{\star}\in \operatorname{argmin}_{x\in \mathbb{R}^n}f(x)$  for a minimizer, if one exists. Throughout this paper, we assume  $f_{\star}\neq -\infty$  but we do not always assume a minimizer  $x_{\star}$  exists. Given a differentiable  $f\colon \mathbb{R}^d\to \mathbb{R}$  and a pre-specified value of  $L > 0$ , we define the notation
$$
[ x, y ] := f (y) - f (x) + \langle \nabla f (y), x - y \rangle
$$
$$
\llbracket x, y \rrbracket := f (y) - f (x) + \langle \nabla f (y), x - y \rangle + \frac {1}{2 L} \left\| \nabla f (x) - \nabla f (y) \right\| ^ {2}
$$
$$
\llbracket x, \star \rrbracket := f _ {\star} - f (x) + \frac {1}{2 L} \| \nabla f (x) \| ^ {2}
$$
for  $x, y \in \mathbb{R}^d$ . A differentiable function  $f \colon \mathbb{R}^d \to \mathbb{R}$  is convex if the convexity inequality  $[x, y] \leq 0$  holds for all  $x, y \in \mathbb{R}^d$ . For  $L > 0$ , a function  $f \colon \mathbb{R}^d \to \mathbb{R}$  is  $L$ -smooth convex if it is differentiable and the cocoercivity inequality  $\llbracket x, y \rrbracket \leq 0$  holds for all  $x, y \in \mathbb{R}^d$  [42]. If  $f$  has a minimizer  $x_\star$ , then  $\llbracket x, \star \rrbracket = \llbracket x, x_\star \rrbracket$ , but the notation  $\llbracket x, \star \rrbracket$  is well defined even when a minimizer  $x_\star$  does not exist. If  $f$  is  $L$ -smooth convex, then  $\llbracket x, \star \rrbracket \leq 0$  holds for all  $x \in \mathbb{R}^d$  [42].
Throughout this paper, we consider the duality between the following two problems.
(P1) Efficiently reduce  $f(x_{N}) - f_{\star}$  assuming  $x_{\star}$  exists and  $\| x_0 - x_\star \| \leq R$ .
(P2) Efficiently reduce  $\frac{1}{2L}\| \nabla f(y_N)\|^2$  assuming  $f_{\star} > -\infty$  and  $f(y_0) - f_{\star} \leq R$ .
Here,  $R \in (0,\infty)$  is a parameter,  $x_0$  and  $y_0$  denote initial points of methods for (P1) and (P2), and  $x_N$  and  $y_N$  denote outputs of methods for (P1) and (P2).
Finally, the standard gradient descent (GD) with stepsize  $h$  is
$$
x _ {i + 1} = x _ {i} - \frac {h}{L} \nabla f (x _ {i}), \quad i = 0, 1, \dots . \tag {GD}
$$
Classically, the goal of optimization methods is to reduce the function value efficiently. In the smooth convex setup, Nesterov's fast gradient method (FGM) [37] achieves an accelerated  $\mathcal{O}(1 / N^2)$ -rate, and the optimized gradient method (OGM) [26] improves this rate by a factor of 2, which is, in fact, exactly optimal [18].
On the other hand, Nesterov initiated the study of methods for reducing the gradient magnitude of convex functions [41] as such methods help us understand non-convex optimization better and design faster non-convex machine learning methods. For smooth convex functions, (GD) achieves a  $\mathcal{O}((f(x_0) - f_\star) / N)$ -rate on the squared gradient magnitude [34, Proposition 3.3.1], while (OGM-G) achieves an accelerated  $\mathcal{O}((f(x_0) - f_\star) / N^2)$ -rate [29], which matches a lower bound and is therefore optimal [35, 36]. Interestingly, (OGM) and (OGM-G) exhibit an interesting hint of symmetry, as we detail in Section 2, and the goal of this work is to derive a more general duality principle from this observation.
In the composite optimization setup, iterative shrinkage-thresholding algorithm (ISTA) [13, 45, 16, 14] achieves a  $\mathcal{O}(\| x_0 - x_\star \| ^2 /N)$ -rate on function-value suboptimality, while the fast iterative shrinkage-thresholding algorithm (FISTA) [11] achieves an accelerated  $\mathcal{O}(\| x_0 - x_\star \| /N^2)$ -rate. On the squared gradient mapping norm, FISTA-G achieves  $\mathcal{O}((F(x_0) - F_\star) / N^2)$ -rate [31], which is optimal [35, 36]. Analysis of an accelerated method often uses the estimate sequence technique [38, 8, 39, 9, 40, 32] or a Lyapunov analysis [37, 11, 50, 10, 53, 1, 5, 6, 7, 43]. In this work, we focus on the Lyapunov analysis technique, as it is simpler and more amenable to a continuous-time view.
The notion of duality is fundamental in many branches of mathematics, including optimization. Lagrange duality [46, 47, 12], Wolfe duality [57, 15, 49, 33], and Fenchel-Rockacheller duality [20, 46] are related (arguably equivalent) notions that consider a pairing of primal and dual optimization problems. The recent gauge duality [21, 22, 3, 58] and radial duality [24, 23] are alternative notions of duality for optimization problems. Attouch-Théra duality [4, 48] generalizes Fenchel-Rockacheller to the setup of monotone inclusion problems. In this work, we present H-duality, which is a notion of duality for optimization algorithms, and it is, to the best of our knowledge, distinct from any previously known duality or symmetry relations.
2 H-duality
In this section, we will introduce H-duality, state the main H-duality theorem, and provide applications. Let  $N \geq 1$  be a pre-specified iteration count. Let  $\{h_{k,i}\}_{0 \leq i < k \leq N}$  be an array of (scalar) stepsizes and identify it with a lower triangular matrix  $H \in \mathbb{R}^{N \times N}$  via  $H_{k+1,i+1} = h_{k+1,i}$  if  $0 \leq i \leq k \leq N-1$  and  $H_{k,i} = 0$  otherwise. An  $N$ -step Fixed Step First Order Method (FSFOM) with  $H$  is
$$
x _ {k + 1} = x _ {k} - \frac {1}{L} \sum_ {i = 0} ^ {k} h _ {k + 1, i} \nabla f (x _ {i}), \quad \forall k = 0, \dots , N - 1 \tag {1}
$$
for any initial point  $x_0 \in \mathbb{R}^d$  and differentiable  $f$ . For  $H \in \mathbb{R}^{N \times N}$ , define its anti-transpose  $H^A \in \mathbb{R}^{N \times N}$  with  $H_{i,j}^A = H_{N - j + 1,N - i + 1}$  for  $i,j = 1,\dots,N$ . We call [FSFOM with  $H^A$ ] the H-dual of [FSFOM with  $H$ ].
2.1 Symmetry between OGM and OGM-G
Let  $f$  be an  $L$ -smooth convex function. Define the notation  $z^{+} = z - \frac{1}{L}\nabla f(z)$  for  $z\in \mathbb{R}^d$ . The accelerated methods OGM [19, 26] and OGM-G [29] are
$$
x _ {k + 1} = x _ {k} ^ {+} + \frac {\theta_ {k} - 1}{\theta_ {k + 1}} \left(x _ {k} ^ {+} - x _ {k - 1} ^ {+}\right) + \frac {\theta_ {k}}{\theta_ {k + 1}} \left(x _ {k} ^ {+} - x _ {k}\right) \tag {OGM}
$$
$$
y _ {k + 1} = y _ {k} ^ {+} + \frac {\left(\theta_ {N - k} - 1\right) \left(2 \theta_ {N - k - 1} - 1\right)}{\theta_ {N - k} \left(2 \theta_ {N - k} - 1\right)} \left(y _ {k} ^ {+} - y _ {k - 1} ^ {+}\right) + \frac {2 \theta_ {N - k - 1} - 1}{2 \theta_ {N - k} - 1} \left(y _ {k} ^ {+} - y _ {k}\right) \quad (\mathrm {O G M - G})
$$
for  $k = 0, \dots, N - 1$ , where  $\{\theta_i\}_{i=0}^N$  are defined as  $\theta_0 = 1$ ,  $\theta_{i+1}^2 - \theta_{i+1} = \theta_i^2$  for  $0 \leq i \leq N - 2$ , and  $\theta_N^2 - \theta_N = 2\theta_{N-1}^2$ .<sup>1</sup> (OGM) and (OGM-G) are two representative accelerated methods for the setups (P1) and (P2), respectively. As a surface-level symmetry, the methods both access the  $\{\theta_i\}_{i=0}^N$  sequence, but (OGM-G) does so in a reversed ordering [29]. There turns out to be a deeper-level symmetry: (OGM) and (OGM-G) are H-duals of each other, i.e.,  $H_{\mathrm{OGM}}^A = H_{\mathrm{OGM-G}}$ . The proof structures of (OGM) and (OGM-G) also exhibit symmetry. We can analyze (OGM) with the Lyapunov function
$$
\mathcal {U} _ {k} = \frac {L}{2} \| x _ {0} - x _ {\star} \| ^ {2} + \sum_ {i = 0} ^ {k - 1} u _ {i} \llbracket x _ {i}, x _ {i + 1} \rrbracket + \sum_ {i = 0} ^ {k} (u _ {i} - u _ {i - 1}) \llbracket x _ {\star}, x _ {i} \rrbracket \tag {2}
$$
for  $-1\leq k\leq N$  with  $\{u_i\}_{i = 0}^N = (2\theta_0^2,\dots ,2\theta_{N - 1}^2,\theta_N^2)$  and  $u_{-1} = 0$ . Since  $[\cdot ,\cdot ]\leq 0$  and  $\{u_i\}_{i = 0}^N$  is a positive monotonically increasing sequence,  $\{\mathcal{U}_k\}_{k = -1}^N$  is dissipative, i.e.,  $\mathcal{U}_N\leq \mathcal{U}_{N - 1}\leq \dots \leq$ $\mathcal{U}_0\leq \mathcal{U}_{-1}$ . So
$$
\theta_ {N} ^ {2} \left(f (x _ {N}) - f _ {\star}\right) \leq \theta_ {N} ^ {2} \left(f (x _ {N}) - f _ {\star}\right) + \frac {L}{2} \| x ^ {\star} - x _ {0} + z \| ^ {2} \stackrel {(\bullet)} {=} \mathcal {U} _ {N} \leq \mathcal {U} _ {- 1} = \frac {L \| x _ {0} - x ^ {\star} \| ^ {2}}{2},
$$
where  $z = \sum_{i=0}^{N} \frac{u_i - u_{i-1}}{L} \nabla f(x_i)$ . The justification of  $(\bullet)$  is the main technical challenge of this analysis, and it is provided in Appendix B.2. Dividing both sides by  $\theta_N^2$ , we conclude the rate
$$
f (x _ {N}) - f _ {\star} \leq \frac {1}{\theta_ {N} ^ {2}} \frac {L}{2} \| x _ {0} - x ^ {\star} \| ^ {2}.
$$
Likewise, we can analyze (OGM-G) with the Lyapunov function
$$
\mathcal {V} _ {k} = v _ {0} \left(f \left(y _ {0}\right) - f _ {\star} + \llbracket y _ {N}, \star \rrbracket\right) + \sum_ {i = 0} ^ {k - 1} v _ {i + 1} \llbracket y _ {i}, y _ {i + 1} \rrbracket + \sum_ {i = 0} ^ {k - 1} \left(v _ {i + 1} - v _ {i}\right) \llbracket y _ {N}, y _ {i} \rrbracket \tag {3}
$$
for  $0 \leq k \leq N$  with  $\{v_i\}_{i=0}^N = \left(\frac{1}{\theta_N^2}, \frac{1}{2\theta_{N-1}^2}, \ldots, \frac{1}{2\theta_0^2}\right)$ . Similarly,  $\{\mathcal{V}_k\}_{k=0}^N$  is dissipative, so
$$
\frac {1}{2 L} \| \nabla f (y _ {N}) \| ^ {2} \stackrel {(\circ)} {=} \mathcal {V} _ {N} \leq \mathcal {V} _ {0} = \frac {1}{\theta_ {N} ^ {2}} \left(f (y _ {0}) - f _ {\star}\right) + \frac {1}{\theta_ {N} ^ {2}} [   [ y _ {N}, \star ]   ] \leq \frac {1}{\theta_ {N} ^ {2}} \left(f (y _ {0}) - f _ {\star}\right).
$$
Throughout this paper, we use the convention of denoting iterates of a given "primal" FSFOM as  $x_{k}$  while denoting the iterates of the H-dual FSFOM as  $y_{k}$ .
Again, the justification of  $(\circ)$  is the main technical challenge of this analysis, and it is provided in Appendix B.2. The crucial observations are (i)  $u_{i} = 1 / v_{N - i}$  for  $0\leq i\leq N$  and (ii) the convergence rates share the identical factor  $1 / \theta_N^2 = 1 / u_N = v_0$ . Interestingly, a similar symmetry relation holds between method pairs  $[(OBL - F_b),(OBL - G_b)][44]$  and  $[(GD),(GD)]$ , which we discuss later in Section 2.4.
2.2 H-duality theorem
The symmetry observed in Section 2.1 is, in fact, not a coincidence. Suppose we have  $N$ -step FSFOMs with  $H$  and  $H^{A}$ . We denote their iterates as  $\{x_{i}\}_{i = 0}^{N}$  and  $\{y_{i}\}_{i = 0}^{N}$ . For clarity, note that  $\{u_i\}_{i = 0}^N$  are free variables and can be appropriately chosen for the convergence rate analysis. For the FSFOM with  $H$ , define  $\{\mathcal{U}_k\}_{k = -1}^N$  with the general form (2) with  $u_{-1} = 0$ . If  $0 = u_{-1}\leq u_0\leq u_1\leq \dots \leq u_N$ , then  $\{\mathcal{U}_k\}_{k = -1}^N$  is monotonically nonincreasing (dissipative). Assume we can show
$$
u _ {N} \left(f \left(x _ {N}\right) - f _ {\star}\right) \leq \mathcal {U} _ {N} \quad (\forall x _ {0}, x _ {\star}, \nabla f (x _ {0}), \dots , \nabla f (x _ {N}) \in \mathbb {R} ^ {d}). \tag {C1}
$$
To clarify, since  $\{x_i\}_{i=0}^N$  lies within  $\operatorname{span}\{x_0, \nabla f(x_0), \ldots, \nabla f(x_N)\}$ , the  $\mathcal{U}_N$  depends on  $(x_0, x_\star, \{\nabla f(x_i)\}_{i=0}^N, \{u_i\}_{i=0}^N, H)$ . If (C1) holds, the FSFOM with  $H$  exhibits the convergence rate
$$
u _ {N} \left(f \left(x _ {N}\right) - f _ {\star}\right) \leq \mathcal {U} _ {N} \leq \dots \leq \mathcal {U} _ {- 1} = \frac {L}{2} \| x _ {0} - x _ {\star} \| ^ {2}. \tag {4}
$$
For the FSFOM with  $H^{A}$ , define  $\{\mathcal{V}_k\}_{k=0}^N$  with the general form (3). Also, note that  $\{v_i\}_{i=0}^N$  are free variables and can be appropriately chosen for the convergence rate analysis. If  $0 \leq v_0 \leq v_1 \leq \dots \leq v_N$ , then  $\{\mathcal{V}_k\}_{k=0}^N$  is monotonically nonincreasing (dissipative). Assume we can show
$$
\frac {1}{2 L} \| \nabla f (y _ {N}) \| ^ {2} \leq \mathcal {V} _ {N} \quad (\forall y _ {0}, \nabla f (y _ {0}), \dots , \nabla f (y _ {N}) \in \mathbb {R} ^ {d}, f _ {\star} \in \mathbb {R}). \tag {C2}
$$
To clarify, since  $\{y_i\}_{i=0}^N$  lies within  $\operatorname{span}\{y_0, \nabla f(y_0), \ldots, \nabla f(y_N)\}$ , the  $\mathcal{V}_N$  depends on  $\left(y_0, \{\nabla f(y_i)\}_{i=0}^N, f_\star, \{v_i\}_{i=0}^N, H^A\right)$ . If (C2) holds, the FSFOM with  $H^A$  exhibits the convergence rate
$$
\frac {1}{2 L} \| \nabla f (y _ {N}) \| ^ {2} \leq \mathcal {V} _ {N} \leq \dots \leq \mathcal {V} _ {0} = v _ {0} \left(f (y _ {0}) - f _ {\star}\right) + v _ {0} [   [ y _ {N}, \star ]   ] \leq v _ {0} \left(f (y _ {0}) - f _ {\star}\right). \tag {5}
$$
We now state our main H-duality theorem, which establishes a correspondence between the two types of bounds for the FSFOMs induced by  $H$  and  $H^{A}$ .
Theorem 1. Consider sequences of positive real numbers  $\{u_i\}_{i=0}^N$  and  $\{v_i\}_{i=0}^N$  related through  $v_i = \frac{1}{u_{N-i}}$  for  $i = 0, \ldots, N$ . Let  $H \in \mathbb{R}^{N \times N}$  be lower triangular. Then,
$$
\left[ \left(\mathrm {C} 1\right) \text {i s s a t i s f i e d w i t h} \left\{u _ {i} \right\} _ {i = 0} ^ {N} \text {a n d} H \right] \Leftrightarrow \left[ \left(\mathrm {C} 2\right) \text {i s s a t i s f i e d w i t h} \left\{v _ {i} \right\} _ {i = 0} ^ {N} \text {a n d} H ^ {A} \right].
$$
Theorem 1 provides a sufficient condition that ensures an FSFOM with  $H$  with a convergence guarantee on  $(f(x_{N}) - f_{\star})$  can be H-dualized to obtain an FSFOM with  $H^{A}$  with a convergence guarantee on  $\| \nabla f(y_{N})\|^{2}$ . To the best of our knowledge, this is the first result establishing a symmetrical relationship between (P1) and (P2). Section 2.3 provides a proof outline of Theorem 1.
2.3 Proof outline of Theorem 1
$$
\mathbf {U} \colon = \mathcal {U} _ {N} - u _ {N} (f (x _ {N}) - f _ {\star}) - \frac {L}{2} \left\| x _ {\star} - x _ {0} + \frac {1}{L} \sum_ {i = 0} ^ {N} (u _ {i} - u _ {i - 1}) \nabla f (x _ {i}) \right\| ^ {2}
$$
$$
\mathbf {V} \colon = \mathcal {V} _ {N} - \frac {1}{2} \| \nabla f (y _ {N}) \| ^ {2}.
$$
Expanding  $\mathbf{U}$  and  $\mathbf{V}$  reveals that all function value terms are eliminated and only quadratic terms of  $\{\nabla f(x_i)\}_{i = 0}^N$  and  $\{\nabla f(y_i)\}_{i = 0}^N$  remain. Now, (C1) and (C2) are equivalent to the conditions
$$
\left[ \mathbf {U} \geq 0, \quad \forall (\nabla f (x _ {0}), \dots , \nabla f (x _ {N}) \in \mathbb {R} ^ {d}) \right], \qquad \left[ \mathbf {V} \geq 0 \quad \forall (\nabla f (y _ {0}), \dots , \nabla f (y _ {N}) \in \mathbb {R} ^ {d}) \right],
$$
respectively. Next, define  $g_{x} = \left[\nabla f(x_{0})|\nabla f(x_{1})|\ldots |\nabla f(x_{N})\right] \in \mathbb{R}^{d\times (N + 1)}$  and  $g_{y} = \left[\nabla f(y_{0})|\nabla f(y_{1})|\ldots |\nabla f(y_{N})\right] \in \mathbb{R}^{d\times (N + 1)}$ . We show that there is  $S(H,u)$  and  $\mathcal{T}(H^A,v) \in \mathbb{S}^{N + 1}$  such that
$$
\mathbf {U} = \operatorname {T r} \left(g _ {x} \mathcal {S} (H, u) g _ {x} ^ {\intercal}\right), \qquad \mathbf {V} = \operatorname {T r} \left(g _ {y} \mathcal {T} \left(H ^ {A}, v\right) g _ {y} ^ {\intercal}\right).
$$
Next, we find an explicit invertible matrix  $M(u) \in \mathbb{R}^{(N + 1)\times (N + 1)}$  such that  $S(H,u) = \mathcal{M}(u)^{\intercal}\mathcal{T}(H^A,v)\mathcal{M}(u)$ . Therefore,
$$
\operatorname {T r} \left(g _ {x} \mathcal {S} (H, u) g _ {x} ^ {\intercal}\right) = \operatorname {T r} \left(g _ {y} \mathcal {T} \left(H ^ {A}, v\right) g _ {y} ^ {\intercal}\right)
$$
with  $g_{y} = g_{x}\mathcal{M}(u)^{\intercal}$  and we conclude the proof. This technique of considering the quadratic forms of Lyapunov functions as a trace of matrices is inspired by the ideas from the Performance Estimation Problem (PEP) literature [19, 55]. The full proof is given in Appendix A.
2.4 Verifying conditions for H-duality theorem
In this section, we illustrate how to verify conditions (C1) and (C2) through examples. Detailed calculations are deferred to Appendices B.1 and B.2.
Example 1. For (OGM) and (OGM-G), the choice
$$
\left\{u _ {i} \right\} _ {i = 0} ^ {N} = \left(2 \theta_ {0} ^ {2}, \dots , 2 \theta_ {N - 1} ^ {2}, \theta_ {N} ^ {2}\right), \qquad \left\{v _ {i} \right\} _ {i = 0} ^ {N} = \left(\frac {1}{\theta_ {N} ^ {2}}, \frac {1}{2 \theta_ {N - 1} ^ {2}}, \dots , \frac {1}{2 \theta_ {0} ^ {2}}\right)
$$
$$
\mathbf {U} = 0, \quad \mathbf {V} = 0.
$$
Therefore, (C1) and (C2) hold.
Example 2. Again, define  $z^{+} = z - \frac{1}{L}\nabla f(z)$  for  $z\in \mathbb{R}^d$ . Consider the FSFSOMs [44]
$$
\begin{array}{l} x _ {k + 1} = x _ {k} ^ {+} + \frac {k}{k + 3} \left(x _ {k} ^ {+} - x _ {k - 1} ^ {+}\right) + \frac {k}{k + 3} \left(x _ {k} ^ {+} - x _ {k}\right) \quad k = 0, \dots , N - 2 \\ x _ {N} = x _ {N - 1} ^ {+} + \frac {N - 1}{2 (\gamma + 1)} \left(x _ {N - 1} ^ {+} - x _ {N - 2} ^ {+}\right) + \frac {N - 1}{2 (\gamma + 1)} \left(x _ {N - 1} ^ {+} - x _ {N - 1}\right) \tag {OBL-Fb} \\ \end{array}
$$
$$
\begin{array}{l} y _ {1} = y _ {0} ^ {+} + \frac {N - 1}{2 (\gamma + 1)} \left(y _ {0} ^ {+} - y _ {- 1} ^ {+}\right) + \frac {N - 1}{2 (\gamma + 1)} \left(y _ {0} ^ {+} - y _ {0}\right) \\ y _ {k + 1} = y _ {k} ^ {+} + \frac {N - k - 1}{N - k + 2} \left(y _ {k} ^ {+} - y _ {k - 1} ^ {+}\right) + \frac {N - k - 1}{N - k + 2} \left(y _ {k} ^ {+} - y _ {k}\right) \quad k = 1, \dots , N - 1 \tag {OBL-G_b} \\ \end{array}
$$
where  $y_{-1}^{+} = y_{0}$ ,  $x_{-1}^{+} = x_{0}$  and  $\gamma = \sqrt{N(N + 1) / 2}$ . It turns out that (OBL- $F_{\flat}$ ) and (OBL- $G_{\flat}$ ) are H-duals of each other. The choice
$$
\{u _ {i} \} _ {i = 0} ^ {N} = \left(\frac {1 \cdot 2}{2}, \ldots , \frac {N (N + 1)}{2}, \gamma^ {2} + \gamma\right), \quad \{v _ {i} \} _ {i = 0} ^ {N} = \left(\frac {1}{\gamma^ {2} + \gamma}, \frac {2}{N (N + 1)}, \ldots , \frac {2}{1 \cdot 2}\right)
$$
$$
\mathbf {U} = \sum_ {i = 0} ^ {N} \frac {u _ {i} - u _ {i - 1}}{2 L} \| \nabla f (x _ {i}) \| ^ {2}, \quad \mathbf {V} = \frac {v _ {0}}{2 L} \| \nabla f (y _ {N}) \| ^ {2} + \sum_ {i = 0} ^ {N - 1} \frac {v _ {i + 1} - v _ {i}}{2 L} \| \nabla f (y _ {i}) - \nabla f (y _ {N}) \| ^ {2}
$$
where  $u_{-1} = 0$ . Since  $\mathbf{U}$  and  $\mathbf{V}$  are expressed as a sum of squares, (C1) and (C2) hold.
Example 3. Interestingly, (GD) is a self-dual FSFOM in the H-dual sense. For the case  $h = 1$ , the choice
$$
\{u _ {i} \} _ {i = 0} ^ {N} = \left(\dots , \frac {(2 N + 1) (i + 1)}{2 N - i}, \dots , 2 N + 1\right), \qquad \{v _ {i} \} _ {i = 0} ^ {N} = \left(\frac {1}{2 N + 1}, \dots , \frac {N + i}{(2 N + 1) (N - i + 1)}, \dots\right)
$$
$$
\mathbf {U} = \sum_ {0 \leq i, j \leq N} \frac {s _ {i j}}{L} \left\langle \nabla f (x _ {i}), \nabla f (x _ {j}) \right\rangle , \quad \mathbf {V} = \sum_ {0 \leq i, j \leq N} \frac {t _ {i j}}{L} \left\langle \nabla f (y _ {i}), \nabla f (y _ {j}) \right\rangle
$$
for some  $\{s_{ij}\}$  and  $\{t_{ij}\}$  stated precisely in Appendix B.2.  $\mathbf{V} \geq 0$  can be established by showing that the  $\{t_{ij}\}$  forms a diagonally dominant and hence positive semidefinite matrix [29].  $\mathbf{U} \geq 0$  can be established with a more elaborate argument [19], but that is not necessary;  $\mathbf{V} \geq 0$  implies (C2), and, by Theorem 1, this implies (C1).
2.5 Applications of the H-duality theorem
A family of gradient reduction methods. Parameterized families of accelerated FSFOMs for reducing function values have been presented throughout the extensive prior literature. Such families generalize Nesterov's method and elucidate the essential algorithmic component that enables acceleration. For reducing gradient magnitude, however, there are only four accelerated FSFOMs (OGM-G), (OBL-  $\mathbf{G}_{\mathfrak{p}}$ ), and M-OGM-G [60], and [17, Lemma 2.6]. Here, we construct a simple parameterized family of accelerated FSFOMs for reducing gradient magnitude by H-dualizing an accelerated FSFOM family for reducing function values.
Let  $\{t_i\}_{i = 0}^N$  and  $\{T_i\}_{i = 0}^N$  be sequences positive real numbers satisfying  $t_i^2\leq 2T_i = 2\sum_{j = 0}^i t_j$  for  $0\leq i\leq N - 1$  and  $t_N^2\leq T_N = \sum_{j = 0}^{N}t_{j}$ . Consider a family of FSFOMs
$$
x _ {k + 1} = x _ {k} ^ {+} + \frac {\left(T _ {k} - t _ {k}\right) t _ {k + 1}}{t _ {k} T _ {k + 1}} \left(x _ {k} ^ {+} - x _ {k - 1} ^ {+}\right) + \frac {\left(t _ {k} ^ {2} - T _ {k}\right) t _ {k + 1}}{t _ {k} T _ {k + 1}} \left(x _ {k} ^ {+} - x _ {k}\right) \tag {6}
$$
for  $k = 0,1,\ldots ,N - 1$ , where  $x_{-1}^{+} = x_{0}$ . This family coincides with the GOGM of [28], and it exhibits the rate [28, Theorem 5]
$$
f (x _ {N}) - f _ {\star} \leq \frac {1}{T _ {N}} \frac {L}{2} \| x _ {0} - x _ {\star} \| ^ {2},
$$
which can be established from (2) with  $u_{i} = T_{i}$  for  $0 \leq i \leq N$ .
Corollary 1. The H-dual of (6) is
$$
y _ {k + 1} = y _ {k} ^ {+} + \frac {T _ {N - k - 1} (t _ {N - k - 1} - 1)}{T _ {N - k} (t _ {N - k} - 1)} \left(y _ {k} ^ {+} - y _ {k - 1} ^ {+}\right) + \frac {(t _ {N - k} ^ {2} - T _ {N - k}) (t _ {N - k - 1} - 1)}{T _ {N - k} (t _ {N - k} - 1)} \left(y _ {k} ^ {+} - y _ {k}\right)
$$
for  $k = 0, \dots, N - 1$ , where  $y_{-1}^{+} = y_{0}$ , and it exhibits the rate