SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {V} (T) - \frac {1}{2} \| \nabla f (Y (T)) \| ^ {2} \\ = - \frac {1}{2} \| g _ {0} \| ^ {2} - \frac {1}{u (T)} \int_ {0} ^ {T} \left\langle g _ {0}, \dot {Y} (s) \right\rangle d s + \int_ {0} ^ {T} \frac {u ^ {\prime} (T - s)}{u (T - s) ^ {2}} \left\langle Y (T) - Y (s), g _ {T - s} - g _ {0} \right\rangle d s \\ - \int_ {0} ^ {T} \frac {1}{u (T - s)} \left\langle \dot {Y} (s), g _ {T - s} - g _ {0} \right\rangle d s \\ = - \frac {1}{2} \| g _ {0} \| ^ {2} + \int_ {0} ^ {T} \frac {u ^ {\prime} (T - s)}{u (T - s) ^ {2}} \left\langle \int_ {s} ^ {T} \dot {Y} (a) d a, g _ {T - s} - g _ {0} \right\rangle d s \\ - \frac {1}{u (T)} \int_ {0} ^ {T} \left\langle g _ {0}, \dot {Y} (s) \right\rangle d s - \int_ {0} ^ {T} \frac {1}{u (T - s)} \left\langle \dot {Y} (s), g _ {T - s} - g _ {0} \right\rangle d s \\ \stackrel {(\circ)} {=} - \frac {1}{2} \| g _ {0} \| ^ {2} + \int_ {0} ^ {T} \left\langle - \frac {1}{u (T)} g _ {0} - \frac {1}{u (T - s)} \left(g _ {T - s} - g _ {0}\right) + \int_ {0} ^ {s} \frac {u ^ {\prime} (T - b)}{u (T - b) ^ {2}} \left(g _ {T - b} - g _ {0}\right) d b, \dot {Y} (s) \right\rangle d s. \tag {80} \\ \end{array}
$$
We use Fubini's Theorem at  $(\circ)$ . Finally, using (78), we obtain
$$
\begin{array}{l} \mathcal {V} (T) - \frac {1}{2} \left\| \nabla f (Y (T)) \right\| ^ {2} \\ = - \frac {1}{2} \| g _ {0} \| ^ {2} - \int_ {0} ^ {T} \left\langle f _ {T - s}, \dot {Y} (s) \right\rangle d s \\ = - \frac {1}{2} \| g _ {0} \| ^ {2} + \int_ {0} ^ {T} \left\langle f _ {T - s}, \int_ {0} ^ {s} H ^ {A} (s, a) g _ {T - a} d a \right\rangle d s \tag {81} \\ = - \frac {1}{2} \left\| g _ {0} \right\| ^ {2} + \int_ {0} ^ {T} \left\langle f _ {T - s}, \int_ {0} ^ {s} H (T - a, T - s) g _ {T - a} d a \right\rangle d s \\ = - \frac {1}{2} \| g _ {0} \| ^ {2} + \int_ {0} ^ {T} \int_ {0} ^ {s} H (s, a) \left\langle f _ {a}, g _ {s} \right\rangle d a d s. \\ \end{array}
$$
Proof of Theorem 2 Define  $\mathcal{A}_1$  and  $\mathcal{A}_2$  as follows.
$$
\begin{array}{l} \mathcal {A} _ {1} = \left\{\left\{f _ {s} \right\} _ {s \in [ 0, T ]} | \text {A n a l y s i s i n} \right. \\ \mathcal {A} _ {2} = \left\{\left\{g _ {s} \right\} _ {s \in [ 0, T ]} \mid \text {A n a l y s i s i n t h e p r e v i o u s p a r a g r a p h h o l d s} \right\}. \\ \end{array}
$$
Now we prove Theorem 2. We have shown that  $(\mathrm{C}3')$  is equivalent to (76)  $\geq 0$  for all  $\{\nabla f(X(s))\}_{s\in [0,T]}\in \mathcal{A}_1$ . By definition of  $\mathcal{A}_1$ , it is equivalent to
$$
- \frac {1}{2} \| g _ {0} \| ^ {2} + \int_ {0} ^ {T} \int_ {0} ^ {s} H (s, a) \langle f _ {a}, g _ {s} \rangle d a d s \geq 0
$$
for any  $\{g_s\}_{s\in [0,T]} \in \mathcal{A}_2$ . Moreover, by (81), it is also equivalent to
$$
- \frac {1}{2} \| g _ {0} \| ^ {2} + \int_ {0} ^ {T} \int_ {0} ^ {s} H (s, a) \langle f _ {a}, g _ {s} \rangle d a d s = \mathcal {V} (T) - \frac {1}{2} \| \nabla f (Y (T)) \| ^ {2} \geq 0
$$
for any  $\{g_s\}_{s\in [0,T]} \in \mathcal{A}_2$ , which is (C4').
C.3 Omitted parts in Section 3.3
Regularity of (10) at  $t = -T$ . To begin with, note that ODE (10) can be expressed as
$$
\left\{ \begin{array}{l} \dot {W} (t) = - \frac {2 p - 1}{T - t} W (t) - C p ^ {2} (T - t) ^ {p - 2} \nabla f (Y (t)) \\ \dot {Y} (t) = W (t) \end{array} \right.
$$
for  $t \in (0, T)$ . Since right hand sides are Lipschitz continuous with respect to  $W$  and  $Y$  in any closed interval  $[0, s] \in [0, T)$ , solution  $(Y, W)$  uniquely exists that satisfies above ODE with initial condition  $(Y(0), W(0)) = (y_0, 0)$ . Next, we give the proof of regularity of ODE (10) at terminal time  $T$  in the following order. The proof structure is based on the regularity proof in [51]:
(i)  $\sup_{t\in [0,T)}\left\| \dot{Y} (t)\right\|$  is bounded
(ii)  $Y(t)$  can be continuously extended to  $T$
(iii)  $\lim_{t\to T - }\left\| \dot{Y} (t)\right\| = 0$
(iv)  $\lim_{t\to T - }\frac{\dot{Y}(t)}{(T - t)^{p - 1}} = Cp\nabla f(Y(T)).$
Step (i):  $\sup_{t\in [0,T)}\left\| \dot{Y} (t)\right\|$  is bounded. ODE (10) is equivalent to
$$
\frac {1}{(T - s) ^ {p - 2}} \ddot {Y} (s) + \frac {2 p - 1}{(T - s) ^ {p - 1}} \dot {Y} (s) + C p ^ {2} \nabla f (Y (s)) = 0. \tag {82}
$$
By multiplying  $\dot{Y}(s)$  and integrating from 0 to  $t$ , we obtain
$$
\int_ {0} ^ {t} \frac {1}{(T - s) ^ {p - 2}} \left\langle \ddot {Y} (s), \dot {Y} (s) \right\rangle d s + \int_ {0} ^ {t} \frac {2 p - 1}{(T - s) ^ {p - 1}} \left\| \dot {Y} (s) \right\| ^ {2} d s + C p ^ {2} \int_ {0} ^ {t} \left\langle \dot {Y} (s), \nabla f (Y (s)) \right\rangle d s = 0
$$
and integration by parts gives us
$$
\frac {1}{2 (T - t) ^ {p - 2}} \left\| \dot {Y} (t) \right\| ^ {2} - \frac {1}{2 T ^ {p - 2}} \left\| \dot {Y} (0) \right\| ^ {2} + \int_ {0} ^ {t} \frac {2 p - 1}{(T - s) ^ {p - 1}} \left\| \dot {Y} (s) \right\| ^ {2} d s + C p ^ {2} \left(f (Y (t)) - f (Y (0))\right) = 0.
$$
Define  $\Psi (t)\colon [0,T)\to \mathbb{R}$  as
$$
\Psi (t) = \frac {1}{2 (T - t) ^ {p - 2}} \left\| \dot {Y} (t) \right\| ^ {2} + C p ^ {2} \left(f (Y (t)) - f (Y (0))\right).
$$
$$
\dot {\Psi} (t) = - \frac {2 p - 1}{(T - s) ^ {p - 1}} \left\| \dot {Y} (s) \right\| ^ {2} d s,
$$
$\Psi (t)$  is a nonincreasing function. Thus
$$
\left\| \dot {Y} (t) \right\| ^ {2} = 2 (T - t) ^ {p - 2} \left(\Psi (t) + C p ^ {2} \left(f (Y (0)) - f (Y (t))\right)\right) \leq 2 T ^ {p - 2} \left(\Psi (0) + C p ^ {2} \left(f (Y (0)) - f _ {\star}\right)\right), \tag {83}
$$
and the right hand side of (83) is constant, which implies  $M := \sup_{t \in [0,T)} \left\| \dot{Y}(t) \right\| < \infty$ .
Step (ii):  $Y(t)$  can be continuously extended to  $T$ . We can prove  $Y(t)$  is uniformly continuous due to the following analysis.
$$
\| Y (t + \delta) - Y (\delta) \| = \left\| \int_ {t} ^ {t + \delta} \dot {Y} (s) d s \right\| \leq \int_ {t} ^ {t + \delta} \left\| \dot {Y} (s) \right\| d s \leq \delta M.
$$
Since a uniformly continuous function  $g \colon D \to \mathbb{R}^d$  can be extended continuously to  $\overline{D}, Y \colon [0, T) \to \mathbb{R}^d$  can be extended to  $[0, T]$ .
Step (iii):  $\lim_{t\to T^{-}}\left\| \dot{Y} (t)\right\| = 0.$  We first prove the limit  $\lim_{t\to T^{-}}\left\| \dot{Y} (t)\right\| = 0$  exists. From C.3, we know  $\lim_{t\to T^{-}}Y(t)$  exists and by continuity of  $f,\lim_{t\to T^{-}}f(Y(t))$  also exists. Moreover,  $\Psi (t)$  is non-increasing and
$$
\Psi (t) = \frac {1}{2 (T - t) ^ {p - 2}} \left\| \dot {Y} (t) \right\| ^ {2} + C p ^ {2} \left(f (Y (t)) - f (Y (0))\right) \geq C p ^ {2} \left(f _ {\star} - f (Y (0))\right),
$$
thus  $\lim_{t\to T - }\Psi (t)$  exists. Therefore,  $\lim_{t\to T - }\frac{\left\|\dot{Y}(t)\right\|^2}{(T - t)^{p - 2}}$  exists, which implies  $\lim_{t\to T - }\left\| \dot{Y} (t)\right\| = 0$  when  $p > 2$ . For the case  $p = 2$ ,  $\lim_{t\to T - }\left\| \dot{Y} (t)\right\|$  exists, and  $\lim_{t\to T - }\left\| \dot{Y} (t)\right\| = 0$  follows since we have
$$
\Psi (t) = \frac {1}{2 T ^ {p}} \left\| \dot {Y} (0) \right\| ^ {2} - \int_ {0} ^ {t} \frac {3}{(T - s) ^ {p - 1}} \left\| \dot {Y} (s) \right\| ^ {2} d s
$$
is bounded below. Thus the value of integration is finite.
Step (iv):  $\lim_{t\to T^{-}}\frac{\dot{Y}(t)}{(T - t)^{p - 1}} = Cp\nabla f(Y(T))$ . The (7) form of process  $Y$  is
$$
\dot {Y} (t) = - \int_ {0} ^ {t} \frac {C p ^ {2} (T - t) ^ {2 p - 1}}{(T - s) ^ {p + 1}} \nabla f (Y (s)) d s.
$$
By dividing  $(T - t)^{p - 1}$  each side, we obtain
$$
\frac {\dot {Y} (t)}{(T - t) ^ {p - 1}} = - (T - t) ^ {p} \int_ {0} ^ {t} \frac {C p ^ {2}}{(T - s) ^ {p + 1}} \nabla f (Y (s)) d s.
$$
By the result of C.3, we can apply L'Hopital's rule, which gives
$$
\lim  _ {t \to T -} \frac {\dot {Y} (t)}{(T - t) ^ {p - 1}} = - \lim  _ {t \to T -} \frac {\int_ {0} ^ {t} \frac {C p ^ {2}}{(T - s) ^ {p + 1}} \nabla f (Y (s)) d s}{(T - t) ^ {- p}} = C p \nabla f (Y (T)).
$$
Also, since  $\nabla f$  is  $L$ -l=Lipshitz,  $\| \nabla f(Y(t)) - \nabla f(Y(T))\| \leq L\| Y(t) - Y(T)\|$ . Therefore,
$$
\lim  _ {t \rightarrow T -} \frac {\nabla f (Y (t)) - \nabla f (Y (T))}{(T - t) ^ {\beta}} = 0 \tag {84}
$$
for any  $\beta <  p$
Applying Theorem 2 to (10). For the case  $p = 2$ , refer [51]. Now we consider the case  $p > 2$  with  $u(t) = Ct^p$  and  $v(t) = \frac{1}{C(T - t)^p}$ . First, we verify conditions (i) and (ii) in Theorem 6. (i) holds since  $u(0) = 0$ , and (ii) holds since
$$
\begin{array}{l} \lim  _ {s \rightarrow T -} v (s) \left(f (Y (s)) - f (Y (T)) + \langle \nabla f (Y (T)), Y (T) - Y (s) \rangle\right) \\ = \frac {1}{C} \lim  _ {s \rightarrow T -} \frac {f (Y (s)) - f (Y (T)) + \langle \nabla f (Y (T)) , Y (T) - Y (s) \rangle}{(T - s) ^ {p}} \\ \stackrel {(\circ)} {=} \frac {1}{C} \lim  _ {s \rightarrow T -} \frac {\left\langle \dot {Y} (s) , \nabla f (Y (s)) - \nabla f (Y (T)) \right\rangle}{- p (T - s) ^ {p - 1}} \\ \stackrel {(\bullet)} {=} - \frac {1}{p C} \lim _ {s \to T -} \left\langle \frac {\dot {Y} (s)}{(T - s) ^ {p - 1}}, \nabla f (Y (s)) - \nabla f (Y (T)) \right\rangle \\ = 0. \\ \end{array}
$$
$(\circ)$  uses L'Hospital's rule and  $(\bullet)$  uses the limit results at the previous paragraph.
To prove
$$
\frac {1}{2} \left\| \nabla f (Y (T)) \right\| ^ {2} \leq \frac {1}{C T ^ {p}} \left(f (Y (0)) - f (Y (T))\right),
$$
we carefully check that for any  $L$ -smooth convex function  $f, \{\nabla f(Y(s))\}_{s \in [0,T]} \in \mathcal{A}_2$ .
Verification of (80). Fubini's Theorem is used for
$$
\begin{array}{l} \int_ {0 \leq s, a \leq T} \mathbf {1} _ {s \leq a} \frac {u ^ {\prime} (T - s)}{u (T - s) ^ {2}} \left\langle \dot {Y} (a), (\nabla f (Y (s)) - \nabla f (Y (T))) \right\rangle \\ = \int_ {0 \leq s, a \leq T} \mathbf {1} _ {s \leq a} \dot {Y} (a) (\nabla f (Y (s)) - \nabla f (Y (T))) \frac {p}{C (T - s) ^ {p + 1}}. \\ \end{array}
$$
The above function is continuous in  $0 \leq s, a \leq T$  due to  $\lim_{t \to T-} \frac{\dot{Y}(t)}{(T - t)^{p-1}} = C p \nabla f(Y(T))$  and (84) with  $\beta = 2 < p$ .
Verification of (79). First,
$$
\lim  _ {t \rightarrow T -} \frac {Y (t) - Y (T)}{(T - t) ^ {p}} = \lim  _ {t \rightarrow T -} \frac {\dot {Y} (t)}{- p (T - t) ^ {p - 1}} = - C \nabla f (Y (T))
$$
holds from (iv). Thus  $\sup_{b\in (0,T]}\left\| \frac{Y(T - b) - Y(T)}{b^p}\right\| <  \infty$  . Now we will show
$$
\lim  _ {a \to + 0} a ^ {1 + \epsilon} f _ {a} = 0, \quad \forall \epsilon > 0.
$$
$$
f _ {a} = \frac {1}{C T ^ {p}} g _ {0} + \frac {1}{C T ^ {p}} \left(g _ {a} - g _ {0}\right) - \int_ {a} ^ {T} \frac {p}{C b ^ {p + 1}} \left(g _ {b} - g _ {a}\right) d b
$$
$$
\begin{array}{l} \left\| a ^ {1 + \epsilon} f _ {a} \right\| = \left\| \frac {a ^ {1 + \epsilon}}{C T ^ {p}} g _ {0} + \frac {a ^ {1 + \epsilon}}{C T ^ {p}} (g _ {a} - g _ {0}) - \int_ {a} ^ {T} \frac {p a ^ {1 + \epsilon}}{C b ^ {p + 1}} (g _ {b} - g _ {a}) d b \right\| \\ \leq \left\| \frac {a ^ {1 + \epsilon}}{C T ^ {p}} g _ {0} + \frac {a ^ {1 + \epsilon}}{C T ^ {p}} \left(g _ {a} - g _ {0}\right) \right\| + \int_ {a} ^ {T} \frac {p a ^ {1 + \epsilon}}{C b ^ {p + 1}} \| g _ {b} - g _ {0} \| d b + \int_ {a} ^ {T} \frac {p a ^ {1 + \epsilon}}{C b ^ {p + 1}} \| g _ {a} - g _ {0} \| d b \\ \leq \left\| \frac {a ^ {1 + \epsilon}}{C T ^ {p}} g _ {0} + \frac {a ^ {1 + \epsilon}}{C T ^ {p}} (g _ {a} - g _ {0}) \right\| + \int_ {a} ^ {T} \frac {L p a ^ {1 + \epsilon}}{C b ^ {p + 1}} \| Y (T - b) - Y (T) \| d b + \int_ {a} ^ {T} \frac {L p a ^ {1 + \epsilon}}{C b ^ {p + 1}} \| Y (T - a) - Y (T) \| d b \\ \leq \left\| \frac {a ^ {1 + \epsilon}}{C T ^ {p}} g _ {0} + \frac {a ^ {1 + \epsilon}}{C T ^ {p}} (g _ {a} - g _ {0}) \right\| + \int_ {a} ^ {T} L p a ^ {\epsilon} \left\| \frac {Y (T - b) - Y (T)}{b ^ {p}} \right\| d b \\ + \frac {L a ^ {1 + \epsilon}}{C} \left(\frac {1}{a ^ {p}} - \frac {1}{T ^ {p}}\right) \| Y (T - a) - Y (T) \|. \\ \end{array}
$$
By using the boundness of  $\frac{Y(T - b) - Y(T)}{b^p}$ , we obtain the desired result.
$$
\begin{array}{l} \left\| a ^ {2 \epsilon - p + 2} \dot {X} (a) \right\| = \left\| \int_ {0} ^ {a} a ^ {\epsilon - p + 2} H (a, b) f _ {b} d b \right\| \\ \leq \int_ {0} ^ {a} \frac {C p ^ {2} b ^ {2 p - 1}}{a ^ {2 p - 1 - 2 \epsilon}} \| f _ {b} \| d b \\ = \int_ {0} ^ {a} \frac {C p ^ {2} b ^ {2 p - 2 - \epsilon}}{a ^ {2 p - 1 - 2 \epsilon}} \left\| b ^ {1 + \epsilon} f _ {b} \right\| d b \\ = \left(\sup  _ {b \in [ 0, a ]} \left\| b ^ {1 + \epsilon} f _ {b} \right\|\right) \frac {C p ^ {2}}{2 p - 1 - \epsilon} a ^ {\epsilon}, \\ \end{array}
$$
$$
\lim  _ {a \rightarrow + 0} a ^ {2 \epsilon - p + 2} \dot {X} (a) = 0, \quad \forall 0 <   \epsilon <   2 p - 1.
$$
For the final step, recall that Fubini's Theorem is used for
$$
\int_ {0 \leq a, s \leq T} \mathbf {1} _ {a \leq s} s ^ {p - 1} \left\langle f _ {s}, \dot {X} (a) \right\rangle d a d s.
$$
To prove that the above function is continuous, take any  $0 < \epsilon < \frac{p - 2}{2}$ . Then
$$
\lim_{0\leq a\leq s,s\to +0}\left\| s^{p - 1}\left\langle f_{s},\dot{X} (a)\right\rangle \right\| \leq \lim_{0\leq a\leq s,s\to +0}\left\| \frac{s^{-2\epsilon + 2p - 3}}{a^{-2\epsilon + p - 2}}\left\langle \dot{X} (a),f_{s}\right\rangle \right\| = \lim_{s\to +0}\left\| s^{-2\epsilon +2p - 3}f_{s}\right\| \cdot \lim_{a\to +0}\left\| \frac{\dot{X}(a)}{a^{-2\epsilon + p - 2}}\right\| = 0,
$$
D Omitted calculation in Section 4
General proximal step and composite FSFOM For  $a > 0$ , we newly define a prox-grad step with step size  $\frac{1}{L}\frac{1}{\alpha}$  as follows:
$$
y ^ {\oplus , \alpha} := \underset {z \in \mathbb {R} ^ {d}} {\operatorname {a r g m i n}} \left(f (y) + \langle \nabla f (y), z - y \rangle + g (z) + \frac {\alpha L}{2} \| z - y \| ^ {2}\right). \tag {85}
$$
First, we will provide a generalized version of prox-grad inequality; Prox-grad inequality is originally used to prove the convergence of FISTA as below [11]:
$$
F (y ^ {\oplus}) - L \left\langle y ^ {\oplus , 1} - y, x - y ^ {\oplus , 1} \right\rangle \leq F (x) + \frac {L}{2} \| y ^ {\oplus , 1} - y \| ^ {2}.
$$
Proposition 4 (Generalized version of prox-grad inequality). For every  $x, y \in \mathbb{R}^d$ , the following holds:
$$
(| x, y |) ^ {\alpha} \colon = F (y ^ {\oplus , \alpha}) - F (x ^ {\oplus , \alpha}) - L a \left\langle y ^ {\oplus , \alpha} - y, x ^ {\oplus , \alpha} - y ^ {\oplus , \alpha} \right\rangle - \frac {L}{2} \| y ^ {\oplus , \alpha} - y \| ^ {2} \leq 0.
$$
Proof. The optimality condition of (85) provides
$$
L \alpha (y ^ {\oplus , \alpha} - y) + \nabla f (y) + u = 0, \quad u \in \partial g (y ^ {\oplus , \alpha}). \tag {86}
$$
In addition,  $L$ -smoothness of  $f$  and convexity inequality of  $f$  and  $g$  give
$$
F (y ^ {\oplus , \alpha}) \leq f (y) + \left\langle \nabla f (y), y ^ {\oplus , \alpha} - y \right\rangle + \frac {L}{2} \| y ^ {\oplus , \alpha} - y \| ^ {2} + g (y ^ {\oplus , \alpha})
$$
$$
g \left(y ^ {\oplus , \alpha}\right) + \left\langle u, x ^ {\oplus , \alpha} - y ^ {\oplus , \alpha} \right\rangle = g \left(y ^ {\oplus , \alpha}\right) + \left\langle - L \alpha \left(y ^ {\oplus , \alpha} - y\right) - \nabla f (y), x ^ {\oplus , \alpha} - y ^ {\oplus , \alpha} \right\rangle \leq g \left(x ^ {\oplus , \alpha}\right),
$$
$$
f (y) + \left\langle \nabla f (y), x ^ {\oplus , \alpha} - y \right\rangle \leq f \left(x ^ {\oplus , \alpha}\right).
$$
Summing the above inequalities provides the generalized version of prox-grad inequality.
Two problem settings for the composite optimization problem. For the composite optimization problem, we consider the following two problems.
(P1') Efficiently reduce  $F(x_{N}^{\oplus,\alpha}) - F_{\star}$  assuming  $x_{\star}$  exists and  $\| x_0 - x_\star \| \leq R$ .
(P2') Efficiently reduce  $\min_{v\in \partial F(y_N^{\oplus ,\alpha})}\| v\|^2$  assuming  $F_{\star} > - \infty$  and  $F(y_0) - F_{\star}\leq R.$
Note that when  $g = 0$ , (P1') and (P2') collapse to (P1) and (P2), respectively.
Parameterized FSFOM that reduces the composite function value : GFPGM [27] First define  $x_{\star} \coloneqq \operatorname*{argmin}_{x \in \mathbb{R}^d} F(x)$ . We recall the composite function minimization FSFOMs with  $x_i^{\oplus, \alpha}$  by a lower triangular matrix  $\{h_{k,i}\}_{0 \leq i < k \leq N}$  as follows:
$$
x _ {k + 1} = x _ {k} - \sum_ {i = 0} ^ {k} \alpha h _ {k + 1, i} \left(x _ {i} - x _ {i} ^ {\oplus , \alpha}\right), \quad \forall k = 0, \dots , N - 1. \tag {87}
$$
In the case  $g = 0$ , (87) collapses to (1) as  $\alpha(z - z^{\oplus, \alpha}) = (z - z^{+}) = \frac{1}{L} \nabla f(z)$ . The iterations of GFGPM are defined as
$$
x _ {k + 1} = x _ {k} ^ {\oplus , 1} + \frac {\left(T _ {k} - t _ {k}\right) t _ {k + 1}}{t _ {k} T _ {k + 1}} \left(x _ {k} ^ {\oplus , 1} - x _ {k - 1} ^ {\oplus , 1}\right) + \frac {\left(t _ {k} ^ {2} - T _ {k}\right) t _ {k + 1}}{t _ {k} T _ {k + 1}} \left(x _ {k} ^ {\oplus , 1} - x _ {k}\right), \quad k = 0, \dots , N - 1 \tag {88}
$$
where  $x_{-1}^{\oplus,1} = x_0$  and  $[t_i > 0, T_i = \sum_{j=0}^{i} t_j \leq t_i^2 \text{ for } 0 \leq i \leq N]$ . (88) reduces the composite function value as follows:
$$
F \left(x _ {N} ^ {\oplus , 1}\right) - F _ {\star} \leq \frac {1}{T _ {N}} \frac {1}{2} \left\| x _ {0} - x _ {\star} \right\| ^ {2}. \tag {89}
$$
We can reconstruct the convergence proof of [27, Theorem 3.3] via our energy function scheme, which is defined as
$$
\mathcal {U} _ {k} = \frac {L}{2} \| x _ {0} - x _ {\star} \| ^ {2} + \sum_ {i = 0} ^ {k - 1} u _ {i} (x _ {i}, x _ {i + 1}) ^ {1} + \sum_ {i = 0} ^ {k} (u _ {i} - u _ {i - 1}) (x _ {\star}, x _ {i}) ^ {1} \tag {90}
$$
for  $k = -1, \dots, N$ . The convergence rate result (89) is proved by using that  $\{\mathcal{U}_k\}_{k = -1}^N$  is dissipative, and
$$
\mathcal {U} _ {N} - T _ {N} \left(F \left(x _ {N} ^ {\oplus , 1}\right) - F _ {\star}\right) = \frac {L}{2} \left\| x _ {0} - x _ {\star} + \sum_ {i = 0} ^ {N} \left(u _ {i} - u _ {i - 1}\right) \left(x _ {i} - x _ {i} ^ {\oplus , 1}\right) \right\| ^ {2} + \sum_ {i = 0} ^ {N} \frac {L \left(T _ {i} - t _ {i} ^ {2}\right)}{2} \left\| x _ {i} ^ {\oplus , 1} - x _ {i} \right\| ^ {2} \geq 0. \tag {91}
$$
Here, we can make a crucial observation.
$$
\text {T h e} (9 0) \text {i s} (\cdot , \cdot ] \text {a r e c h a n g e d i n t o} (\cdot , \cdot) ^ {1} \text {a n d} u _ {i} = T _ {i}.
$$
Moreover, we can express the recursive formula of the  $H$  matrix of (88) as
$$
h _ {k + 1, i} = \left\{ \begin{array}{l l} 1 + \frac {\left(t _ {k} - 1\right) t _ {k + 1}}{T _ {k + 1}} & i = k \\ \frac {\left(T _ {k} - t _ {k}\right) t _ {k + 1}}{t _ {k} T _ {k + 1}} \left(h _ {k, k - 1} - 1\right) & i = k - 1 \\ \frac {\left(T _ {k} - t _ {k}\right) t _ {k + 1}}{t _ {k} T _ {k + 1}} h _ {k, i} & i = 0, \dots , k - 2 \end{array} . \right. \tag {92}
$$
As we used a parameterized family in the convex function (see Section 2.5), we will use these convergence results to construct the method for minimizing  $\min_{v\in \partial F(y_N^{\oplus ,\alpha})}\| v\|^2$
Relationship between minimizing  $\frac{\alpha^2}{2L}\left\| y_N^{\oplus ,\alpha} - y_N\right\|^2$  and minimizing  $\min_{v\in \partial F(y_N^{\oplus ,\alpha})}\| v\|^2$
Note that  $L\alpha (y_N^{\oplus ,\alpha} - y_N) + \nabla f(y_N) + u = 0$ ,  $u\in \partial g(y_N^{\oplus ,\alpha})$ , which gives
$$
\min  _ {v \in \partial F \left(y _ {N} ^ {\oplus , \alpha}\right)} \| v \| ^ {2} \leq (\bullet) \left(\left\| \nabla f \left(y _ {N}\right) - \nabla f \left(y _ {N} ^ {\oplus , \alpha}\right) \right\| + L \alpha \left\| y _ {N} - y _ {N} ^ {\oplus , \alpha} \right\|\right) ^ {2} \leq (\circ) L ^ {2} (\alpha + 1) ^ {2} \left\| y _ {N} - y _ {N} ^ {\oplus , \alpha} \right\| ^ {2}. \tag {93}
$$
$(\bullet)$  is triangle inequality and  $(\circ)$  comes from the  $L$ -smoothness of  $f$ .
D.2 Proof of Theorem 3
In this section, we give the proof outline of Theorem 3 and discuss the construction of matrix  $C$ .
Proof outline of Theorem 3. We begin by proposing a parameterized family reduces  $\frac{1}{2L}\left\| y_N^{\oplus ,\alpha} - y_N\right\|^2$  under a fixed value of  $a > 0$ . To construct this family, take  $\{t_i\}_{i = 0}^N$ ,  $T_{i} = \sum_{j = 0}^{i}t_{j}$  that satisfies (96). We define the  $H$  matrix of the family as  $\left(\frac{1}{\alpha} H_0 + \frac{1}{\alpha^2} C\right)^A$ , where  $H_{0}$  has the same formulation as the  $H$  matrix of GFGPM (88) and  $C$  follows the recursive formula (97). We refer to this family as (SFG-family).
We can prove that (SFG-family) exhibits the convergence rate
$$
\frac {\alpha L}{2} \left\| y _ {N} ^ {\oplus , \alpha} - y _ {N} \right\| ^ {2} \leq \frac {1}{T _ {N}} \left(F \left(y _ {0}\right) - F _ {\star}\right), \tag {94}
$$
which is motivated by the observation  $(\diamond)$ . In parallel with  $(\diamond)$ , we consider the energy function
$$
\mathcal {V} _ {k} = \frac {F (y _ {0}) - F (y _ {N} ^ {\oplus , \alpha}) + (\left\lfloor y _ {- 1} , y _ {0} \right\rfloor) ^ {\alpha}}{T _ {N}} + \sum_ {i = 0} ^ {k - 1} \frac {1}{T _ {N - i - 1}} (\left\lfloor y _ {i}, y _ {i + 1} \right\rfloor) ^ {\alpha} + \sum_ {i = 0} ^ {k - 1} \left(\frac {1}{T _ {N - i - 1}} - \frac {1}{T _ {N - i}}\right) (\left\lfloor y _ {N}, y _ {i} \right\rfloor) ^ {\alpha}
$$
for  $k = 0,\dots ,N$  , which  $\llbracket y_N,\star \rrbracket$  changed into  $(y_{-1},y_0)$  , all other  $\llbracket \cdot ,\cdot \rrbracket$  terms changed into  $(\cdot ,\cdot)^{\alpha}$  and  $v_{i} = \frac{1}{T_{N - i}}$
(94) follows from the fact that  $\{\mathcal{V}_i\}_{i = 0}^N$  is dissipative and
$$
\frac {\alpha L}{2} \left\| y _ {N} ^ {\oplus , \alpha} - y _ {N} \right\| ^ {2} \leq \mathcal {V} _ {N}.
$$
Detailed justification is given in Appendix D.3.
For the next step, we show that (SFG) is an instance of (SFG-family), under the choice of
$$
\alpha = 4, \quad T _ {i} = \frac {(i + 2) (i + 3)}{4}, \quad 0 \leq i \leq N.
$$
The detailed derivation is given in Appendix D.4.
Finally, combining the convergence result
$$
2 L \left\| y _ {N} ^ {\oplus , 4} - y _ {N} \right\| ^ {2} \leq \mathcal {V} _ {N} \leq \mathcal {V} _ {0} \leq \frac {4}{(N + 2) (N + 3)} \left(F \left(y _ {0}\right) - F _ {\star}\right)
$$
and (93) gives
$$
\min  _ {v \in \partial F (y _ {N} ^ {\oplus , 4})} \| v \| ^ {2} \leq 2 5 L ^ {2} \left\| y _ {N} ^ {\oplus , 4} - y _ {N} \right\| ^ {2} \leq \frac {5 0}{(N + 2) (N + 3)} \left(F (y _ {0}) - F _ {\star}\right).
$$
Construction of matrix  $C$ . Define
$$
g \colon = L \alpha \left[ y _ {0} - y _ {0} ^ {\oplus , \alpha} \right| \dots \left| y _ {N} - y _ {N} ^ {\oplus , \alpha} \right].
$$
For an FSFOM (87) with matrix  $H_{1}$ , we have  $\mathcal{V}_N - \frac{\alpha L}{2} \|y_N^{\oplus,\alpha} - y_N\|^2 = \frac{1}{2L\alpha^2} \operatorname{Tr}(g^\intercal g\mathcal{T}^\oplus)$  with
$$
\begin{array}{l} \mathcal {T} ^ {\oplus}: = \underbrace {\alpha^ {2} \left[ \sum_ {i = 0} ^ {N} \frac {1}{T _ {N - i}} \left(\mathcal {H} _ {1} ^ {\intercal} (\mathbf {e} _ {i} + \cdots + \mathbf {e} _ {N}) (\mathbf {e} _ {i} - \mathbf {e} _ {i - 1}) ^ {\intercal} + (\mathbf {e} _ {i} - \mathbf {e} _ {i - 1}) (\mathbf {e} _ {i} + \cdots + \mathbf {e} _ {N}) ^ {\intercal} \mathcal {H} _ {1}\right) \right]} _ {\mathcal {A} _ {3}} \\ + \underbrace {\alpha \left[ \sum_ {i = 0} ^ {N} \frac {1}{T _ {N - i}} \left((\mathbf {e} _ {i - 1} - \mathbf {e} _ {i}) (\mathbf {e} _ {i - 1} - \mathbf {e} _ {N}) ^ {\intercal} + (\mathbf {e} _ {i - 1} - \mathbf {e} _ {N}) (\mathbf {e} _ {i - 1} - \mathbf {e} _ {i}) ^ {\intercal}\right) - \frac {1}{T _ {N}} \mathbf {e} _ {0} \mathbf {e} _ {0} ^ {\intercal} - \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal} \right]} _ {\mathcal {B} _ {3}} \\ + \left[ \frac {\alpha}{T _ {N}} \mathbf {e} _ {0} \mathbf {e} _ {0} ^ {\intercal} - \sum_ {i = 0} ^ {N - 1} \frac {1}{T _ {N - 1 - i}} \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal} - \frac {1}{T _ {0}} \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal} \right] \\ \end{array}
$$