SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} (u _ {i _ {0} + 1} - u _ {i _ {0}}) \sum_ {l = j} ^ {i _ {0}} h _ {l + 1, j} + u _ {i _ {0}} h _ {i _ {0} + 1, j} - (u _ {i _ {0} + 1} - u _ {i _ {0}}) (u _ {j} - u _ {j - 1}) \\ = \left(u _ {i _ {0} + 1} - u _ {i _ {0}}\right) \left(\sum_ {l = j} ^ {i _ {0} - 1} h _ {l + 1, j} + h _ {i _ {0} + 1, j}\right) + u _ {i _ {0}} h _ {i _ {0} + 1, j} - \left(u _ {i _ {0} + 1} - u _ {i _ {0}}\right) \left(u _ {j} - u _ {j - 1}\right) \\ = \left(u _ {i _ {0} + 1} - u _ {i _ {0}}\right) \left(\frac {\left(u _ {i _ {0}} - u _ {i _ {0} - 1}\right) \left(u _ {j} - u _ {j - 1}\right) - u _ {i _ {0} - 1} h _ {i _ {0} , j}}{u _ {i _ {0}} - u _ {i _ {0} - 1}} + h _ {i _ {0} + 1, j}\right) + u _ {i _ {0}} h _ {i _ {0} + 1, j} \\ - \left(u _ {i _ {0} + 1} - u _ {i _ {0}}\right) \left(u _ {j} - u _ {j - 1}\right) \\ = u _ {i _ {0} + 1} h _ {i _ {0} + 1, j} - \frac {u _ {i _ {0} - 1} \left(u _ {i _ {0} + 1} - u _ {i _ {0}}\right)}{u _ {i _ {0}} - u _ {i _ {0} - 1}} h _ {i _ {0}, j} \\ = \left\{ \begin{array}{l} \frac {(i _ {0} + 2) (i _ {0} + 3)}{2} h _ {i _ {0} + 1, j} - \frac {i _ {0} (i _ {0} + 1) (i _ {0} + 2)}{2 (i _ {0} + 1)} h _ {i _ {0}, j} \quad 0 \leq i _ {0} \leq N - 2 \\ (\gamma^ {2} + \gamma) h _ {N, j} - \frac {(N - 1) N \gamma}{2 N} h _ {N - 1, j} \quad i _ {0} = N - 1 \end{array} \right. \\ = 0. \\ \end{array}
$$
Next, we calculate  $\{t_{ij}\}$  for (OBL-  $\mathbf{G}_b$ ). We need to deal with the sum  $\sum_{l = k}^{N - 1}h_{l + 1,i}$ , which can be expressed as
$$
\sum_ {l = i} ^ {N - 1} h _ {l + 1, j} = \left\{ \begin{array}{l l} 1 + \frac {(N + 2) (N - 1)}{4 (\gamma + 1)} & \quad i = 0,   j = 0 \\ \frac {(N - i + 2) (N - i + 1) (N - i) (N - i - 1)}{4 (\gamma + 1) N (N + 1)} & \quad i = 1, \ldots , N - 1,   j = 0 \\ \frac {(N - i + 2) (N - i + 1) (N - i) (N - i - 1)}{2 (N - j) (N - j + 1) (N - j + 2)} & \quad i = j + 1, \ldots , N - 1,   j = 1, \ldots , N - 1 \\ 1 + \frac {N - i - 1}{2} & \quad i = j,   j = 1, \ldots , N - 1 \end{array} \right..
$$
By combining  $v_{0} = \frac{1}{\gamma^{2} + \gamma}$ ,  $v_{i} = \frac{1}{(N - i + 1)(N - i + 2)}$  for  $1 \leq i \leq N$  and (35), we obtain
$$
\begin{array}{l} t _ {i j} = \left\{ \begin{array}{l l} 1 & \quad i = N,   j = N \\ \frac {1}{2 N (N + 1)} - \frac {v _ {0}}{2} & \quad i = 0,   j = 1 \\ v _ {0} - \frac {1}{N (N + 1)} & \quad i = N,   j = 0 \\ \frac {1}{(N - i) (N - i + 1) (N - i + 2)} & \quad i = 1, \ldots , N - 1,   j = i \\ - \frac {2}{(N - i) (N - i + 1) (N - i + 2)} & \quad i = N,   j = 1, \ldots , N - 1 \\ 0 & \quad \text {o t h e r w i s e} \end{array} \right. \\ = \left\{ \begin{array}{l l} \frac {v _ {N}}{2} & \quad i = N, j = N \\ \frac {v _ {i + 1} - v _ {i}}{2} & \quad i = 0, \ldots , N - 1, j = i \\ - v _ {i + 1} + v _ {i} & \quad i = N, j = 0, \ldots , N - 1 \\ 0 & \quad \text {o t h e r w i s e} \end{array} \right. \\ \end{array}
$$
$$
\mathbf {V} = \sum_ {0 \leq j \leq i \leq N} \frac {t _ {i j}}{L} \left\langle \nabla f (y _ {i}), \nabla f (y _ {j}) \right\rangle = \frac {v _ {0}}{2 L} \| \nabla f (y _ {N}) \| ^ {2} + \sum_ {i = 0} ^ {N - 1} \frac {v _ {i + 1} - v _ {i}}{2 L} \left\| \nabla f (y _ {i}) - \nabla f (y _ {N}) \right\| ^ {2}.
$$
B.2.3 Calculation of energy function of GD
We calculate  $\{t_{ij}\}$  first. Recall that  $\{v_i\}_{i=0}^N = \left(\frac{1}{2N+1}, \ldots, \frac{N+i}{(2N+1)(N-i+1)}, \ldots\right)$  and  $h_{i+1,k} = \delta_{i,k}$  to (35), and making  $\{t_{ij}\}$  symmetric gives us
$$
t _ {i j} = \left\{ \begin{array}{l l} \frac {1}{2} v _ {0} & i = j,   i = 0 \\ v _ {i} & i = j,   1 \leq i \leq N - 1 \\ v _ {N} - \frac {1}{2} & i = j,   i = N \\ \frac {1}{2} \left(v _ {\min (i, j)} - v _ {\min (i, j) + 1}\right) & i \neq j. \end{array} \right.
$$
We can verify that the matrix  $\{t_{ij}\}_{0\leq i,j\leq N}$  is diagonally dominant:  $t_{ii} = |\sum_{j\neq i}t_{ij}|$ . Therefore,  $\sum_{0\leq i,j\leq N - 1}\frac{t_{ij}}{L}\langle \nabla f(y_i),\nabla f(y_j)\rangle \geq 0$  for any  $\{\nabla f(y_i)\}_{i = 0}^N$ . This proof is essentially the same as the proof in [29], but we repeat it here with our notation for the sake of completeness.
Next, we prove that (GD) with  $h = 1$  and  $\{u_i\}_{i=0}^N = \left( \dots, \frac{(2N+1)(i+1)}{2N-i}, \dots, 2N+1 \right)$  satisfies (C1), by showing more general statement:
$$
\left[ \left(\mathrm {G D}\right) \text {a n d} \left\{u _ {i} \right\} _ {i = 0} ^ {N} = \left(\dots , \frac {(2 N h + 1) (i + 1)}{2 N - i}, \dots , 2 N h + 1\right) \text {s a t i s f i e s} (\mathrm {C} 1) \right] \tag {39}
$$
Note that (39) gives
$$
(2 N h + 1) \left(f \left(x _ {N}\right) - f ^ {\star}\right) \leq \mathcal {U} _ {N} \leq \mathcal {U} _ {- 1} = \frac {L}{2} \| x _ {0} - x ^ {\star} \| ^ {2}. \tag {40}
$$
Later in the proof of Corollary 2, we will utilize the equation (39). The result (39) is proved in [19, Theorem 3.1], and we give the proof outline here.
In order to demonstrate (39), we will directly expand the expression  $\mathcal{U}_N - u_N(f(x_N) - f^{\star})$ , instead of employing  $\mathbf{U}$  as a intermediary step. Define  $\{u_i'\}_{i=0}^N := \left( \dots, \frac{(2N+1)(i+1)}{2N-i}, \dots, 2N+1 \right)$ . Then
$$
\mathcal {U} _ {N} - u _ {N} \left(f (x _ {N}) - f ^ {\star}\right) = \frac {1}{L} \operatorname {T r} \left(g ^ {\intercal} g S\right)
$$
where  $g = [\nabla f(x_0)|\ldots |\nabla f(x_N)|L(x_0 - x^\star)] \in \mathbb{R}^{d \times (N + 2)}$  and  $S \in \mathbb{S}^{N + 2}$  is given by
$$
S = \left[ \begin{array}{c c} S ^ {\prime} & \lambda \\ \lambda & \frac {1}{2} \end{array} \right],
$$
$$
\lambda = [ u _ {0} | u _ {1} - u _ {0} | \dots | u _ {N} - u _ {N - 1} ] ^ {\intercal}, S ^ {\prime} = \frac {2 N h + 1}{2 N + 1} (h S _ {0} + (1 - h) S _ {1}),
$$
$$
\begin{array}{l} S _ {0} = \sum_ {i = 0} ^ {N - 1} \frac {u _ {i} ^ {\prime}}{2} \left(e _ {i + 1} e _ {i} ^ {\intercal} + e _ {i} e _ {i + 1} ^ {\intercal} + (e _ {i} - e _ {i + 1}) (e _ {i} - e _ {i + 1}) ^ {\intercal}\right) \\ + \sum_ {i = 0} ^ {N} \frac {u _ {i} ^ {\prime} - u _ {i - 1} ^ {\prime}}{2} (e _ {i} (e _ {0} + \dots + e _ {i - 1}) ^ {\intercal} + (e _ {0} + \dots + e _ {i - 1}) e _ {i} ^ {\intercal} + e _ {i} e _ {i} ^ {\intercal}) \\ \end{array}
$$
$$
S _ {1} = \sum_ {i = 0} ^ {N - 1} \frac {u _ {i} ^ {\prime}}{2} (e _ {i} - e _ {i + 1}) (e _ {i} - e _ {i + 1}) ^ {\intercal} + \sum_ {i = 0} ^ {N} \frac {u _ {i} ^ {\prime} - u _ {i - 1} ^ {\prime}}{2} e _ {i} e _ {i} ^ {\intercal}.
$$
Now we will show  $S \succeq 0$  to obtain  $\left[\mathcal{U}_N - u_N\left(f(x_N) - f^\star\right) \geq 0, \forall g\right]$ , which is (C1). By using Sylvester's Criterion,  $S_0 \succ 0$  follows.  $S_1 \succ 0$  follows from the fact that  $S_1$  expressed by the sum of positive semi-definite matrices  $zz^{\intercal}$ . Since the convex sum of two positive semi-definite matrices is also positive semi-definite,  $S' = hS_0 + (1 - h)S_1 \succ 0$ .
Next, we argue that  $\det S = 0$ . Indeed, take  $\tau = (1,\dots, - (2Nh + 1))^{\intercal}$  to show  $S\tau = 0$ , which gives  $\det S = 0$ . Note that the determinant of  $S$  can also be expressed by
$$
\det  (S) = \left(\frac {1}{2} - \lambda^ {\mathsf {T}} \left(S ^ {\prime}\right) ^ {- 1} \lambda\right) \det  \left(S ^ {\prime}\right). \tag {41}
$$
We have shown that  $S' \succ 0$ , (41) implies  $\frac{1}{2} - \lambda^{\mathsf{T}}(S')^{-1}\lambda = 0$ , which is the Schur complement of the matrix  $S$ . By a well-known lemma on the Schur complement, we conclude  $S \succeq 0$ .
B.3 Omitted proof in Section 2.5
B.3.1 Omitted calculation of Corollary 1
Here, we will give the general formulation of H-dual FSFOM of
$$
x _ {k + 1} = x _ {k} + \beta_ {k} \left(x _ {k} ^ {+} - x _ {k - 1} ^ {+}\right) + \gamma_ {k} \left(x _ {k} ^ {+} - x _ {k}\right), \quad k = 0, \dots , N - 1. \tag {42}
$$
Proposition 1. The H-dual of (42) is
$$
y _ {k + 1} = y _ {k} + \beta_ {k} ^ {\prime} \left(y _ {k} ^ {+} - y _ {k - 1} ^ {+}\right) + \gamma_ {k} ^ {\prime} \left(y _ {k} ^ {+} - y _ {k}\right), \quad k = 0, \dots , N - 1 \tag {43}
$$
$$
\beta_ {k} ^ {\prime} = \frac {\beta_ {N - k} \left(\beta_ {N - 1 - k} + \gamma_ {N - 1 - k}\right)}{\beta_ {N - k} + \gamma_ {N - k}}
$$
$$
\gamma_ {k} ^ {\prime} = \frac {\gamma_ {N - k} (\beta_ {N - 1 - k} + \gamma_ {N - 1 - k})}{\beta_ {N - k} + \gamma_ {N - k}}
$$
for  $k = 0,\dots ,N - 1$  and  $(\beta_{N},\gamma_{N})$  is any value that  $\beta_N + \gamma_N\neq 0$
Proof. The  $H$  matrix  $\{h_{k,i}\}_{0\leq i < k\leq N}$  satisfies
$$
h _ {k + 1, i} = \left\{ \begin{array}{l l} 1 + \beta_ {k} + \gamma_ {k} & i = k, k = 0, \ldots , N - 1 \\ \beta_ {k} (h _ {k, i} - 1) & i = k - 1, k = 1, \ldots , N - 1 \\ \beta_ {k} h _ {k, i} & i = k - 2, k = 2, \ldots , N - 1 \end{array} \right..
$$
$$
h _ {k + 1, i} = \left(\prod_ {j = i + 1} ^ {k} \beta_ {j}\right) \left(\beta_ {i} + \gamma_ {i} + \delta_ {k, i}\right)
$$
where  $\delta_{k,i}$  is a Kronecker Delta function. Similarly,  $H$  matrix of (43)  $\{g_{k,i}\}_{0\leq i < k\leq N}$  satisfies
$$
\begin{array}{l} g _ {k + 1, i} = \left(\prod_ {j = i + 1} ^ {k} \beta_ {j} ^ {\prime}\right) \left(\beta_ {i} ^ {\prime} + \gamma_ {i} ^ {\prime} + \delta_ {k, i}\right) \\ = \left(\prod_ {j = i + 1} ^ {k} \frac {\beta_ {N - j} \left(\beta_ {N - 1 - j} + \gamma_ {N - 1 - j}\right)}{\beta_ {N - j} + \gamma_ {N - j}}\right) \left(\beta_ {N - 1 - i} + \gamma_ {N - 1 - i} + \delta_ {k, i}\right) \\ = \left(\prod_ {j = i + 1} ^ {k} \beta_ {N - j}\right) \left(\beta_ {N - k - 1} + \gamma_ {N - i - 1} + \delta_ {N - k - 1, N - i - 1}\right). \\ \end{array}
$$
Thus  $g_{k + 1,i} = h_{N - i,N - 1 - k}$
Now we derive the H-dual of (6) by applying Proposition 1. Note that
$$
\beta_ {k} = \frac {(T _ {k} - t _ {k}) t _ {k + 1}}{t _ {k} T _ {k + 1}}, \qquad \gamma_ {k} = \frac {(t _ {k} ^ {2} - T _ {k}) t _ {k + 1}}{t _ {k} T _ {k + 1}}, \quad k = 0, \ldots , N - 1.
$$
Next, define  $\beta_{N}$  and  $\gamma_{N}$  as a same form of  $\{\beta_i,\gamma_i\}_{i = 0}^{N - 1}$  with any  $t_{N + 1} > 0$  . Note that
$$
\beta_ {k} + \gamma_ {k} = \frac {t _ {k + 1} \left(t _ {k} ^ {2} - t _ {k}\right)}{t _ {k} T _ {k + 1}} = \frac {t _ {k + 1} \left(t _ {k} - 1\right)}{T _ {k + 1}}.
$$
By applying the formula at Proposition 1, we obtain
$$
\beta_ {k} ^ {\prime} = \frac {\left(T _ {N - k} - t _ {N - k}\right) \frac {t _ {N - k} \left(t _ {N - k - 1} - 1\right)}{T _ {N - k}}}{t _ {N - k} ^ {2} - t _ {N - k}} = \frac {\left(T _ {N - k} - t _ {N - k}\right) \left(t _ {N - k - 1} - 1\right)}{T _ {N - k} \left(t _ {N - k} - 1\right)} = \frac {T _ {N - k - 1} \left(t _ {N - k - 1} - 1\right)}{T _ {N - k} \left(t _ {N - k} - 1\right)}
$$
$$
\gamma_ {k} ^ {\prime} = \frac {(t _ {N - k} ^ {2} - T _ {N - k}) \frac {t _ {N - k} (t _ {N - k - 1} - 1)}{T _ {N - k}}}{t _ {N - k} ^ {2} - t _ {N - k}} = \frac {(t _ {N - k} ^ {2} - T _ {N - k}) (t _ {N - k - 1} - 1)}{T _ {N - k} (t _ {N - k} - 1)}.
$$
3Here, note that FSFOM (43) is independent with the any choice of  $\beta_N,\gamma_N$  since  $y_{1} = y_{0}+$ $(\beta_0^{\prime} + \gamma_0^{\prime})(y_0^{+} - y_0) = y_0 + (\beta_{N - 1} + \gamma_{N - 1})(y_0^{+} - y_0)$
B.3.2 Proof of Corollary 2
First, recall (39) when  $0 < h \leq 1$ :
$$
\left[ (\mathrm {G D}) \text {a n d} \{u _ {i} \} _ {i = 0} ^ {N} = \left(\dots , \frac {(2 N h + 1) (i + 1)}{2 N - i}, \dots , 2 N h + 1\right) \text {s a t i s f i e s} (\mathrm {C} 1) \right]
$$
Additionally, observe that the  $H$  matrix of (GD) is  $\operatorname{diag}(h, \ldots, h)$ , which gives the fact that the H-dual of (GD) is itself.
Next, use Theorem 1 to obtain
$$
\left[ (\mathrm {G D}) \text {w i t h} 0 <   h \leq 1 \text {a n d} \{v _ {i} \} _ {i = 0} ^ {N} = \left(\frac {1}{2 N h + 1}, \dots , \frac {N + i}{(2 N h + 1) (N - i + 1)}, \dots\right) \text {s a t i s f i e s (C 2)} \right]. \tag {44}
$$
By using the same argument with (5), (44) gives
$$
\frac {1}{2 L} \| \nabla f \left(y _ {N}\right) \| ^ {2} \leq \mathcal {V} _ {N} \leq \mathcal {V} _ {0} = \frac {1}{2 N h + 1} \left(f \left(y _ {0}\right) - f _ {\star}\right) + \frac {1}{2 N h + 1} \llbracket y _ {N}, y _ {\star} \rrbracket \leq \frac {1}{2 N h + 1} \left(f \left(y _ {0}\right) - f _ {\star}\right). \tag {45}
$$
In addition, we can achieve the convergence rate of the gradient norm under the initial condition of  $\| x_0 - x_\star \|^2$ :
$$
\frac {1}{2 L} \| \nabla f (x _ {2 N}) \| ^ {2} \leq \frac {1}{2 N h + 1} \left(f (x _ {N}) - f _ {\star}\right) \leq \frac {1}{(2 N h + 1) ^ {2}} \frac {L}{2} \| x _ {0} - x _ {\star} \| ^ {2}
$$
$$
\frac {1}{2 L} \| \nabla f (x _ {2 N + 1}) \| ^ {2} \leq \frac {1}{2 (N + 1) h + 1} \left(f (x _ {N}) - f _ {\star}\right) \leq \frac {1}{(2 (N + 1) h + 1) (2 N h + 1)} \frac {L}{2} \| x _ {0} - x _ {\star} \| ^ {2}.
$$
The first inequality comes from (45) and the second inequality comes from (40).
B.3.3 Proof of  $\mathcal{A}_{\star}$ -optimality of (OGM-G) and (OBL-  $\mathbf{G}_b$
Definition of  $\mathcal{A}^{\star}$ -optimal FSFOM. For the given inequality sets  $\mathcal{L}$ ,  $\mathcal{A}^{\star}$ -optimal FSFOM with respect to  $[\mathcal{L},\mathrm{P1}]$  is defined as a FSFOM which  $H$  matrix is the solution of following minimax problem:
$$
\underset {H \in \mathbb {R} _ {L} ^ {N \times N}} {\text {m i n i m i z e}} \underset {f} {\text {m a x i m i z e}} f (x _ {N}) - f _ {\star}
$$
$$
\text {s u b j e c t . t o .} [ x _ {0}, \dots , x _ {N} \text {a r e g e n e r a t e d b y F S F O M w i t h t h e m a t r i x} H ] \tag {46}
$$
$$
[ \forall l \in \mathcal {L}, f \text {s a t i s f i e s} l ]
$$
Similarly,  $\mathcal{A}^{\star}$ -optimal FSFOM with respect to  $[\mathcal{L}, (P2)]$  is specified with its  $H$  matrix, which is the solution of the following minimax problem:
$$
\underset {H \in \mathbb {R} _ {L} ^ {N \times N}} {\text {m i n i m i z e}} \underset {f} {\text {m a x i m i z e}} \frac {1}{2 L} \| \nabla f (y _ {N}) \| ^ {2}
$$
$$
\text {s u b j e c t . t o .} \left[ y _ {0}, \dots , y _ {N} \right. \text {a r e g e n e r a t e d b y F S F O M w i t h t h e m a t r i x} H ] \tag {47}
$$
$$
f (y _ {0}) - f _ {\star} \leq \frac {1}{2} L R ^ {2}
$$
Here  $\mathbb{R}_L^{N\times N}$  is the set of lower triangular matrices. Next we denote the inner maximize problem of (46) and (47) as  $\mathcal{P}_1(\mathcal{L},H,R)$  and  $\mathcal{P}_2(\mathcal{L},H,R)$ , respectively. For a more rigorous definition of  $\mathcal{A}^{\star}$ -optimal FSFOM, refer [44].
Remark. We discuss the minimax problems (46) and (47) and their interpretation. Specifically, we consider the meaning of the maximization problem in these minimax problems, which can be thought of as calculating the worst-case performance for a fixed FSFOM with  $H$ . In other words, the minimization problem in (46) and (47) can be interpreted as determining the optimal value of  $H$  that minimizes the worst-case performance.
Prior works. The prior works for  $\mathcal{A}^{\star}$ -optimality are summarized as follows. Consider the following sets of inequalities.
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {F}} = \left\{\left[ \left[ x _ {i}, x _ {i + 1} \right] \right] \right\} _ {i = 0} ^ {N - 1} \cup \left\{\left[ \left[ x _ {\star}, x _ {i} \right] \right] \right\} _ {i = 0} ^ {N} \\ = \left\{f (x _ {i}) \geq f (x _ {i + 1}) + \langle \nabla f (x _ {i + 1}), x _ {i} - x _ {i + 1} \rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) - \nabla f (x _ {i + 1}) \| ^ {2} \right\} _ {i = 0} ^ {N - 1} \\ \bigcup \left\{f _ {\star} \geq f (x _ {i}) + \langle \nabla f (x _ {i}), x _ {\star} - x _ {i} \rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) \| ^ {2} \right\} _ {i = 0} ^ {N}, \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {G}} = \{\llbracket y _ {i}, y _ {i + 1} \rrbracket \} _ {i = 0} ^ {N - 1} \cup \{\llbracket y _ {N}, y _ {i} \rrbracket \} _ {i = 0} ^ {N - 1} \cup \{\llbracket y _ {N}, \star \rrbracket \} \\ = \left\{f (y _ {i}) \geq f (y _ {i + 1}) + \langle \nabla f (y _ {i + 1}), y _ {i} - y _ {i + 1} \rangle + \frac {1}{2 L} \| \nabla f (y _ {i}) - \nabla f (y _ {i + 1}) \| ^ {2} \right\} _ {i = 0} ^ {N - 1} \\ \bigcup \left\{f (y _ {N}) \geq f (y _ {i}) + \langle \nabla f (y _ {i}), y _ {N} - y _ {i} \rangle + \frac {1}{2 L} \| \nabla f (y _ {i}) - \nabla f (y _ {N}) \| ^ {2} \right\} _ {i = 0} ^ {N - 1} \\ \bigcup \left\{f \left(y _ {N}\right) \geq f _ {\star} + \frac {1}{2 L} \| \nabla f \left(y _ {N}\right) \| ^ {2} \right\}, \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {G} ^ {\prime}} = \{\llbracket y _ {i}, y _ {i + 1} \rrbracket \} _ {i = 0} ^ {N - 1} \cup \left\{\llbracket y _ {N}, y _ {i} \rrbracket - \frac {1}{2 L} \| \nabla f (y _ {i}) - \nabla f (y _ {N}) \| ^ {2} \right\} _ {i = 0} ^ {N - 1} \cup \left\{\llbracket y _ {N}, \star \rrbracket - \frac {1}{2 L} \| \nabla f (y _ {N}) \| ^ {2} \right\} \\ = \left\{f (y _ {i}) \geq f (y _ {i + 1}) + \langle \nabla f (y _ {i + 1}), y _ {i} - y _ {i + 1} \rangle + \frac {1}{2 L} \| \nabla f (y _ {i}) - \nabla f (y _ {i + 1}) \| ^ {2} \right\} _ {i = 0} ^ {N - 1} \\ \bigcup \left\{f \left(y _ {N}\right) \geq f \left(y _ {i}\right) + \left\langle \nabla f \left(y _ {i}\right), y _ {N} - y _ {i} \right\rangle \right\} _ {i = 0} ^ {N - 1} \bigcup \left\{f \left(y _ {N}\right) \geq f _ {\star} \right\}, \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {F} ^ {\prime}} = \{\llbracket x _ {i}, x _ {i + 1} \rrbracket \} _ {i = 0} ^ {N - 1} \cup \left\{\llbracket x _ {\star}, x _ {i} \rrbracket - \frac {1}{2 L} \| \nabla f (x _ {i}) \| ^ {2} \right\} _ {i = 0} ^ {N} \\ = \left\{f (x _ {i}) \geq f (x _ {i + 1}) + \langle \nabla f (x _ {i + 1}), x _ {i} - x _ {i + 1} \rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) - \nabla f (x _ {i + 1}) \| ^ {2} \right\} _ {i = 0} ^ {N - 1} \\ \bigcup \left\{f _ {\star} \geq f \left(x _ {i}\right) + \left\langle \nabla f \left(x _ {i}\right), x _ {\star} - x _ {i} \right\rangle \right\} _ {i = 0} ^ {N}, \\ \end{array}
$$
$$
\mathcal {L} _ {\text {e x a c t}} = \{\llbracket x _ {i}, x _ {j} \rrbracket \} _ {(i, j) \in \{\star , 0, 1, \dots , N \} ^ {2}}.
$$
(OGM) is  $\mathcal{A}^{\star}$ -optimal with respect to  $[\mathcal{L}_{\mathrm{F}},(\mathrm{P1})]$  [26]. Furthermore, (OGM) is also  $\mathcal{A}^{\star}$ -optimal under  $[\mathcal{L}_{\mathrm{exact}},(\mathrm{P1})]$  which implies that (OGM) is the exact optimal FSFOM with respect to (P1). In addition, the  $\mathcal{A}^{\star}$ -optimality of (OGM-G) with respect to  $[\mathcal{L}_{\mathrm{G}},(\mathrm{P2})]$  was presented as a conjecture in [29],  $(\mathrm{OBL - F_b})$  is  $\mathcal{A}^{\star}$ -optimal with respect to  $[\mathcal{L}_{\mathrm{F'}}$ $(P1)]$  [44, Theorem 4], and the  $\mathcal{A}^{\star}$ -optimality of  $(\mathrm{OBL - G_b})$  with respect to  $[\mathcal{L}_{\mathrm{G'}}$ $(\mathrm{P2})]$  was presented as a conjecture [44, Conjecture 8].
In the remaining parts, we give the proof of the following two theorems.
Theorem 4. (OGM-G) is  $A^{\star}$ -optimal with respect to  $[\mathcal{L}_{\mathrm{G}},(\mathrm{P2})]$ .
Theorem 5. (OBL-G $_b$ ) is an  $A^\star$ -optimal with respect to  $[\mathcal{L}_{\mathrm{G}'}(\mathrm{P2})]$ .
4Originally, the inequality set suggested that  $(\mathrm{OBL - G_b})$  would  $\mathcal{A}^{\star}$  -optimal is which  $(f(y_{N})\geq f_{\star})$  is replaced with  $\left(f(y_N)\geq f_\star +\frac{1}{2L}\| \nabla f(y_N)\| ^2\right)$  in  $\mathcal{L}_{\mathrm{G}}'$ .
Proof of  $\mathcal{A}^{\star}$ -optimality of (OGM-G) We provide an alternative formulation of  $\mathcal{P}_2(\mathcal{L}_{\mathrm{G}}, H, R)$  by using the methodology called PEP [19, 52]
maximize  $\frac{1}{2L}\| \nabla f(y_N)\|^2$
subject to  $f(y_{i + 1}) - f(y_i) + \langle \nabla f(y_{i + 1}),y_i - y_{i + 1}\rangle +\frac{1}{2L}\| \nabla f(y_i) - \nabla f(y_{i + 1})\| ^2\leq 0,\quad i = 0,\ldots ,N - 1.$
$$
\begin{array}{l} f (y _ {i}) - f (y _ {N}) + \langle \nabla f (y _ {i}), y _ {N} - y _ {i} \rangle + \frac {1}{2 L} \| \nabla f (y _ {i}) - \nabla f (y _ {N}) \| ^ {2} \leq 0, \quad i = 0, \dots , N - 1 \\ f _ {\star} + \frac {1}{2 L} \| \nabla f (y _ {N}) \| ^ {2} \leq f (y _ {N}) \\ f (y _ {0}) - f _ {\star} \leq \frac {1}{2} L R ^ {2} \\ y _ {i + 1} = y _ {i} - \frac {1}{L} \sum_ {j = 0} ^ {i} h _ {i + 1, j} \nabla f (y _ {j}), \quad i = 0, \dots , N - 1 \tag {48} \\ \end{array}
$$
Next, define  $\mathbf{F},\mathbf{G},\{\mathbf{e}_i\}_{i = 0}^N$  and  $\{\mathbf{x}_i\}_{i = 0}^N$  as
$$
\mathbf {G} := \left( \begin{array}{c c c c} \langle \nabla f (y _ {0}), \nabla f (y _ {0}) \rangle & \langle \nabla f (y _ {0}), \nabla f (y _ {1}) \rangle & \dots & \langle \nabla f (y _ {0}), \nabla f (y _ {N}) \rangle \\ \vdots & \vdots & \vdots \\ \langle \nabla f (y _ {N}), \nabla f (y _ {N}) \rangle & \langle \nabla f (y _ {N}), \nabla f (y _ {1}) \rangle & \dots & \langle \nabla f (y _ {N}), \nabla f (y _ {N}) \rangle \end{array} \right) \in \mathbb {S} ^ {N + 1},
$$
$$
\mathbf {F} := \left( \begin{array}{c} f (y _ {0}) - f _ {\star} \\ f (y _ {1}) - f _ {\star} \\ \vdots \\ f (y _ {N}) - f _ {\star} \end{array} \right) \mathbb {R} ^ {(N + 1) \times 1},
$$
$\mathbf{e}_i \in \mathbb{R}^{N+1}$  is a unit vector which  $(i+1)$ -th component is 1, and
$$
\mathbf {y} _ {0} := 0, \qquad \mathbf {y} _ {i + 1} := \mathbf {y} _ {i} - \frac {1}{L} \sum_ {j = 0} ^ {i} h _ {i + 1, j} \mathbf {e} _ {j} \quad i = 0, \dots , N - 1.
$$
By using the definition of  $\mathbf{F}$ ,  $\mathbf{G}$ ,  $\{\mathbf{e}_i\}_{i=0}^N$  and  $\{\mathbf{y}_i\}_{i=0}^N$ ,  $\mathcal{P}_2(\mathcal{L}_{\mathbf{G}}, H, R)$  can be converted into
$$
\underset {\mathbf {F}, \mathbf {G} \succeq 0} {\text {m i n i m i z e}} - \frac {1}{2 L} \operatorname {T r} \left(\mathbf {G} \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal}\right)
$$
subject to  $\mathbf{F}(\mathbf{e}_{i + 1} - \mathbf{e}_i)^\top +\mathrm{Tr}(\mathbf{GA}_i)\leq 0,\quad i = 0,\ldots ,N - 1$
$$
\begin{array}{l} \mathbf {F} \left(\mathbf {e} _ {i} - \mathbf {e} _ {N}\right) ^ {\intercal} + \operatorname {T r} \left(\mathbf {G B} _ {i}\right) \leq 0, \quad i = 0, \dots , N - 1 \tag {49} \\ - \mathbf {F} e _ {N} ^ {\intercal} + \frac {1}{2} \operatorname {T r} \left(\mathbf {G} \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal}\right) \leq 0 \\ \end{array}
$$
$$
\mathbf {F} e _ {0} ^ {\intercal} - \frac {1}{2} L R ^ {2} \leq 0
$$
$$
\mathbf {A} _ {i} := \frac {1}{2} \mathbf {e} _ {i + 1} (\mathbf {y} _ {i} - \mathbf {y} _ {i + 1}) ^ {\intercal} + \frac {1}{2} (\mathbf {y} _ {i} - \mathbf {y} _ {i + 1}) \mathbf {e} _ {i + 1} ^ {\intercal} + \frac {1}{2 L} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) ^ {\intercal}
$$
$$
\mathbf {B} _ {i} := \frac {1}{2} \mathbf {e} _ {i} (\mathbf {y} _ {N} - \mathbf {y} _ {i}) ^ {\intercal} + \frac {1}{2} (\mathbf {y} _ {N} - \mathbf {y} _ {i}) \mathbf {e} _ {i} ^ {\intercal} + \frac {1}{2 L} \left(\mathbf {e} _ {i} - \mathbf {e} _ {N}\right) \left(\mathbf {e} _ {i} - \mathbf {e} _ {N}\right) ^ {\intercal}.
$$
Moreover, under the condition  $d \geq N + 2$ , we can take the Cholesky factorization of  $\mathbf{G}$  to recover the triplet  $\{(y_i, f(y_i), \nabla f(y_i))\}_{i=0}^N$ . Thus (48) and (49) are equivalent. The next step is calculating the Lagrangian of (49) and deriving the Lagrangian dual problem of it. In [54], they argued about the strong duality of (49).
<sup>5</sup>Cholesky factorization is unique if  $\mathbf{G} > 0$  but it may be not unique if  $\mathbf{G} \succeq 0$ . In this case, we choose one representation of Cholesky factorization.
Fact 1. Assume  $h_{i+1,i} \neq 0$  for  $0 \leq i \leq N-1$ . Then the strong duality holds between (48) and (49).
Denote the dual variables of each constraints as  $\{\delta_i\}_{i=0}^{N-1}$ ,  $\{\lambda_i\}_{i=0}^{N-1}$ ,  $\delta_N$  and  $\tau$ . Then Lagrangian becomes
$$
\mathcal {L} (\delta , \lambda , \tau , \mathbf {F}, \mathbf {G}) = \mathbf {F} \cdot \mathbf {X} ^ {\intercal} + \operatorname {T r} \left(\mathbf {G} \cdot \mathbf {T}\right) - \frac {\tau L}{2} R ^ {2}
$$
$$
\mathbf {X} = \sum_ {i = 0} ^ {N - 1} \delta_ {i} (\mathbf {e} _ {i + 1} - \mathbf {e} _ {i}) + \sum_ {i = 0} ^ {N - 1} \lambda_ {i} (\mathbf {e} _ {i} - \mathbf {e} _ {N}) - \delta_ {N} \mathbf {e} _ {N} + \tau \mathbf {e} _ {0},
$$