SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $e_{-1} = 0$ ,  $e_i$  is a unit vector which  $(i + 1) - th$  component is 1 and  $\mathbb{R}^{N + 1}$  and  $\mathcal{H}_1 \colon = \begin{bmatrix} 0 & 0 \\ H_1 & 0 \end{bmatrix}$ . If we take  $H_1$  that makes  $\mathcal{T}^{\oplus} \succeq 0$ ,  $\mathcal{V}_N \geq \frac{\alpha L}{2} \| y_N^{\oplus, \alpha} - y_N\|^2$  follows. Next we observe  $\frac{1}{\alpha^2}\mathcal{A}_3 + \frac{1}{\alpha}\mathcal{B}_3 = \mathcal{T}(H_1,v)$  for  $v_i = \frac{1}{T_{N - i}}$  which is defined as (22). By expanding (91) and defining  $g' \coloneqq L[x_0 - x_0^{\oplus, \alpha}] \ldots |x_N - x_N^{\oplus, \alpha}]$ ,
$$
\mathcal {U} _ {N} - T _ {N} \left(F (x _ {N} ^ {\oplus , 1}) - F _ {\star}\right) - \frac {L}{2} \left\| x _ {0} - x _ {\star} + \sum_ {i = 0} ^ {N} (u _ {i} - u _ {i - 1}) (x _ {i} - x _ {i} ^ {\oplus , 1}) \right\| ^ {2} = \operatorname {T r} \left(g ^ {\prime} (g ^ {\prime}) ^ {\intercal} \mathcal {S} ^ {\oplus}\right)
$$
follows where
$$
\mathcal {S} ^ {\oplus}: = \mathcal {S} (H _ {0}, u) - \sum_ {i = 0} ^ {N - 1} \left(T _ {i} - t _ {i} ^ {2}\right) \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal}.
$$
Here,  $u_{i} = T_{i}$ , and  $S(H_0,u)$  is given by the formulation (19). Using (91), we obtain
$$
\mathcal {S} (H _ {0}, u) = \sum_ {i = 0} ^ {N - 1} (2 T _ {i} - t _ {i} ^ {2}) \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal} + (T _ {N} - t _ {N} ^ {2}) \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal}.
$$
Now recall matrix  $\mathcal{M}(u)$  (26) and the result of Theorem 1:
$$
\mathcal {S} \left(H _ {0}, u\right) = \mathcal {M} (u) ^ {\intercal} \mathcal {T} \left(H _ {0} ^ {A}, v\right) \mathcal {M} (u). \tag {95}
$$
By substituting  $H_{1} = \frac{1}{\alpha} H_{0}^{A} + X$  and using (95), the result is
$$
\begin{array}{l} \mathcal {M} (u) ^ {\intercal} \mathcal {T} ^ {\oplus} \mathcal {M} (u) = \alpha \Bigg [ \sum_ {i = 0} ^ {N - 1} (2 T _ {i} - t _ {i} ^ {2}) \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal} + (T _ {N} - t _ {N} ^ {2}) \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal} \Bigg ] \\ + \underbrace {\alpha^ {2} \mathcal {M} (u) ^ {\intercal} \left[ \sum_ {i = 0} ^ {N} \frac {1}{T _ {N - i}} \left(\mathcal {X} ^ {\intercal} (\mathbf {e} _ {i} + \cdots + \mathbf {e} _ {N}) (\mathbf {e} _ {i} - \mathbf {e} _ {i - 1}) ^ {\intercal} + (\mathbf {e} _ {i} - \mathbf {e} _ {i - 1}) (\mathbf {e} _ {i} + \cdots + \mathbf {e} _ {N}) ^ {\intercal} \mathcal {X}\right) \right] \mathcal {M} (u)} _ {\mathcal {A} _ {4}} \\ + \underbrace {\mathcal {M} (u) ^ {\intercal} \left[ \frac {\alpha}{T _ {N}} \mathbf {e} _ {0} \mathbf {e} _ {0} ^ {\intercal} - \sum_ {i = 0} ^ {N - 1} \frac {1}{T _ {N - 1 - i}} \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal} - \frac {1}{T _ {0}} \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal} \right] \mathcal {M} (u)} _ {\mathcal {C} _ {3}} \\ \end{array}
$$
where  $\mathcal{X}\colon = \left[ \begin{array}{ll}0 & 0\\ X & 0 \end{array} \right].$
Next, consider  $\mathcal{A}_4$  as a function of the lower triangular matrix  $\mathcal{X}$ . The key observation is that if we choose  $\mathcal{X}$  appropriately, all non-diagonal terms of  $\mathcal{C}_3$  can be eliminated. With this choice of  $\mathcal{X}$ , we have
$$
\mathcal {T} ^ {\oplus} = \alpha \sum_ {i = 0} ^ {N} (2 T _ {i} - t _ {i} ^ {2}) \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal} - T _ {0} \mathbf {e} _ {0} \mathbf {e} _ {0} ^ {\intercal} - \sum_ {i = 1} ^ {N} \left(\frac {T _ {i} ^ {2}}{T _ {i - 1}} + t _ {i} ^ {2} \sum_ {j = 0} ^ {i - 2} \frac {1}{T _ {j}}\right) \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal}.
$$
Note that  $\mathcal{T}^{\oplus}$  contains only diagonal terms. Therefore,  $\mathcal{T}^{\oplus} \succeq 0$  is equivalent to all coefficients of  $\mathbf{e}_i\mathbf{e}_i^\intercal$  being nonnegative, which can be formulated using  $t_i$  and  $T_i$ .
Remark. In the proof of Theorem 1 ( $\mathbf{B}_2$  and  $\mathbf{B}_1$  at (27) and (28)), we have shown that
$$
\mathcal {A} _ {4} = \alpha^ {2} \left[ \sum_ {i = 0} ^ {N} \left(\mathcal {X} ^ {A}\right) ^ {\intercal} T _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) ^ {\intercal} + T _ {i} \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) ^ {\intercal} \mathcal {X} ^ {A} \right].
$$
Observe that  $(\mathbf{e}_i - \mathbf{e}_{i + 1})(\mathbf{e}_0 + \dots +\mathbf{e}_i)^\intercal$  is a lower triangular matrix and  $\mathcal{X}^A$  is a strictly lower triangular matrix, i.e., all diagonal component is 0. Thus it is worth noting that  $\mathcal{A}_4$  cannot induce any diagonal term  $\mathbf{e}_i\mathbf{e}_i^\intercal$ , choosing matrix  $C$  as the optimal one.
D.3 Parameterized family reduces the gradient mapping norm
FSFOM that reduces  $\frac{1}{2L}\left\| y_N^{\oplus ,\alpha} - y_N\right\|^2$  in the composite minimization. We propose the parameterized family that reduces  $\frac{1}{2L}\left\| y_N^{\oplus ,\alpha} - y_N\right\|^2$ . For fixed  $a > 0$ , take  $\{t_i\}_{i = 0}^N$ ,  $T_{i} = \sum_{j = 0}^{i}t_{j}$  that satisfies the following conditions:
$$
\alpha (2 T _ {0} - t _ {0} ^ {2}) \geq T _ {0},
$$
$$
\alpha \left(2 T _ {k} - t _ {k} ^ {2}\right) \geq \frac {T _ {k} ^ {2}}{T _ {k - 1}} + t _ {k} ^ {2} \left(\frac {1}{T _ {0}} + \sum_ {i = 0} ^ {k - 2} \frac {1}{T _ {i}}\right) \quad k = 1, \dots , N \tag {96}
$$
where we define  $\sum_{i=0}^{k-2} \frac{1}{T_i} = 0$  for  $k = 1$  case. We define an lower triangular matrix  $C = \{c_{k,i}\}_{0 \leq i < k \leq N}$  as
$$
c _ {k + 1, i} = \left\{ \begin{array}{l l} \frac {t _ {1}}{T _ {1}} & i = 0, k = 0 \\ \frac {t _ {k + 1}}{T _ {k + 1}} \left(\frac {t _ {k}}{T _ {0}} + t _ {k} \sum_ {j = 0} ^ {k - 2} \frac {1}{T _ {j}} + \frac {T _ {k}}{T _ {k - 1}}\right) & i = k, k = 1, \dots , N - 1 \\ \frac {t _ {k + 1} \left(T _ {k} - t _ {k}\right)}{t _ {k} T _ {k + 1}} c _ {k, i} & i = 0, \dots , k - 1, i = 2, \dots , N - 1 \end{array} . \right. \tag {97}
$$
Matrix  $C$  has a crucial role in the correspondence between composite function value minimization and composite function gradient norm minimization with  $H \rightarrow \left( \frac{1}{\alpha} H + \frac{1}{\alpha^2} C \right)^A$ .
Proposition 5 (SFG family). Assume that we choose  $H_0$  matrix of (88), and define  $C$  as (97). Consider an FSFOM (87) which  $H$  matrix is  $\left(\frac{1}{\alpha} H_0 + \frac{1}{\alpha^2} C\right)^A$ . Such FSFOM can be expressed as
$$
y _ {k + 1} = y _ {k} ^ {\oplus , \alpha} + \beta_ {k} ^ {\prime} \left(y _ {k} ^ {\oplus , \alpha} - y _ {k - 1} ^ {\oplus , \alpha}\right) + \gamma_ {k} ^ {\prime} \left(y _ {k} ^ {\oplus , \alpha} - y _ {k}\right), \quad k = 0, \dots , N - 1,
$$
$$
\beta_ {k} ^ {\prime} = \frac {\beta_ {N - k} \left(\beta_ {N - 1 - k} + \gamma_ {N - 1 - k}\right)}{\beta_ {N - k} + \gamma_ {N - k}}, \quad \gamma_ {k} ^ {\prime} = \frac {\gamma_ {N - k} \left(\beta_ {N - 1 - k} + \gamma_ {N - 1 - k}\right)}{\beta_ {N - k} + \gamma_ {N - k}}, \tag {SFG-family}
$$
$$
\beta_ {k} = \frac {t _ {k + 1} (T _ {k} - t _ {k})}{t _ {k} T _ {k + 1}}, \qquad \gamma_ {k} = \frac {t _ {k + 1} (t _ {k} ^ {2} - T _ {k})}{t _ {k} T _ {k + 1}} + \frac {1}{\alpha} c _ {k + 1, k}.
$$
and  $y_{-1}^{\oplus ,\alpha} = y_0$  . Then it exhibits the convergence rate
$$
\frac {\alpha L}{2} \left\| y _ {N} ^ {\oplus} - y _ {N} \right\| ^ {2} \leq \frac {1}{T _ {N}} \left(F \left(y _ {0}\right) - F _ {\star}\right).
$$
Proof of Proposition 5. To start, we give the claim about matrix  $C$ .
Claim 1.  $\mathcal{C} \colon = \begin{bmatrix} 0 & 0 \\ C & 0 \end{bmatrix}$  satisfies the following equality.
$$
\begin{array}{l} 2 \left(\sum_ {i = 0} ^ {N} T _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \left(e _ {0} + \dots + \mathbf {e} _ {i}\right) ^ {\intercal}\right) \mathcal {C} \\ = \sum_ {i = 1} ^ {N} \frac {1}{T _ {i - 1}} \mathbf {f} _ {N - i} \mathbf {f} _ {N - i} ^ {\intercal} + \frac {1}{T _ {0}} \mathbf {f} _ {N} \mathbf {f} _ {N} ^ {\intercal} - T _ {0} e _ {0} e _ {0} ^ {\intercal} - \sum_ {i = 1} ^ {N} \left(\frac {T _ {i} ^ {2}}{T _ {i - 1}} + t _ {i} ^ {2} \sum_ {j = 0} ^ {i - 2} \frac {1}{T _ {j}}\right) \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal} \tag {98} \\ \end{array}
$$
where  $\{\mathbf{e}_i\}_{i=0}^N$  is a any basis of  $\mathbb{R}^{N+1}$ ,  $\mathbf{e}_{N+1}$ ,  $\mathbf{f}_{-1}$  are zero vectors and  $\{\mathbf{f}_i\}_{i=0}^N$  are another basis of  $\mathbb{R}^{N+1}$  which is defined as
$$
T _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) = \mathbf {f} _ {N - i} - \mathbf {f} _ {N - i - 1}, \quad i = 0, 1, \dots , N.
$$
The proof of Claim 1 will be provided after the proof of Proposition 5. To begin with, for the FSFOM with matrix  $\left(\frac{1}{\alpha} H_0 + \frac{1}{\alpha^2} C\right)^A$ , define the energy function
$$
\mathcal {V} _ {k} = \frac {F (y _ {0}) - F (y _ {N} ^ {\oplus , \alpha}) + (\| y _ {- 1} , y _ {0} \|) ^ {\alpha}}{T _ {N}} + \sum_ {i = 0} ^ {k - 1} \frac {1}{T _ {N - i - 1}} (\| y _ {i}, y _ {i + 1} \|) ^ {\alpha} + \sum_ {i = 0} ^ {k - 1} \left(\frac {1}{T _ {N - i - 1}} - \frac {1}{T _ {N - i}}\right) (\| y _ {N}, y _ {i} \|) ^ {\alpha},
$$
for  $k = 0,\dots ,N$ .  $\{\mathcal{V}_k\}_{k = 0}^N$  is dissipative since  $(\cdot ,\cdot)^\alpha \leq 0$ . This energy function is inspired by (3). Note that if we can prove
$$
\frac {\alpha L}{2} \left\| y _ {N} ^ {\oplus , \alpha} - y _ {N} \right\| ^ {2} \leq \mathcal {V} _ {N}, \tag {99}
$$
$$
\frac {\alpha L}{2} \left\| y _ {N} ^ {\oplus , \alpha} - y _ {N} \right\| ^ {2} \leq \mathcal {V} _ {N} \leq \mathcal {V} _ {0} \leq \frac {1}{T _ {N}} \left(F (y _ {0}) - F (y _ {N} ^ {\oplus , \alpha})\right) \leq \frac {1}{T _ {N}} \left(F (y _ {0}) - F _ {\star}\right).
$$
Thus, it is enough to show (99). Defining  $g_{i} \colon = L\alpha(y_{i} - y_{i}^{\oplus,\alpha})$  for  $0 \leq i \leq N$  gives
$$
\begin{array}{l} \left( \right.\left| y _ {i}, y _ {j} \right| ^ {\alpha} = F \left(y _ {j} ^ {\oplus , \alpha}\right) - F \left(y _ {i} ^ {\oplus , \alpha}\right) + \left\langle g _ {j}, y _ {i} - y _ {j} \right\rangle + \frac {1}{L a} \left\langle g _ {j}, g _ {j} - g _ {i} \right\rangle - \frac {1}{2 L \alpha^ {2}} \left\| g _ {j} \right\| ^ {2}, \\ \left(\left| y _ {- 1}, y _ {0} \right| \right\rangle^ {\alpha} = F \left(y _ {0} ^ {\oplus , \alpha}\right) - F \left(y _ {0}\right) + \frac {1}{L} \frac {2 \alpha - 1}{2 \alpha^ {2}} \left\| g _ {0} \right\| ^ {2}. \\ \end{array}
$$
Plugging the above equalities provides
$$
\begin{array}{l} 2 L \alpha^ {2} \left(\mathcal {V} _ {N} - \frac {\alpha L}{2} \left\| y _ {N} ^ {\oplus , \alpha} - y _ {N} \right\| ^ {2}\right) \\ = 2 L \alpha^ {2} \left[ \sum_ {i = 0} ^ {N - 1} \frac {1}{T _ {N - i - 1}} \left\langle g _ {i + 1}, y _ {i} - y _ {i + 1} \right\rangle + \sum_ {i = 0} ^ {N - 1} \left(\frac {1}{T _ {N - i - 1}} - \frac {1}{T _ {N - i}}\right) \left\langle g _ {i}, y _ {N} - y _ {i} \right\rangle \right] + \frac {2 \alpha - 1}{T _ {N}} \| g _ {0} \| ^ {2} - \alpha \| g _ {N} \| ^ {2} \\ + \sum_ {i = 0} ^ {N - 1} \frac {1}{T _ {N - i - 1}} \left(2 \alpha \langle g _ {i + 1}, g _ {i + 1} - g _ {i} \rangle - \| g _ {i + 1} \| ^ {2}\right) + \sum_ {i = 0} ^ {N - 1} \left(\frac {1}{T _ {N - i - 1}} - \frac {1}{T _ {N - i}}\right) \left(2 \alpha \langle g _ {i}, g _ {i} - g _ {N} \rangle - \| g _ {i} \| ^ {2}\right) \\ = 2 L \alpha^ {2} \left(\sum_ {i = 0} ^ {N - 1} \frac {1}{T _ {N - i - 1}} \langle g _ {i + 1}, y _ {i} - y _ {i + 1} \rangle + \sum_ {i = 0} ^ {N - 1} \left(\frac {1}{T _ {N - i - 1}} - \frac {1}{T _ {N - i}}\right) \langle g _ {i}, y _ {N} - y _ {i} \rangle\right) \\ + \left[ \sum_ {i = 0} ^ {N - 1} \frac {\alpha}{T _ {N - i - 1}} \| g _ {i + 1} - g _ {i} \| ^ {2} + \sum_ {i = 0} ^ {N - 1} \left(\frac {\alpha}{T _ {N - i - 1}} - \frac {\alpha}{T _ {N - i}}\right) \| g _ {i} - g _ {N} \| ^ {2} - a \| g _ {N} \| ^ {2} + \frac {\alpha}{T _ {N}} \| g _ {N} \| ^ {2} \right] \\ + \sum_ {i = 0} ^ {N - 1} \frac {1}{T _ {N - i - 1}} \left(- \alpha \| g _ {i} \| ^ {2} + (a - 1) \| g _ {i + 1} \| ^ {2}\right) + \sum_ {i = 0} ^ {N - 1} \left(\frac {1}{T _ {N - i - 1}} - \frac {1}{T _ {N - i}}\right) \left(- \alpha \| g _ {N} \| ^ {2} + (\alpha - 1) \| g _ {i} \| ^ {2}\right) \\ + \frac {2 \alpha - 1}{T _ {N}} \| g _ {0} \| ^ {2} - \frac {\alpha}{T _ {N}} \| g _ {N} \| ^ {2} \\ = 2 L \alpha^ {2} \left(\sum_ {i = 0} ^ {N - 1} \frac {1}{T _ {N - i - 1}} \left\langle g _ {i + 1}, y _ {i} - y _ {i + 1} \right\rangle + \sum_ {i = 0} ^ {N - 1} \left(\frac {1}{T _ {N - i - 1}} - \frac {1}{T _ {N - i}}\right) \left\langle g _ {i}, y _ {N} - y _ {i} \right\rangle\right) \\ + \left[ \sum_ {i = 0} ^ {N - 1} \frac {\alpha}{T _ {N - i - 1}} \| g _ {i + 1} - g _ {i} \| ^ {2} + \sum_ {i = 0} ^ {N - 1} \left(\frac {\alpha}{T _ {N - i - 1}} - \frac {\alpha}{T _ {N - i}}\right) \| g _ {i} - g _ {N} \| ^ {2} - \alpha \| g _ {N} \| ^ {2} + \frac {\alpha}{T _ {N}} \| g _ {N} \| ^ {2} \right] \\ + \frac {\alpha}{T _ {N}} \| g _ {0} \| ^ {2} - \sum_ {i = 0} ^ {N - 1} \frac {1}{T _ {N - 1 - i}} \| g _ {i} \| ^ {2} - \frac {1}{T _ {0}} \| g _ {N} \| ^ {2}. \\ \end{array}
$$
Next, define  $g \colon = [g_0|g_1\ldots |g_N]$  and
$$
\mathcal {H} _ {1} \colon = \left[ \begin{array}{c c} 0 & 0 \\ \left(\frac {1}{\alpha} H _ {0} + \frac {1}{\alpha^ {2}} C\right) ^ {A} & 0 \end{array} \right], \quad \mathcal {H} _ {0} \colon = \left[ \begin{array}{c c} 0 & 0 \\ H _ {0} & 0 \end{array} \right].
$$
By the same procedure as the proof of Theorem 1, we obtain  $2L\alpha^{2}\left(\mathcal{V}_{N} - \frac{\alpha L}{2}\left\| y_{N}^{\oplus ,\alpha} - y_{N}\right\|^{2}\right) = \operatorname {Tr}\left(g^{\intercal}g\mathcal{T}^{\oplus}\right)$  where
$$
\begin{array}{l} \mathcal {T} ^ {\oplus} = \alpha^ {2} \left[ \sum_ {i = 0} ^ {N} \frac {1}{T _ {N - i}} \left(\mathcal {H} _ {1} ^ {\intercal} \left(\mathbf {f} _ {i} + \dots + \mathbf {f} _ {N}\right) \left(\mathbf {f} _ {i} - \mathbf {f} _ {i - 1}\right) ^ {\intercal} + \left(\mathbf {f} _ {i} - \mathbf {f} _ {i - 1}\right) \left(\mathbf {f} _ {i} + \dots + \mathbf {f} _ {N}\right) ^ {\intercal} \mathcal {H} _ {1}\right) \right. \\ + \alpha \left[ \sum_ {i = 0} ^ {N} \frac {1}{T _ {N - i}} \left(\left(\mathbf {f} _ {i - 1} - \mathbf {f} _ {i}\right) \left(\mathbf {f} _ {i - 1} - \mathbf {f} _ {N}\right) ^ {\intercal} + \left(\mathbf {f} _ {i - 1} - \mathbf {f} _ {N}\right) \left(\mathbf {f} _ {i - 1} - \mathbf {f} _ {i}\right) ^ {\intercal}\right) - \frac {1}{T _ {N}} \mathbf {f} _ {0} \mathbf {f} _ {0} ^ {\intercal} - \mathbf {f} _ {N} \mathbf {f} _ {N} ^ {\intercal} \right] \\ + \left[ \frac {\alpha}{T _ {N}} \mathbf {f} _ {0} \mathbf {f} _ {0} ^ {\intercal} - \sum_ {i = 0} ^ {N - 1} \frac {1}{T _ {N - 1 - i}} \mathbf {f} _ {i} \mathbf {f} _ {i} ^ {\intercal} - \frac {1}{T _ {0}} \mathbf {f} _ {N} \mathbf {f} _ {N} ^ {\intercal} \right] \\ \end{array}
$$
And consider the vector  $\mathbf{e}_i\in \mathbb{R}^{(N + 1)\times 1}$  which is same with (29).
In the proof of Theorem 1 ((27), (28)), we have shown that
$$
\begin{array}{l} \frac {1}{2 L} \sum_ {i = 0} ^ {N} \frac {1}{T _ {N - i}} \left(\mathcal {H} _ {1} ^ {\intercal} \left(\mathbf {f} _ {i} + \dots + \mathbf {f} _ {N}\right) \left(\mathbf {f} _ {i} - \mathbf {f} _ {i - 1}\right) ^ {\intercal} + \left(\mathbf {f} _ {i} - \mathbf {f} _ {i - 1}\right) \left(\mathbf {f} _ {i} + \dots + \mathbf {f} _ {N}\right) ^ {\intercal} \mathcal {H} _ {1}\right) \\ = \frac {1}{2 L} \left(\frac {\mathcal {H} _ {0} ^ {\intercal}}{\alpha} + \frac {\mathcal {C} ^ {\intercal}}{\alpha^ {2}}\right) \left[ \sum_ {i = 0} ^ {N} T _ {i} (\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}) (\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}) ^ {\intercal} \right] + \frac {1}{2 L} \left[ \sum_ {i = 0} ^ {N} T _ {i} (\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}) (\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}) ^ {\intercal} \right] \left(\frac {\mathcal {H} _ {0}}{\alpha} + \frac {\mathcal {C}}{\alpha^ {2}}\right) \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{2 L} \sum_ {i = 0} ^ {N} \frac {1}{T _ {N - i}} \left(\left(\mathbf {f} _ {i - 1} - \mathbf {f} _ {i}\right) \left(\mathbf {f} _ {i - 1} - \mathbf {f} _ {N}\right) ^ {\intercal} + \left(\mathbf {f} _ {i - 1} - \mathbf {f} _ {N}\right) \left(\mathbf {f} _ {i - 1} - \mathbf {f} _ {i}\right) ^ {\intercal}\right) - \frac {1}{2 T _ {N} L} \mathbf {f} _ {0} \mathbf {f} _ {0} ^ {\intercal} - \frac {1}{2 L} \mathbf {f} _ {N} \mathbf {f} _ {N} ^ {\intercal} \\ = - \frac {1}{2 L} \left(\sum_ {i = 0} ^ {N} \left(T _ {i} - T _ {i - 1}\right) \mathbf {e} _ {i}\right) \left(\sum_ {i = 0} ^ {N} \left(T _ {i} - T _ {i - 1}\right) \mathbf {e} _ {i}\right) ^ {\intercal} \\ + \frac {1}{2 L} \left[ \sum_ {i = 0} ^ {N} T _ {i} \left(\left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \mathbf {e} _ {i} ^ {\intercal} + \mathbf {e} _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) ^ {\intercal}\right) - T _ {N} \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal} \right] \\ \end{array}
$$
under the above transformation. Therefore,  $\mathcal{T}^{\oplus}$  can be expressed as
$$
\begin{array}{l} \mathcal {T} ^ {\oplus} = \alpha^ {2} \left(\left(\frac {\mathcal {H} _ {0} ^ {\intercal}}{\alpha} + \frac {\mathcal {C} ^ {\intercal}}{\alpha^ {2}}\right) \left[ \sum_ {i = 0} ^ {N} T _ {i} (\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}) (\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}) ^ {\intercal} \right] + \left[ \sum_ {i = 0} ^ {N} T _ {i} (\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}) (\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}) ^ {\intercal} \right] \left(\frac {\mathcal {H} _ {0}}{\alpha} + \frac {\mathcal {C}}{\alpha^ {2}}\right)\right) \\ + \alpha \left(- \left(\sum_ {i = 0} ^ {N} \left(T _ {i} - T _ {i - 1}\right) \mathbf {e} _ {i}\right) \left(\sum_ {i = 0} ^ {N} \left(T _ {i} - T _ {i - 1}\right) \mathbf {e} _ {i}\right) ^ {\intercal} + \left[ \sum_ {i = 0} ^ {N} T _ {i} \left(\left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \mathbf {e} _ {i} ^ {\intercal} + \mathbf {e} _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) ^ {\intercal}\right) - T _ {N} \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal} \right]\right) \\ + \left[ \frac {\alpha}{T _ {N}} \mathbf {f} _ {0} \mathbf {f} _ {0} ^ {\intercal} - \sum_ {i = 0} ^ {N - 1} \frac {1}{T _ {N - 1 - i}} \mathbf {f} _ {i} \mathbf {f} _ {i} ^ {\intercal} - \frac {1}{T _ {0}} \mathbf {f} _ {N} \mathbf {f} _ {N} ^ {\intercal} \right] \\ = \alpha \mathcal {A} + \mathcal {B} + \frac {\alpha}{T _ {N}} \mathbf {f} _ {0} \mathbf {f} _ {0} ^ {\intercal} \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {A} = \mathcal {H} _ {0} ^ {\intercal} \left[ \sum_ {i = 0} ^ {N} T _ {i} \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) ^ {\intercal} \right] + \left[ \sum_ {i = 0} ^ {N} T _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) ^ {\intercal} \right] \mathcal {H} _ {0} \\ \left. - \left(\sum_ {i = 0} ^ {N} \left(T _ {i} - T _ {i - 1}\right) \mathbf {e} _ {i}\right) \left(\sum_ {i = 0} ^ {N} \left(T _ {i} - T _ {i - 1}\right) \mathbf {e} _ {i}\right) ^ {\intercal} + \left[ \sum_ {i = 0} ^ {N} T _ {i} \left(\left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \mathbf {e} _ {i} ^ {\intercal} + \mathbf {e} _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) ^ {\intercal}\right) - T _ {N} \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal} \right] \right. \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {B} = \mathcal {C} ^ {\intercal} \left[ \sum_ {i = 0} ^ {N} T _ {i} \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) ^ {\intercal} \right] + \left[ \sum_ {i = 0} ^ {N} T _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) ^ {\intercal} \right] \mathcal {C} \\ - \sum_ {i = 0} ^ {N - 1} \frac {1}{T _ {N - 1 - i}} \mathbf {f} _ {i} \mathbf {f} _ {i} ^ {\intercal} - \frac {1}{T _ {0}} \mathbf {f} _ {N} \mathbf {f} _ {N} ^ {\intercal}. \\ \end{array}
$$
To calculate  $\mathcal{A}$ , expand the energy function (91). Then we obtain
$$
\mathcal {A} = \sum_ {i = 0} ^ {N - 1} (2 T _ {i} - t _ {i} ^ {2}) \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal} + (T _ {N} - t _ {N} ^ {2}) \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal}.
$$
Moreover, by Claim 1,
$$
\mathcal {B} = - T _ {0} \mathbf {e} _ {0} \mathbf {e} _ {0} ^ {\intercal} - \sum_ {i = 1} ^ {N} \left(\frac {T _ {i} ^ {2}}{T _ {i - 1}} + t _ {i} ^ {2} \sum_ {j = 0} ^ {i - 2} \frac {1}{T _ {j}}\right) \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal}.
$$
Combining above results and  $T_{N}\mathbf{e}_{N} = \mathbf{f}_{0}$ , we achieve
$$
\mathcal {T} ^ {\oplus} = \alpha \sum_ {i = 0} ^ {N} (2 T _ {i} - t _ {i} ^ {2}) \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal} - T _ {0} \mathbf {e} _ {0} \mathbf {e} _ {0} ^ {\intercal} - \sum_ {i = 1} ^ {N} \left(\frac {T _ {i} ^ {2}}{T _ {i - 1}} + t _ {i} ^ {2} \sum_ {j = 0} ^ {i - 2} \frac {1}{T _ {j}}\right) \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal}
$$
and the condition (96) makes each coefficient of  $\mathbf{e}_i\mathbf{e}_i^\intercal$  nonnegative, which gives  $\mathcal{T}\succeq 0$ . To achieve the iteration formula (SFG-family), consider the FSFOM (87) with  $\frac{1}{\alpha} H_0 + \frac{1}{\alpha^2} C$ , which can be
expressed as
$$
x _ {k + 1} = x _ {k} ^ {\oplus , \alpha} + \beta_ {k} \left(x _ {k} ^ {\oplus , \alpha} - x _ {k - 1} ^ {\oplus , \alpha}\right) + \gamma_ {k} \left(x _ {k} ^ {\oplus , \alpha} - x _ {k}\right), \quad k = 0, \dots , N - 1,
$$
$$
\beta_ {k} = \frac {t _ {k + 1} (T _ {k} - t _ {k})}{t _ {k} T _ {k + 1}}, \quad \gamma_ {k} = \frac {t _ {k + 1} (t _ {k} ^ {2} - T _ {k})}{t _ {k} T _ {k + 1}} + \frac {1}{\alpha} c _ {k + 1, k}.
$$
For last step, We make an observation that Proposition 1 can be applied when all  $z^{+}$  terms changed into  $z^{\oplus ,\alpha}$ . Proposition 1 gives the H-dual.
Proof of Claim 1. The left hand side of (98) is
$$
\begin{array}{l} 2 \left(\sum_ {i = 0} ^ {N} T _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) ^ {\intercal}\right) \mathcal {C} \\ = 2 \left(\sum_ {i = 0} ^ {N} T _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) ^ {\intercal}\right) \left(\sum_ {k > i} c _ {k, i} \mathbf {e} _ {k} \mathbf {e} _ {i} ^ {\intercal}\right) \\ = 2 \left(\sum_ {i = 0} ^ {N} \left(\mathbf {f} _ {N - i} - \mathbf {f} _ {N - i + 1}\right) \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) ^ {\intercal}\right) \left(\sum_ {k > i} c _ {k, i} \mathbf {e} _ {k} \mathbf {e} _ {i} ^ {\intercal}\right) \\ = 2 \left(\sum_ {i = 0} ^ {N} \mathbf {f} _ {N - i} \mathbf {e} _ {i} ^ {\mathsf {T}}\right) \left(\sum_ {k > i} c _ {k, i} \mathbf {e} _ {k} \mathbf {e} _ {i} ^ {\mathsf {T}}\right) \\ = 2 \sum_ {i = 0} ^ {N} \left(\sum_ {k = i + 1} ^ {N} c _ {k, i} \mathbf {f} _ {N - k}\right) \mathbf {e} _ {i} ^ {\intercal} \\ \end{array}
$$
Now we calculate the right-hand side of (98). By plugging
$$
\mathbf {f} _ {N - i} = T _ {i} \mathbf {e} _ {i} + t _ {i + 1} \mathbf {e} _ {i + 1} + \dots + t _ {N} \mathbf {e} _ {N},
$$
$$
\sum_ {i = 1} ^ {N} \frac {1}{T _ {i - 1}} \mathbf {f} _ {N - i} \mathbf {f} _ {N - i} ^ {\intercal} + \frac {1}{T _ {0}} \mathbf {f} _ {N} \mathbf {f} _ {N} ^ {\intercal} - T _ {0} \mathbf {e} _ {0} \mathbf {e} _ {0} ^ {\intercal} - \sum_ {i = 1} ^ {N} \left(\frac {T _ {i} ^ {2}}{T _ {i - 1}} + t _ {i} ^ {2} \sum_ {j = 0} ^ {i - 2} \frac {1}{T _ {j}}\right) \mathbf {e} _ {i} \mathbf {e} _ {i} ^ {\intercal} = 2 \sum_ {i = 0} ^ {N - 1} \mathbf {a} _ {i} \mathbf {e} _ {i} ^ {\intercal}
$$
$$
\mathbf {a} _ {0} = t _ {1} \mathbf {e} _ {1} + \dots + t _ {N} \mathbf {e} _ {N},
$$
$$
\mathbf {a} _ {i} = \left(\frac {t _ {i}}{T _ {0}} + \sum_ {j = 0} ^ {i - 2} \frac {t _ {i}}{T _ {j}} + \frac {T _ {i}}{T _ {i - 1}}\right) (t _ {i + 1} \mathbf {e} _ {i + 1} + \dots + t _ {N} \mathbf {e} _ {N}), \quad i = 1, \ldots , N - 1.
$$
Now we claim that  $\mathbf{a}_i = \sum_{k=i+1}^{N} c_{k,i} \mathbf{f}_{N-k}$  for  $i = 0, \ldots, N-1$ . Note that