SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {k = i + 1} ^ {N} c _ {k, i} \mathbf {f} _ {N - k} = \sum_ {k = i + 1} ^ {N} c _ {k, i} \left(T _ {k} \mathbf {e} _ {k} + t _ {k + 1} \mathbf {e} _ {k + 1} + \dots + t _ {N} \mathbf {e} _ {N}\right) \\ = \sum_ {k = i + 1} ^ {N} \left(c _ {k, i} T _ {k} + c _ {k - 1, i} t _ {k} + \dots + c _ {i + 1, i} t _ {k}\right) \mathbf {e} _ {k}. \\ \end{array}
$$
Coefficients of  $\mathbf{e}_{i + 1}$  are coincides since
$$
c _ {1, 0} T _ {1} = t _ {1},
$$
$$
c _ {i + 1, i} T _ {i + 1} = t _ {i + 1} \left(\frac {t _ {i}}{T _ {0}} + \sum_ {j = 0} ^ {i - 2} \frac {t _ {i}}{T _ {j}} + \frac {T _ {i}}{T _ {i - 1}}\right) \quad i = 1, \dots , N - 1.
$$
Now assume
$$
\frac {t _ {j}}{t _ {j + 1}} = \frac {c _ {j , i} T _ {j} + c _ {j - 1 , i} t _ {j} + \cdots + c _ {i + 1 , i} t _ {j}}{c _ {j + 1 , i} T _ {j + 1} + c _ {j , i} t _ {j + 1} + \cdots + c _ {i + 1 , i} t _ {j + 1}}. \tag {100}
$$
By multiplying the above equation recursively, we obtain
$$
\frac {t _ {i + 1}}{t _ {j + 1}} = \frac {c _ {i + 1 , i} T _ {i}}{c _ {j + 1 , i} T _ {j + 1} + c _ {j , i} t _ {j + 1} + \cdots + c _ {i + 1 , i} t _ {j + 1}}
$$
for  $1 \leq j \leq N - 1$ . which implies  $\mathbf{a}_i = \sum_{k = i + 1}^N c_{k,i}\mathbf{f}_{N - k}$ .
To prove (100), expand and obtain
$$
t _ {j + 1} c _ {j, i} T _ {j} = t _ {j} \left(c _ {j + 1, i} T _ {j + 1} + c _ {j, i} t _ {j + 1}\right).
$$
Above equation holds due to the recursive formula of  $C$  (97).
D.4 Instances of SFG family
Derivation of (SFG) Here, we will show that (SFG) is the instance of SFG family, under the choice of
First,  $t_i = \frac{i + 2}{2}$  for  $i \geq 1$  and  $t_0 = \frac{3}{2}$ . Also (96) holds since
$$
\begin{array}{l} 4 \left(3 - \frac {9}{4}\right) \geq \frac {3}{2} \\ 4 \left(\frac {(k + 2) (k + 3)}{2} - \frac {(k + 2) ^ {2}}{4}\right) \geq \frac {\frac {(k + 2) ^ {2} (k + 3) ^ {2}}{1 6}}{\frac {(k + 1) (k + 2)}{4}} + \frac {(k + 2) ^ {2}}{4} \left(\frac {2}{3} + 4 \left(\frac {1}{2} - \frac {1}{k + 1}\right)\right) \quad k = 1, \ldots , N. \\ \end{array}
$$
Plugging above values into (92) and (97), we obtain
$$
h _ {k + 1, i} = \left\{ \begin{array}{l l} 1 + \frac {1}{4} & i = k,   k = 0 \\ 1 + \frac {k}{k + 4} & i = k,   k = 1, \ldots , N - 1 \\ \frac {k + 1}{k + 4} (h _ {k, k - 1} - 1) & i = k - 1 \\ \frac {k + 1}{k + 4} h _ {k, i} & i = 0, \ldots , k - 2 \end{array} \right..
$$
$$
c _ {k + 1, i} = \left\{ \begin{array}{l l} \frac {1}{2} & i = 0, k = 0 \\ \frac {2 (4 k + 5)}{3 (k + 4)} & i = k, k = 1, \ldots , N - 1 \\ \frac {k + 1}{k + 4} c _ {k, i} & i = 0, \ldots , k - 1, i = 2, \ldots , N - 1 \end{array} \right..
$$
Other terms come directly, and
$$
\begin{array}{l} c _ {k + 1, k} = \frac {t _ {k + 1}}{T _ {k + 1}} \left(\frac {t _ {k}}{T _ {0}} + t _ {k} \sum_ {j = 0} ^ {k - 2} \frac {1}{T _ {j}} + \frac {T _ {k}}{T _ {k - 1}}\right) \\ = \frac {t _ {k + 1}}{T _ {k + 1}} \left(\frac {t _ {k}}{T _ {0}} + t _ {k} \sum_ {j = 0} ^ {k - 1} \frac {1}{T _ {j}} + 1\right) \\ = \frac {\frac {k + 3}{2}}{\frac {(k + 3) (k + 4)}{4}} \left(\frac {k + 2}{2} \times \frac {2}{3} + \frac {k + 2}{2} \left(\frac {4}{2 \times 3} + \dots + \frac {4}{(k + 1) (k + 2)}\right) + 1\right) \\ = \frac {2}{k + 4} \left(\frac {k + 2}{3} + \frac {k + 2}{2} \frac {2 k}{k + 2} + 1\right) \\ = \frac {2 (4 k + 5)}{3 (k + 4)}. \\ \end{array}
$$
To sum up, the matrix  $\{g_{k,i}\}_{0\leq i < k\leq N}\colon = \frac{1}{4} H_0 + \frac{1}{16} C$  satisfies the following recursive formula.
$$
g _ {k + 1, i} = \left\{ \begin{array}{l l} \frac {1}{4} \left(1 + \frac {3}{8}\right) & i = 0, k = 0 \\ \frac {1}{4} \left(1 + \frac {1 0 k + 5}{6 (k + 4)}\right) & i = k, k = 1, \ldots , N - 1 \\ \frac {k + 1}{k + 4} \left(g _ {k, i} - \frac {1}{4}\right) & i = k - 1 \\ \frac {k + 1}{k + 4} g _ {k, i} & i = 0, \ldots , k - 2 \end{array} \right..
$$
$$
\begin{array}{l} g _ {k + 1, k} = \frac {1}{4} \left(1 + \frac {k}{k + 4}\right) + \frac {1}{1 6} \left(\frac {2 (4 k + 5)}{3 (k + 4)}\right) \\ = \frac {1}{4} \left(1 + \frac {k}{k + 4} + \frac {4 k + 5}{6 (k + 4)}\right) \\ = \frac {1}{4} \left(1 + \frac {1 0 k + 5}{6 (k + 4)}\right). \\ \end{array}
$$
Therefore, the FSFOM with  $\{g_{k,i}\}_{0\leq i < k\leq N}$  can be expressed as
$$
\begin{array}{l} x _ {1} = x _ {0} ^ {\oplus , 4} + \frac {1}{4} \cdot \left(x _ {0} ^ {\oplus , 4} - x _ {- 1} ^ {\oplus , 4}\right) + \frac {1}{8} \left(x _ {0} ^ {\oplus , 4} - x _ {0}\right), \\ x _ {k + 1} = x _ {k} ^ {\oplus , 4} + \frac {k + 1}{k + 4} \left(x _ {k} ^ {\oplus , 4} - x _ {k - 1} ^ {\oplus , 4}\right) + \frac {4 k - 1}{6 (k + 4)} \left(x _ {k} ^ {\oplus , 4} - x _ {k}\right), \quad k = 1, \dots , N - 1 \\ \end{array}
$$
where  $x_{-1}^{\oplus,4} = x_0$ . To obtain the H-dual, we apply Proposition 1.
$$
\begin{array}{l} y _ {k + 1} = y _ {k} ^ {\oplus , 4} + \frac {(N - k + 1) (2 N - 2 k - 1)}{(N - k + 3) (2 N - 2 k + 1)} \left(y _ {k} ^ {\oplus , 4} - y _ {k - 1} ^ {\oplus , 4}\right) + \frac {(4 N - 4 k - 1) (2 N - 2 k - 1)}{6 (N - k + 3) (2 N - 2 k + 1)} \left(y _ {k} ^ {\oplus , 4} - y _ {k}\right) \\ y _ {N} = y _ {N - 1} ^ {\oplus , 4} + \frac {3}{1 0} \left(y _ {N - 1} ^ {\oplus , 4} - y _ {N - 2} ^ {\oplus , 4}\right) + \frac {3}{4 0} \left(y _ {N - 1} ^ {\oplus , 4} - y _ {N - 1}\right) \\ \end{array}
$$
where  $k = 0,\ldots ,N - 2$  and  $y_{-1}^{\oplus ,4} = y_0$
Fastest method among the SFG family via a numerical choice of  $a$  In this section, we give simple form of SFG, when all inequality conditions in (96) holds as equalities.  $\{g_{k,i}\}_{0\leq i < k\leq N}$  becomes
$$
g _ {1, 0} = \frac {1}{\alpha} \left(1 + \frac {(t _ {0} - 1) t _ {1}}{T _ {1}}\right) + \frac {1}{\alpha^ {2}} \left(\frac {t _ {1}}{T _ {1}}\right)
$$
and for  $k > 0$
$$
\begin{array}{l} g _ {k + 1, k} = \frac {1}{\alpha} \left(1 + \frac {\left(t _ {k} - 1\right) t _ {k + 1}}{T _ {k + 1}}\right) + \frac {1}{\alpha^ {2}} \frac {t _ {k + 1}}{T _ {k + 1}} \left(t _ {k} \left(\frac {1}{T _ {0}} + \frac {1}{T _ {0}} + \dots + \frac {1}{T _ {k - 2}}\right) + \frac {T _ {k}}{T _ {k - 1}}\right) \\ = \frac {1}{\alpha} \left(1 + \frac {(t _ {k} - 1) t _ {k + 1}}{T _ {k + 1}}\right) + \frac {1}{\alpha^ {2}} \frac {t _ {k + 1}}{T _ {k + 1}} \left(\frac {\alpha (2 T _ {k} - t _ {k} ^ {2}) - \frac {T _ {k} ^ {2}}{T _ {k - 1}}}{t _ {k}} + \frac {T _ {k}}{T _ {k - 1}}\right) \\ = \frac {1}{\alpha} \left(1 + \frac {(t _ {k} - 1) t _ {k + 1}}{T _ {k + 1}}\right) + \frac {1}{\alpha^ {2}} \frac {t _ {k + 1}}{T _ {k + 1}} \left(\frac {\alpha (2 T _ {k} - t _ {k} ^ {2})}{t _ {k}} - \frac {T _ {k}}{t _ {k}}\right) \\ = \frac {1}{\alpha} + \frac {1}{\alpha} \frac {t _ {k + 1}}{T _ {k + 1}} \left(t _ {k} - 1 + \frac {2 T _ {k} - t _ {k} ^ {2}}{t _ {k}} - \frac {T _ {k}}{t _ {k}}\right) - \left(\frac {1}{\alpha^ {2}} - \frac {1}{\alpha}\right) \frac {t _ {k + 1} T _ {k}}{T _ {k + 1} t _ {k}} \\ = \frac {1}{\alpha} + \frac {1}{\alpha} \frac {\left(T _ {k} - t _ {k}\right) t _ {k + 1}}{t _ {k} T _ {k + 1}} + \left(\frac {1}{\alpha} - \frac {1}{\alpha^ {2}}\right) \frac {t _ {k + 1} T _ {k}}{T _ {k + 1} t _ {k}}, \\ \end{array}
$$
$$
\begin{array}{l} g _ {k + 1, k - 1} = \frac {t _ {k + 1} \left(T _ {k} - t _ {k}\right)}{t _ {k} T _ {k + 1}} \left(g _ {k, k - 1} - \frac {1}{\alpha}\right), \\ g _ {k + 1, i} = \frac {t _ {k + 1} \left(T _ {k} - t _ {k}\right)}{t _ {k} T _ {k + 1}} g _ {k, i}, \qquad i = k - 2, \dots , 0. \\ \end{array}
$$
Hence FSFOM with  $\frac{1}{\alpha} H_0 + \frac{1}{\alpha^2} C$  is
$$
x _ {k + 1} = x _ {k} ^ {\oplus , \alpha} + \frac {T _ {k - 1} t _ {k + 1}}{t _ {k} T _ {k + 1}} \left(x _ {k} ^ {\oplus , \alpha} - x _ {k - 1} ^ {\oplus , \alpha}\right) + \left(1 - \frac {1}{\alpha}\right) \frac {t _ {k + 1} T _ {k}}{T _ {k + 1} t _ {k}} \left(x _ {k} ^ {\oplus , \alpha} - x _ {k}\right)
$$
where  $T_{-1} = \frac{2a - 1}{\alpha^2}$ . By using Proposition 1, we obtain H-dual.
$$
y _ {k + 1} = y _ {k} ^ {\oplus , \alpha} + \beta_ {k} ^ {\prime} \left(y _ {k} ^ {\oplus , \alpha} - y _ {k - 1} ^ {\oplus , \alpha}\right) + \left(1 - \frac {1}{\alpha}\right) \frac {T _ {N - k}}{T _ {N - k + 1}} \beta_ {N - k} \left(y _ {k} ^ {\oplus , \alpha} - y _ {k}\right)
$$
$$
\beta_ {k} ^ {\prime} = \frac {T _ {N - k - 1} t _ {N - k} \left(T _ {N - k - 2} + \left(1 - \frac {1}{\alpha}\right) T _ {N - k - 1}\right)}{t _ {N - k - 1} T _ {N - k} \left(T _ {N - k - 1} + \left(1 - \frac {1}{\alpha}\right) T _ {N - k}\right)}, \qquad k = 0, \dots , N - 1.
$$
Since all equality holds at (96), the above FSFOM achieves the fastest rate among the SFG family under fixed  $\alpha$ . Now we optimize  $a$  to achieve the fastest convergence rate. Combine (93) and the result of Proposition 5 to obtain
$$
\min  _ {v \in \partial F (y _ {N} ^ {\oplus})} \| v \| ^ {2} \leq L ^ {2} (\alpha + 1) ^ {2} \left\| y _ {N} - y _ {N} ^ {\oplus , \alpha} \right\| ^ {2} \leq L ^ {2} \frac {2 (\alpha + 1) ^ {2}}{\alpha T _ {N}} \left(F (y _ {0}) - F _ {\star}\right).
$$
To achieve the tightest convergence guarantee, we solve the following optimization problem under the fixed  $N$ .
$$
\underset {\alpha} {\text {m i n i m i z e}} \quad \frac {2 (\alpha + 1) ^ {2}}{\alpha T _ {N}}
$$
subject to.  $\alpha (2T_0 - t_0^2) = T_0$
$$
\alpha \left(2 T _ {k} - t _ {k} ^ {2}\right) = \frac {T _ {k} ^ {2}}{T _ {k - 1}} + t _ {k} ^ {2} \left(\frac {1}{T _ {0}} + \sum_ {i = 0} ^ {k - 2} \frac {1}{T _ {i}}\right) \quad k = 1, \dots , N.
$$
Denote the solution of the above optimization problem as  $R(\alpha, N)$ . Since  $R(\alpha, N)$  depends on  $a$  and cannot achieve a closed-form solution, we numerically choose  $a$  and observe an asymptotic behavior of  $R(\alpha, N)$ . By choosing  $\alpha = 3.8$ , asymptotic rate of  $R(3.8, N)$  is about  $\frac{46}{N^2}$ .
Our analysis concerns  $L$ -smooth convex functions. Although this assumption is standard in optimization theory, many functions that arise in machine learning practice are neither smooth nor convex.
7In fact, when  $a = 1$  , the FSFOM becomes FISTA-G [31].