SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {1}{2} \| \mathbf {z} _ {k} - \mathbf {x} ^ {*} \| ^ {2} \leq A _ {k} (f (\mathbf {x} _ {k}) - f ^ {*}) + \frac {1}{2} \| \mathbf {z} _ {k} - \mathbf {x} ^ {*} \| ^ {2} \leq A _ {0} (f (\mathbf {x} _ {0}) - f ^ {*}) + \frac {1}{2} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| ^ {2} = \frac {1}{2} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| ^ {2}.
$$
Hence, we get  $\| \mathbf{z}_k - \mathbf{x}^*\| \leq \| \mathbf{z}_0 - \mathbf{x}^*\|$  for any  $k\geq 0$ . To show the second inequality, we distinguish two cases and in both cases we will prove that
$$
A _ {k + 1} \| \mathbf {x} _ {k + 1} - \mathbf {x} ^ {*} \| ^ {2} \leq A _ {k} \| \mathbf {x} _ {k} - \mathbf {x} ^ {*} \| ^ {2} + \left(A _ {k + 1} - A _ {k}\right) \frac {2 \sigma^ {2} a _ {k} ^ {2}}{\eta_ {k} ^ {2}} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| ^ {2} + 2 \left(A _ {k + 1} - A _ {k}\right) \| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \| ^ {2}. \tag {34}
$$
Case I:  $\hat{\eta}_k = \eta_k$ . Recall that in the proof of Proposition 2 we defined  $\tilde{\mathbf{z}}_{k + 1} = \hat{\mathbf{x}}_{k + 1} + \frac{A_k}{a_k} (\hat{\mathbf{x}}_{k + 1} - \mathbf{x}_k)$ .
Since  $\mathbf{x}_{k + 1} = \hat{\mathbf{x}}_{k + 1}$ , we have  $\mathbf{x}_{k + 1} = \frac{A_k}{A_k + a_k}\mathbf{x}_k + \frac{a_k}{A_k + a_k}\tilde{\mathbf{z}}_{k + 1}$  and by Jensen's inequality
$$
\left\| \mathbf {x} _ {k + 1} - \mathbf {x} ^ {*} \right\| ^ {2} \leq \frac {A _ {k}}{A _ {k} + a _ {k}} \left\| \mathbf {x} _ {k} - \mathbf {x} ^ {*} \right\| ^ {2} + \frac {a _ {k}}{A _ {k} + a _ {k}} \left\| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {x} ^ {*} \right\| ^ {2}.
$$
Furthermore, we have
$$
\left\| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {x} ^ {*} \right\| ^ {2} \leq 2 \left\| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k + 1} \right\| ^ {2} + 2 \left\| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \right\| ^ {2} \leq \frac {2 \sigma^ {2} a _ {k} ^ {2}}{\eta_ {k} ^ {2}} \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\| ^ {2} + 2 \left\| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \right\| ^ {2}, \tag {35}
$$
where we used (26) in the last inequality. By combining the above two inequalities, we obtain
$$
(A _ {k} + a _ {k}) \| \mathbf {x} _ {k + 1} - \mathbf {x} ^ {*} \| ^ {2} \leq A _ {k} \| \mathbf {x} _ {k} - \mathbf {x} ^ {*} \| ^ {2} + a _ {k} \frac {2 \sigma^ {2} a _ {k} ^ {2}}{\eta_ {k} ^ {2}} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| ^ {2} + 2 a _ {k} \| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \| ^ {2},
$$
which leads to (34) (note that  $A_{k + 1} = A_k + a_k$  in Case I).
Case II: Since  $\mathbf{x}_{k + 1} = \frac{(1 - \gamma_k)A_k}{A_k + \gamma_ka_k}\mathbf{x}_k + \frac{\gamma_k(A_k + a_k)}{A_k + \gamma_ka_k}\hat{\mathbf{x}}_{k + 1}$  and  $\hat{\mathbf{x}}_{k + 1} = \frac{A_k}{A_k + a_k}\mathbf{x}_k + \frac{a_k}{A_k + a_k}\tilde{\mathbf{z}}_{k + 1}$ , we have
$$
\mathbf {x} _ {k + 1} = \frac {A _ {k}}{A _ {k} + \gamma_ {k} a _ {k}} \mathbf {x} _ {k} + \frac {\gamma_ {k} a _ {k}}{A _ {k} + \gamma_ {k} a _ {k}} \tilde {\mathbf {z}} _ {k + 1}.
$$
Similarly, by Jensen's inequality we have
$$
\left(A _ {k} + \gamma_ {k} a _ {k}\right) \| \mathbf {x} _ {k + 1} - \mathbf {x} ^ {*} \| ^ {2} \leq A _ {k} \| \mathbf {x} _ {k} - \mathbf {x} ^ {*} \| ^ {2} + \gamma_ {k} a _ {k} \| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {x} ^ {*} \| ^ {2}.
$$
Combining this inequality with (35), we obtain
$$
\left(A _ {k} + \gamma_ {k} a _ {k}\right) \left\| \mathbf {x} _ {k + 1} - \mathbf {x} ^ {*} \right\| ^ {2} \leq A _ {k} \left\| \mathbf {x} _ {k} - \mathbf {x} ^ {*} \right\| ^ {2} + \gamma_ {k} a _ {k} \frac {2 \sigma^ {2} a _ {k} ^ {2}}{\eta_ {k} ^ {2}} \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\| ^ {2} + 2 \gamma_ {k} a _ {k} \left\| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \right\| ^ {2}. \tag {36}
$$
which leads to (34) (note that  $A_{k + 1} = A_k + \gamma_k a_k$  in Case II).
Now by summing (34) over  $k = 0,\dots ,N - 1$  , we get
$$
\begin{array}{l} A _ {N} \left\| \mathbf {x} _ {N} - \mathbf {x} ^ {*} \right\| ^ {2} \leq \sum_ {k = 0} ^ {N - 1} \left(A _ {k + 1} - A _ {k}\right) \frac {2 \sigma^ {2} a _ {k} ^ {2}}{\eta_ {k} ^ {2}} \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\| ^ {2} + \sum_ {k = 0} ^ {N - 1} 2 \left(A _ {k + 1} - A _ {k}\right) \left\| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \right\| ^ {2} (37) \\ \leq 2 \sigma^ {2} \sum_ {k = 0} ^ {N - 1} \left(A _ {k + 1} - A _ {k}\right) \sum_ {k = 0} ^ {N - 1} \frac {a _ {k} ^ {2}}{\eta_ {k} ^ {2}} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| ^ {2} + 2 \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| ^ {2} \sum_ {k = 0} ^ {N - 1} \left(A _ {k + 1} - A _ {k}\right) (38) \\ \leq \frac {2 \sigma^ {2}}{1 - \sigma^ {2}} A _ {N} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| ^ {2} + 2 A _ {N} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| ^ {2} (39) \\ = \frac {2 A _ {N}}{1 - \sigma^ {2}} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| ^ {2}. (40) \\ \end{array}
$$
Hence, this implies that  $\| \mathbf{x}_k - \mathbf{x}^*\|^2 \leq \frac{2}{1 - \sigma^2}\| \mathbf{z}_0 - \mathbf{x}^*\|^2$  for any  $k \geq 0$ .
A key term appearing in several of our bounds is  $\frac{a_{k+1}}{A_{k+1} + a_{k+1}}$ . In the next lemma, we establish an upper bound for this ratio based on a factor of its previous value, for both cases of our algorithm.
Lemma 6. Without loss of generality assume  $\beta > 1/5$ . In Case I we have  $\frac{a_{k+1}}{A_{k+1} + a_{k+1}} \leq \frac{1}{\sqrt{\beta}} \frac{a_k}{A_k + a_k}$ . Otherwise, in Case II we have  $\frac{a_{k+1}}{A_{k+1} + a_{k+1}} \leq \frac{2\sqrt{\beta}}{\sqrt{\beta} + 1} \frac{a_k}{A_k + a_k}$ .
Proof. By the choice of  $a_{k}$  in (2) we have  $\eta_k(A_k + a_k) = a_k^2$  for all  $k\geq 0$ . As a result, we have
$$
\frac {a _ {k}}{A _ {k} + a _ {k}} = \frac {\eta_ {k}}{a _ {k}} = \frac {2 \eta_ {k}}{\eta_ {k} + \sqrt {\eta_ {k} ^ {2} + 4 \eta_ {k} A _ {k}}} = \frac {2}{1 + \sqrt {1 + 4 \frac {A _ {k}}{\eta_ {k}}}},
$$
and similarly
$$
\frac {a _ {k + 1}}{A _ {k + 1} + a _ {k + 1}} = \frac {2}{1 + \sqrt {1 + 4 \frac {A _ {k + 1}}{\eta_ {k + 1}}}}.
$$
In Case I, we have  $\eta_{k + 1} = \eta_k / \beta$  and  $A_{k + 1}\geq A_k$ . Hence, it implies that  $A_{k + 1} / \eta_{k + 1}\geq \beta A_k / \eta_k$ , which leads to
$$
\frac {a _ {k + 1}}{A _ {k + 1} + a _ {k + 1}} \leq \frac {2}{1 + \sqrt {1 + \frac {4 \beta A _ {k}}{\eta_ {k}}}} \leq \frac {2}{\sqrt {\beta} + \sqrt {\beta + \frac {4 \beta A _ {k}}{\eta_ {k}}}} = \frac {1}{\sqrt {\beta}} \frac {2}{1 + \sqrt {1 + 4 \frac {A _ {k}}{\eta_ {k}}}} = \frac {1}{\sqrt {\beta}} \frac {a _ {k}}{A _ {k} + a _ {k}}.
$$
where the second inequality follows from the fact that  $\beta \leq 1$ .
In Case II, we have  $\eta_{k + 1} = \hat{\eta}_k = \gamma_k\eta_k$  and  $A_{k + 1} = A_{k} + \gamma_{k}a_{k}$ . Since we also have  $a_{k}\geq \eta_{k}$  and  $\gamma_{k}\leq \beta$ , we obtain  $A_{k + 1} / \eta_{k + 1}\geq A_k / (\gamma_k\eta_k) + 1\geq A_k / (\beta \eta_k) + 1$ . Hence,
$$
\frac {a _ {k + 1}}{A _ {k + 1} + a _ {k + 1}} \leq \frac {2}{1 + \sqrt {5 + \frac {4 A _ {k}}{\beta \eta_ {k}}}} \leq \frac {2}{1 + \frac {1}{\sqrt {\beta}} \sqrt {1 + \frac {4 A _ {k}}{\eta_ {k}}}} \leq \frac {2 \sqrt {\beta}}{\sqrt {\beta} + 1} \frac {2}{1 + \sqrt {1 + \frac {4 A _ {k}}{\eta_ {k}}}} = \frac {2 \sqrt {\beta}}{\sqrt {\beta} + 1} \frac {a _ {k}}{A _ {k} + a _ {k}},
$$
where we used  $\beta > 1/5$  in the second inequality and the fact that  $1 + \frac{1}{\sqrt{\beta}} x \geq \frac{\sqrt{\beta} + 1}{2\sqrt{\beta}} (1 + x)$  for  $x \geq 1$  in the last inequality.
Remark 5. If  $\beta \leq 1/5$ , then in Case I we still have  $\frac{a_{k+1}}{A_{k+1} + a_{k+1}} \leq \frac{1}{\sqrt{\beta}} \frac{a_k}{A_k + a_k}$ , while in Case II we have  $\frac{a_{k+1}}{A_{k+1} + a_{k+1}} \leq \frac{2}{\sqrt{5} + 1} \frac{a_k}{A_k + a_k}$ . Thus, in the case where  $\beta \leq 1/5$ , the derivation below still holds except that the absolute constant  $C_2$  will be different.
Next, as a corollary of Lemma 6, we establish an upper bound on the series  $\sum_{k=0}^{N-1} \frac{a_k}{A_k + a_k}$ . Moreover, we use this result to establish an upper bound for  $\sum_{k=0}^{N-1} \|\hat{\mathbf{x}}_{k+1} - \mathbf{y}_k\|$ .
Lemma 7. We have
$$
\sum_ {k = 0} ^ {N - 1} \frac {a _ {k}}{A _ {k} + a _ {k}} \leq \frac {1 + 2 \sqrt {\beta} - \beta}{\sqrt {\beta} - \beta} \left(1 + \log \frac {A _ {N}}{A _ {1}}\right). \tag {41}
$$
$$
\sum_ {k = 0} ^ {N - 1} \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\| \leq \sqrt {\frac {1}{1 - \sigma^ {2}} \frac {1 + 2 \sqrt {\beta} - \beta}{\sqrt {\beta} - \beta} \left(1 + \log \frac {A _ {N}}{A _ {1}}\right)} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \|. \tag {42}
$$
Proof. Given the initial values of  $A_{k}$  and  $a_{k}$  we have
$$
\sum_ {k = 0} ^ {N - 1} \frac {a _ {k}}{A _ {k} + a _ {k}} = 1 + \sum_ {k = 1} ^ {N - 1} \frac {a _ {k}}{A _ {k} + a _ {k}} = 1 + \sum_ {k \in \mathcal {B}, k \geq 1} \frac {a _ {k}}{A _ {k} + a _ {k}} + \sum_ {k \notin \mathcal {B}, k \geq 1} \frac {a _ {k}}{A _ {k} + a _ {k}} \tag {43}
$$
Note that using the result in Lemma 6
$$
\begin{array}{l} \sum_ {k \in \mathcal {B}, k > 1} \frac {a _ {k}}{A _ {k} + a _ {k}} \leq \sum_ {k = 0} ^ {N - 2} \frac {a _ {k + 1}}{A _ {k + 1} + a _ {k + 1}} (44) \\ = \sum_ {k \notin \mathcal {B}, k \geq 0} \frac {a _ {k + 1}}{A _ {k + 1} + a _ {k + 1}} + \sum_ {k \in \mathcal {B}, k \geq 0} \frac {a _ {k + 1}}{A _ {k + 1} + a _ {k + 1}} (45) \\ \leq \sum_ {k \notin \mathcal {B}, k \geq 0} \frac {1}{\sqrt {\beta}} \frac {a _ {k}}{A _ {k} + a _ {k}} + \sum_ {k \in \mathcal {B}, k \geq 0} \frac {2 \sqrt {\beta}}{\sqrt {\beta} + 1} \frac {a _ {k}}{A _ {k} + a _ {k}} (46) \\ \leq \frac {1}{\sqrt {\beta}} + \sum_ {k \notin \mathcal {B}, k \geq 1} \frac {1}{\sqrt {\beta}} \frac {a _ {k}}{A _ {k} + a _ {k}} + \sum_ {k \in \mathcal {B}, k \geq 1} \frac {2 \sqrt {\beta}}{\sqrt {\beta} + 1} \frac {a _ {k}}{A _ {k} + a _ {k}}. (47) \\ \end{array}
$$
Hence, if we move the last term in the above upper bound to the left hand side and rescale both sides of the resulted inequality we obtain
$$
\sum_ {k \in \mathcal {B}, k \geq 1} \frac {a _ {k}}{A _ {k} + a _ {k}} \leq \frac {1 + \sqrt {\beta}}{\sqrt {\beta} - \beta} \bigg (1 + \sum_ {k \notin \mathcal {B}, k \geq 1} \frac {a _ {k}}{A _ {k} + a _ {k}} \bigg).
$$
Now, if we replace the above upper bound into (43) we obtain
$$
\sum_ {k = 0} ^ {N - 1} \frac {a _ {k}}{A _ {k} + a _ {k}} \leq \frac {1 + 2 \sqrt {\beta} - \beta}{\sqrt {\beta} - \beta} \left(1 + \sum_ {k \notin \mathcal {B}, k \geq 1} \frac {a _ {k}}{A _ {k} + a _ {k}}\right). \tag {48}
$$
Moreover, note that for  $k \notin \mathcal{B}$ , we have  $A_{k+1} = A_k + a_k$ . Hence,
$$
\begin{array}{l} \sum_ {k \notin \mathcal {B}, k \geq 1} \frac {a _ {k}}{A _ {k} + a _ {k}} = \sum_ {k \notin \mathcal {B}, k \geq 1} \left(1 - \frac {A _ {k}}{A _ {k + 1}}\right) \leq \sum_ {k \notin \mathcal {B}, k \geq 1} \left(\log \left(A _ {k + 1}\right) - \log \left(A _ {k}\right)\right) \\ \leq \sum_ {k = 1} ^ {N - 1} \left(\log \left(A _ {k + 1}\right) - \log \left(A _ {k}\right)\right) = \log \frac {A _ {N}}{A _ {1}}. \\ \end{array}
$$
Now if we replace the above upper bound, i.e.,  $\log \frac{A_N}{A_1}$  with  $\sum_{k\notin \mathcal{B},k\geq 1}\frac{a_k}{A_k + a_k}$  into the expression in the right-hand side of (48) we obtain the result in (41).
Next, note that by Cauchy-Schwarz inequality, we have
$$
\sum_ {k = 0} ^ {N - 1} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| \leq \sqrt {\sum_ {k = 0} ^ {N - 1} \frac {\eta_ {k} ^ {2}}{a _ {k} ^ {2}} \sum_ {k = 0} ^ {N - 1} \frac {a _ {k} ^ {2}}{\eta_ {k} ^ {2}} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| ^ {2}} \leq \sqrt {\frac {1}{1 - \sigma^ {2}} \sum_ {k = 0} ^ {N - 1} \frac {\eta_ {k} ^ {2}}{a _ {k} ^ {2}}} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \|,
$$
where the last inequality follows from (21). Moreover, based on the expression for  $a_{k}$  in (2) and the result in (41) that we just proved, we have
$$
\sum_ {k = 0} ^ {N - 1} \frac {\eta_ {k} ^ {2}}{a _ {k} ^ {2}} = \sum_ {k = 0} ^ {N - 1} \frac {a _ {k} ^ {2}}{(A _ {k} + a _ {k}) ^ {2}} \leq \sum_ {k = 0} ^ {N - 1} \frac {a _ {k}}{A _ {k} + a _ {k}} \leq \frac {1 + 2 \sqrt {\beta} - \beta}{\sqrt {\beta} - \beta} \bigg (1 + \log \frac {A _ {N}}{A _ {1}} \bigg).
$$
Combining the two inequalities above leads to (42).
Now we are ready to present and prove Lemma 8 which characterizes a bound on the path length of the sequence  $\{\mathbf{y}_k\}_{k=0}^N$
Lemma 8. Consider the iterates generated by Algorithm 1. Then for any  $N$ ,
$$
\sum_ {k = 0} ^ {N - 1} \left\| \mathbf {y} _ {k + 1} - \mathbf {y} _ {k} \right\| \leq C _ {2} \left(1 + \log \frac {A _ {N}}{A _ {1}}\right) \left\| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \right\|.
$$
$$
C _ {2} = 2 \sqrt {\frac {1}{1 - \sigma^ {2}} \frac {1 + 2 \sqrt {\beta} - \beta}{\sqrt {\beta} - \beta}} + \frac {1}{\sqrt {\beta}} \left(1 + \sqrt {\frac {2}{1 - \sigma^ {2}}}\right) \frac {1 + 2 \sqrt {\beta} - \beta}{\sqrt {\beta} - \beta} \tag {49}
$$
Proof. By the triangle inequality, we have
$$
\left\| \mathbf {y} _ {k} - \mathbf {y} _ {k + 1} \right\| \leq \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\| + \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k + 1} \right\|. \tag {50}
$$
We again distinguish two cases.
Case I:  $\hat{\eta}_k = \eta_k$ . In this case  $\hat{\mathbf{x}}_{k + 1} = \mathbf{x}_{k + 1}$  and  $\mathbf{y}_{k + 1} = \frac{A_{k + 1}}{A_{k + 1} + a_{k + 1}}\mathbf{x}_{k + 1} + \frac{a_{k + 1}}{A_{k + 1} + a_{k + 1}}\mathbf{z}_{k + 1}$ , hence
$$
\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k + 1} \| = \| \mathbf {x} _ {k + 1} - \mathbf {y} _ {k + 1} \| = \frac {a _ {k + 1} \| \mathbf {z} _ {k + 1} - \mathbf {x} _ {k + 1} \|}{A _ {k + 1} + a _ {k + 1}} \leq \frac {1}{\sqrt {\beta}} \bigg (1 + \sqrt {\frac {2}{1 - \sigma^ {2}}} \bigg) \frac {a _ {k} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \|}{A _ {k} + a _ {k}},
$$
where we used Lemma 6 and the fact that  $\| \mathbf{z}_{k + 1} - \mathbf{x}_{k + 1}\| \leq \| \mathbf{z}_{k + 1} - \mathbf{x}^*\| +\| \mathbf{x}_{k + 1} - \mathbf{x}^*\| \leq$ $(1 + \sqrt{\frac{2}{1 - \sigma^2}})\| \mathbf{z}_0 - \mathbf{x}^*\|$  in the last inequality. Therefore, using (50) and the above bound we have
$$
\left\| \mathbf {y} _ {k} - \mathbf {y} _ {k + 1} \right\| \leq \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\| + \frac {1}{\sqrt {\beta}} \left(1 + \sqrt {\frac {2}{1 - \sigma^ {2}}}\right) \frac {a _ {k}}{A _ {k} + a _ {k}} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \|. \tag {51}
$$
Case II:  $\hat{\eta}_k < \eta_k$ . Since  $\mathbf{x}_{k+1} = \frac{A_k}{A_k + \gamma_k a_k} \mathbf{x}_k + \frac{\gamma_k a_k}{A_k + \gamma_k a_k} \tilde{\mathbf{z}}_{k+1}$  and  $\hat{\mathbf{x}}_{k+1} = \frac{A_k}{A_k + a_k} \mathbf{x}_k + \frac{a_k}{A_k + a_k} \tilde{\mathbf{z}}_{k+1}$ , we get
$$
\hat {\mathbf {x}} _ {k + 1} = \frac {A _ {k}}{A _ {k} + a _ {k}} \Big (\mathbf {x} _ {k + 1} + \frac {\gamma_ {k} a _ {k}}{A _ {k}} (\mathbf {x} _ {k + 1} - \tilde {\mathbf {z}} _ {k + 1}) \Big) + \frac {a _ {k}}{A _ {k} + a _ {k}} \tilde {\mathbf {z}} _ {k + 1} = \frac {A _ {k} + \gamma_ {k} a _ {k}}{A _ {k} + a _ {k}} \mathbf {x} _ {k + 1} + \frac {(1 - \gamma_ {k}) a _ {k}}{A _ {k} + a _ {k}} \tilde {\mathbf {z}} _ {k + 1}.
$$
Thus, given the above equality and the expression  $\mathbf{y}_{k + 1} = \frac{A_{k + 1}}{A_{k + 1} + a_{k + 1}}\mathbf{x}_{k + 1} + \frac{a_{k + 1}}{A_{k + 1} + a_{k + 1}}\mathbf{z}_{k + 1}$ , we have
$$
\left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k + 1} \right\| \leq \frac {(1 - \gamma_ {k}) a _ {k}}{A _ {k} + a _ {k}} \| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k + 1} \| + \left| \frac {(1 - \gamma_ {k}) a _ {k}}{A _ {k} + a _ {k}} - \frac {a _ {k + 1}}{A _ {k + 1} + a _ {k + 1}} \right| \| \mathbf {z} _ {k + 1} - \mathbf {x} _ {k + 1} \|. \tag {52}
$$
Moreover, based on the result in (26), we can upper bound  $\| \tilde{\mathbf{z}}_{k + 1} - \mathbf{z}_{k + 1}\|$  by  $\sigma \frac{a_k}{\eta_k}\| \hat{\mathbf{x}}_{k + 1} - \mathbf{y}_k\|$  which implies that
$$
\frac {(1 - \gamma_ {k}) a _ {k}}{A _ {k} + a _ {k}} \| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k + 1} \| \leq \sigma \frac {(1 - \gamma_ {k}) a _ {k} ^ {2}}{\eta_ {k} (A _ {k} + a _ {k})} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| = \sigma (1 - \gamma_ {k}) \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| \leq \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|
$$
where the equality holds due to the definition of  $a_{k}$ , and the last inequality holds as both  $\gamma_{k}$  and  $\sigma$  are in (0, 1). On the other hand, note that
$$
\frac {(1 - \gamma_ {k}) a _ {k}}{A _ {k} + a _ {k}} - \frac {a _ {k + 1}}{A _ {k + 1} + a _ {k + 1}} \leq \frac {(1 - \gamma_ {k}) a _ {k}}{A _ {k} + a _ {k}} \leq \frac {a _ {k}}{A _ {k} + a _ {k}}, \tag {53}
$$
$$
\frac {a _ {k + 1}}{A _ {k + 1} + a _ {k + 1}} - \frac {(1 - \gamma_ {k}) a _ {k}}{A _ {k} + a _ {k}} \leq \frac {2 \sqrt {\beta} a _ {k}}{\sqrt {\beta + 1} (A _ {k} + a _ {k})} - \frac {(1 - \gamma_ {k}) a _ {k}}{A _ {k} + a _ {k}} \leq \frac {a _ {k}}{A _ {k} + a _ {k}}. \tag {54}
$$
where in the second bound we used the result in Lemma 6 and the fact that  $\frac{2sqrt{\beta}}{\sqrt{\beta + 1}} < 1$ . Hence, we get
$$
\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k + 1} \| \leq \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| + \frac {a _ {k} \| \mathbf {z} _ {k + 1} - \mathbf {x} _ {k + 1} \|}{A _ {k} + a _ {k}} \leq \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| + \left(1 + \sqrt {\frac {2}{1 - \sigma^ {2}}}\right) \frac {a _ {k} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \|}{A _ {k} + a _ {k}},
$$
where the last inequality follows from the fact  $\| \mathbf{z}_{k + 1} - \mathbf{x}_{k + 1}\| \leq \| \mathbf{z}_{k + 1} - \mathbf{x}^*\| +\| \mathbf{x}_{k + 1} - \mathbf{x}^*\|$  and the bounds in Lemma 5. Now by applying the above upper bound into (50) we obtain that
$$
\left\| \mathbf {y} _ {k} - \mathbf {y} _ {k + 1} \right\| \leq 2 \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\| + \left(1 + \sqrt {\frac {2}{1 - \sigma^ {2}}}\right) \frac {a _ {k}}{A _ {k} + a _ {k}} \left\| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \right\|. \tag {55}
$$
Considering the upper bounds established for  $\| \mathbf{y}_k - \mathbf{y}_{k + 1}\|$  in case I (equation (51)) and case II (equation (55)), we can conclude that
$$
\left\| \mathbf {y} _ {k} - \mathbf {y} _ {k + 1} \right\| \leq 2 \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| + \frac {1}{\sqrt {\beta}} \left(1 + \sqrt {\frac {2}{1 - \sigma^ {2}}}\right) \frac {a _ {k}}{A _ {k} + a _ {k}} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \|. \tag {56}
$$
Finally, Lemma 8 follows from summing (56) over  $k = 0$  to  $N - 1$  and the result of Lemma 7.
Subroutine 1 Backtracking line search
1: Input: iterate  $\mathbf{y} \in \mathbb{R}^d$ , gradient  $\mathbf{g} \in \mathbb{R}^d$ , Hessian approximation  $\mathbf{B} \in \mathbb{S}_+^d$ , initial trial step size  $\eta > 0$   
2: Parameters: line search parameters  $\beta \in (0,1)$ ,  $\alpha_1 \geq 0$  and  $\alpha_2 > 0$  such that  $\alpha_1 + \alpha_2 < 1$   
3: Set  $\hat{\eta} \gets \eta$   
4: Compute  $s_+$ $\leftarrow$  LinearSolver(I +  $\hat{\eta}$ B, - $\hat{\eta}$ g;  $\alpha_1$ ) and  $\hat{x}_+$ $\leftarrow$  y + s+  
5: while  $\|\hat{x}_+ - y + \hat{\eta}\nabla f(\hat{x}_+)\|_2 > (\alpha_1 + \alpha_2)\|\hat{x}_+ - y\|_2$  do  
6: Set  $\tilde{x}_+$ $\leftarrow$ $\hat{x}_+$  and  $\hat{\eta} \gets \beta\hat{\eta}$   
7: Compute  $s_+$ $\leftarrow$  LinearSolver(I +  $\hat{\eta}$ B, - $\hat{\eta}$ g;  $\alpha_1$ ) and  $\hat{x}_+$ $\leftarrow$  y + s+  
8: end while  
9: if  $\hat{\eta} = \eta$  then  
10: Return  $\hat{\eta}$  and  $\hat{x}_+$   
11: else  
12: Return  $\hat{\eta}, \hat{x}_+$  and  $\tilde{x}_+$   
13: end if
B Line Search Subroutine
In this section, we provide further details on our line search subroutine in Section 3.1. For completeness, the pseudocode of our line search scheme is shown in Subroutine 1. In Section B.1, we prove that Subroutine 1 will always terminate in a finite number of steps. In Section B.2, we provide the proof of Lemma 1.
B.1 The Line Search Subroutine Terminates Properly
Recall that in our line search scheme, we keep decreasing the step size  $\hat{\eta}$  by a factor of  $\beta$  until we find a pair  $(\hat{\eta},\hat{\mathbf{x}}_{+})$  satisfying (11) (also see Lines 5 and 6 in Subroutine 1). In the following lemma, we show that when the step size  $\hat{\eta}$  is smaller than a certain threshold, then the pair  $(\hat{\eta},\hat{\mathbf{x}}_{+})$  satisfies both conditions in (10) and (11), which further implies that Subroutine 1 will stop in a finite number of steps.
Lemma 9. Suppose Assumption 1 holds. If  $\hat{\eta} < \frac{\alpha_2}{L_1 + \| \mathbf{B} \|_{\mathrm{op}}}$  and  $\hat{\mathbf{x}}_+$  is computed according to (12), then the pair  $(\hat{\eta}, \hat{\mathbf{x}}_+)$  satisfies the conditions in (10) and (11).
Proof. By Definition 1, the pair  $(\hat{\eta}, \hat{\mathbf{x}}_{+})$  always satisfies the condition in (10) when  $\hat{\mathbf{x}}_{+}$  is computed from (12). Hence, in the following we only need to prove that the condition in (11) also holds. Recall that  $\mathbf{g} = \nabla f(\mathbf{y})$ . By Assumption 1, the function  $f$  is  $L_{1}$ -smooth and thus we have
$$
\left\| \nabla f (\hat {\mathbf {x}} _ {+}) - \mathbf {g} \right\| = \left\| \nabla f (\hat {\mathbf {x}} _ {+}) - \nabla f (\mathbf {y}) \right\| \leq L _ {1} \left\| \hat {\mathbf {x}} _ {+} - \mathbf {y} \right\|.
$$
Moreover, by using the triangle inequality, we get
$$
\| \nabla f (\hat {\mathbf {x}} _ {+}) - \mathbf {g} - \mathbf {B} (\hat {\mathbf {x}} _ {+} - \mathbf {y}) \| \leq \| \nabla f (\hat {\mathbf {x}} _ {+}) - \mathbf {g} \| + \| \mathbf {B} (\hat {\mathbf {x}} _ {+} - \mathbf {y}) \| \leq (L _ {1} + \| \mathbf {B} \| _ {\mathrm {o p}}) \| \hat {\mathbf {x}} _ {+} - \mathbf {y} \|.
$$
Hence, if  $\hat{\eta} \leq \frac{\alpha_2}{L_1 + \| \mathbf{B} \|_{\mathrm{op}}}$ , we have
$$
\hat {\eta} \| \nabla f (\hat {\mathbf {x}} _ {+}) - \mathbf {g} - \mathbf {B} (\hat {\mathbf {x}} _ {+} - \mathbf {y}) \| \leq \alpha_ {2} \| \hat {\mathbf {x}} _ {+} - \mathbf {y} \|. \tag {57}
$$
Finally, by using the triangle inequality, we can combine (10) and (57) to show that
$$
\begin{array}{l} \| \hat {\mathbf {x}} _ {+} - \mathbf {y} + \hat {\eta} \nabla f (\hat {\mathbf {x}} _ {+}) \| = \| \hat {\mathbf {x}} _ {+} - \mathbf {y} + \hat {\eta} (\mathbf {g} + \mathbf {B} (\hat {\mathbf {x}} _ {+} - \mathbf {y})) + \hat {\eta} (\nabla f (\hat {\mathbf {x}} _ {+}) - \mathbf {g} - \mathbf {B} (\hat {\mathbf {x}} _ {+} - \mathbf {y})) \| \\ \leq \left\| \hat {\mathbf {x}} _ {+} - \mathbf {y} + \hat {\eta} (\mathbf {g} + \mathbf {B} (\hat {\mathbf {x}} _ {+} - \mathbf {y})) \right\| + \left\| \hat {\eta} \left(\nabla f (\hat {\mathbf {x}} _ {+}) - \mathbf {g} - \mathbf {B} (\hat {\mathbf {x}} _ {+} - \mathbf {y})\right) \right\| \\ \leq \alpha_ {1} \| \hat {\mathbf {x}} _ {+} - \mathbf {y} \| + \alpha_ {2} \| \hat {\mathbf {x}} _ {+} - \mathbf {y} \| \\ \leq \left(\alpha_ {1} + \alpha_ {2}\right) \| \hat {\mathbf {x}} _ {+} - \mathbf {y} \|, \\ \end{array}
$$
which means the condition in (11) is satisfied. The proof is now complete.
B.2 Proof of Lemma 1
We follow a similar proof strategy as Lemma 3 in [35]. In the first case where  $k \notin \mathcal{B}$ , by definition, the line search subroutine accepts the initial step size  $\eta_{k}$ , i.e.,  $\hat{\eta}_{k} = \eta_{k}$ . In the second case where  $k \in \mathcal{B}$ , the line search subroutine backtracks and returns the auxiliary iterate  $\tilde{\mathbf{x}}_{k + 1}$ , which is computed from
(12) using the step size  $\tilde{\eta}_k \triangleq \hat{\eta}_k / \beta$ . Since the step size  $\tilde{\eta}_k$  is rejected in our line search subroutine, it implies that the pair  $(\tilde{\mathbf{x}}_{k + 1}, \tilde{\eta}_k)$  does not satisfy (11), i.e.,