SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f (\mathbf {y}) \leq f (\mathbf {x}) + \left\langle \nabla f (\mathbf {x}), \mathbf {y} - \mathbf {x} \right\rangle | + \frac {L}{2} \| \mathbf {y} - \mathbf {x} \| ^ {2}, \forall \mathbf {x}, \mathbf {y} \in \mathbb {R} ^ {d}. \tag {A.4}
$$
Lemma 10 (Nesterov [39], Theorem 2.1.5). If a convex function  $f$  satisfies (A.4), then it holds that:
$$
f (\mathbf {y}) \geq f (\mathbf {x}) + \left\langle \nabla f (\mathbf {x}), \mathbf {y} - \mathbf {x} \right\rangle | + \frac {1}{2 L} \left| \left| \nabla f (\mathbf {y}) - \nabla f (\mathbf {x}) \right| \right| ^ {2}, \forall \mathbf {x}, \mathbf {y} \in \mathbb {R} ^ {d}. \tag {A.5}
$$
Lemma 11 (Ward et al. [55]). For any non-negative sequence  $a_0, \dots, a_T$ , the following holds:
$$
\sqrt {\sum_ {t = 0} ^ {T} a _ {t}} \leq \sum_ {t = 0} ^ {T} \frac {a _ {t}}{\sqrt {\sum_ {i = 0} ^ {t} a _ {i}}} \leq 2 \sqrt {\sum_ {t = 0} ^ {T} a _ {t}}. \tag {A.6}
$$
If  $a_0 \geq 1$ , then the following holds:
$$
\sum_ {t = 0} ^ {T} \frac {a _ {t}}{\sum_ {i = 0} ^ {t} a _ {i}} \leq \log \left(\sum_ {t = 0} ^ {T} a _ {t}\right) + 1. \tag {A.7}
$$
Proof. To show equation (A.6), we proceed with the proof by induction. For  $t = 0$ , (A.6) holds trivially since  $\sqrt{a_0} \leq \sqrt{a_0} \leq 2\sqrt{a_0}$ . Assume equation (A.6) holds for  $T - 1$ . For RHS, we have:
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \frac {a _ {t}}{\sum_ {i = 0} ^ {t} a _ {i}} + \frac {a _ {T}}{\sqrt {\sum_ {i = 0} ^ {T} a _ {i}}} \leq 2 \sqrt {\sum_ {t = 0} ^ {T - 1} a _ {t}} + \frac {a _ {T}}{\sqrt {\sum_ {i = 0} ^ {T} a _ {i}}} \\ = 2 \sqrt {\sum_ {t = 0} ^ {T} a _ {t} - a _ {T}} + \frac {a _ {T}}{\sqrt {\sum_ {t = 0} ^ {T} a _ {t}}} \tag {A.8} \\ \leq 2 \sqrt {\sum_ {t = 0} ^ {T} a _ {t}}. \\ \end{array}
$$
where the last inequality is due to the fact that  $2\sqrt{x - y} + \frac{y}{\sqrt{x}} \leq 2\sqrt{x}$  for any  $x \geq y \geq 0$ . For LHS, we have:
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \frac {a _ {t}}{\sum_ {i = 0} ^ {t} a _ {i}} + \frac {a _ {T}}{\sqrt {\sum_ {i = 0} ^ {T} a _ {i}}} \geq \sqrt {\sum_ {t = 0} ^ {T - 1} a _ {t}} + \frac {a _ {T}}{\sqrt {\sum_ {i = 0} ^ {T} a _ {i}}} \\ = \sqrt {\sum_ {t = 0} ^ {T} a _ {t} - a _ {T}} + \frac {a _ {T}}{\sqrt {\sum_ {t = 0} ^ {T} a _ {t}}} \tag {A.9} \\ \geq \sqrt {\sum_ {t = 0} ^ {T} a _ {t}}. \\ \end{array}
$$
where the last inequality is due to the fact that  $\sqrt{x - y} + \frac{y}{\sqrt{x}} \geq \sqrt{x}$  for any  $x \geq y \geq 0$ .
We next show equation (A.7) by induction. For  $t = 0$ , equation (A.7) trivially holds since  $1 \leq \log(a_0) + 1$ . Assume (A.7) holds for  $T - 1$ , we have:
$$
\begin{array}{l} \sum_ {t = 0} ^ {T} \frac {a _ {t}}{\sum_ {i = 0} ^ {t} a _ {i}} \leq \log \left(\sum_ {t = 0} ^ {T - 1} a _ {t}\right) + 1 + \frac {a _ {T}}{\sum_ {i = 0} ^ {T} a _ {i}} \tag {A.10} \\ \leq \log (\sum_ {t = 0} ^ {T} a _ {t}) + 1. \\ \end{array}
$$
where the last inequality is due to the fact that  $\log (x - y) + \frac{y}{x} \leq \log (x)$  for any  $x \geq y \geq 0$  since  $e^{\frac{y}{x}} \leq \frac{1 + \frac{y}{x}}{1 - \frac{y^2}{x^2}}$ .
Lemma 12 (Dubois-Taine et al. [14, Lemma 5]). If  $x^{2} \leq a(x + b)$  for  $a \geq 0$  and  $b \geq 0$ , then it holds that:
$$
x \leq a + \sqrt {a b}. \tag {A.11}
$$
The following Lemma is an extension of Lemma 5 in [44].
Lemma 13. Let  $z_{t+1} \leq (1 - a\eta_t)z_t + \eta_t b$  and  $z_t \geq 0$  where  $a > 0, b > 0$  and  $\eta_t > 0, \eta_{t+1} \leq \eta_t, \forall t \geq 0$ . It holds that:
$$
z _ {t} \leq \max  \left\{\frac {b}{a}, z _ {0}, \eta_ {0} b \right\}, \forall t \geq 0. \tag {A.12}
$$
Proof. Since  $\eta_t$  is non-increasing,  $1 - a\eta_t \leq 0$  is non-decreasing. For any  $t \geq 0$  such that  $1 - a\eta_t \leq 0$ , we have  $z_{t+1} \leq \eta_t b \leq \eta_0 b$ . If  $1 - a\eta_t \leq 0$  for all  $t \geq 0$ , then the proof is done. Otherwise, let us assume there exists a first index  $j$  such that  $1 - a\eta_j > 0$  and we have  $z_j \leq \max\{z_0, \eta_0 b\} \coloneqq \tilde{z}_0$ . We proceed with the proof starting with the index  $j$  by induction. For  $t = j$ , the lemma trivially holds. Let us assume  $z_t \leq \max\{\frac{b}{a}, \tilde{z}_0\}$  for  $t > j$ . If  $\frac{b}{a} \geq \tilde{z}_0$ , then by induction, we have:
$$
z _ {t + 1} \leq (1 - a \eta_ {t}) \frac {b}{a} + \eta_ {t} b = \frac {b}{a}. \tag {A.13}
$$
If instead  $\frac{b}{a} \leq \tilde{z}_0$ , then by induction, we have:
$$
z _ {t + 1} \leq (1 - a \eta_ {t}) \tilde {z} _ {0} + \eta_ {t} b = \tilde {z} _ {0} - \eta_ {t} (a \tilde {z} _ {0} - b) \leq \tilde {z} _ {0}. \tag {A.14}
$$
Combining the above cases concludes the proof.
The following lemma is commonly used in the works on Polyak stepsize [34, 20].
Lemma 14. Suppose a function  $f$  is  $L$ -smooth and  $\mu$ -strongly convex, then the following holds:
$$
\frac {1}{2 L} \leq \frac {f (\mathbf {x}) - f ^ {\star}}{| | \nabla f (\mathbf {x}) | | ^ {2}} \leq \frac {1}{2 \mu}. \tag {A.15}
$$
The following lemma provides upper and lower bounds for the stepsize of AdaSPS.
Lemma 15. Suppose each  $f_{i}$  is  $L$ -smooth, then the stepsize of AdaSPS (AdaSPS) satisfies:
$$
\frac {1}{2 c _ {p} L} \frac {1}{\sqrt {\sum_ {s = 0} ^ {t} f _ {i _ {s}} (\mathbf {x} _ {s}) - \ell_ {i _ {s}} ^ {\star}}} \leq \eta_ {t} \leq \frac {f _ {i _ {t}} (\mathbf {x} _ {t}) - \ell_ {i _ {t}} ^ {\star}}{c _ {p} \| \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) \| ^ {2}} \frac {1}{\sqrt {\sum_ {s = 0} ^ {t} f _ {i _ {s}} (\mathbf {x} _ {s}) - \ell_ {i _ {s}} ^ {\star}}}. \tag {A.16}
$$
Proof. The upper bound follows from the definition of the stepsize (AdaSPS). To prove the lower bound, we note that the stepsize (AdaSPS) is composed of two parts where the first component  $\frac{f_{i_s}(\mathbf{x}_s) - \ell_{i_s}^*}{c_p||\nabla f_{i_s}(\mathbf{x}_s)||^2} \geq \frac{1}{2c_pL}$  for all  $0 \leq s \leq t$  due to (A.15), and the second component is always decreasing. Finally, recall that  $\eta_{-1} = +\infty$  and thus the proof is completed.
The following lemma provides upper and lower bounds for the stepsize of AdaSLS. We refer to Appendix D for details of the line-search procedure.
Lemma 16. Suppose each  $f_{i}$  is  $L$ -smooth, then the stepsize of AdaSLS (AdaSLS) satisfies:
$$
\min  \left\{\frac {1 - \rho}{L}, \gamma_ {\max } \right\} \frac {1}{c _ {l} \sqrt {\sum_ {s = 0} ^ {t} \gamma_ {s} | | \nabla f _ {i _ {s}} (\mathbf {x} _ {s}) | | ^ {2}}} \leq \eta_ {t} \leq \frac {\gamma_ {t}}{c _ {l} \sqrt {\sum_ {s = 0} ^ {t} \gamma_ {s} | | \nabla f _ {i _ {s}} (\mathbf {x} _ {s}) | | ^ {2}}}. \tag {A.17}
$$
Proof. The upper bound is due to the definition of the stepsize (AdaSLS). We next prove the lower bound. From the smoothness definition, the following holds for all  $\gamma_t$ :
$$
f _ {i _ {t}} \left(\mathbf {x} _ {t} - \gamma_ {t} \nabla f _ {i _ {t}} (\mathbf {x} _ {t})\right) \stackrel {(A. 4)} {\leq} f _ {i _ {t}} (\mathbf {x} _ {t}) - \gamma_ {t} \| \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) \| ^ {2} + \frac {L}{2} \gamma_ {t} ^ {2} \| \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) \| ^ {2}. \tag {A.18}
$$
For any  $0 < \gamma_t \leq \frac{2(1 - \rho)}{L}$ , we have:
$$
f _ {i t} \left(\mathbf {x} _ {t} - \gamma_ {t} \nabla f _ {i t} \left(\mathbf {x} _ {t}\right)\right) \leq f _ {i t} \left(\mathbf {x} _ {t}\right) - \rho \gamma_ {t} \| \nabla f _ {i t} (\mathbf {x} _ {t}) \| ^ {2}, \tag {A.19}
$$
which satisfies the line-search condition (4). From the procedure of Backtracking line-search (Alg. 4), if  $\gamma_{\mathrm{max}} \leq \frac{1 - \rho}{L}$ , then  $\gamma_t = \gamma_{\mathrm{max}}$  is accepted. Otherwise, since we require the decreasing factor  $\beta$  to be no smaller than  $\frac{1}{2}$  in Algorithm 4, we must have  $\gamma_t \geq \frac{2(1 - \rho)}{2L}$ . Therefore,  $\gamma_t$  is always lower bounded by  $\min \left\{\frac{1 - \rho}{L}, \gamma_{\mathrm{max}}\right\}$ . The second component of AdaSLS is always decreasing, and recall that  $\eta_{-1} = +\infty$ . The proof is thus completed.
B Proofs of main results
Proof. We follow a common proof routine for the general convex optimization [15, 14, 44]. Using the update rule of PSGD (5), we have:
$$
\begin{array}{l} \left| \left| \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} \right| \right| ^ {2} = \left| \left| \Pi_ {\mathcal {X}} \left(\mathbf {x} _ {t} - \eta_ {t} \nabla f _ {i _ {t}} (\mathbf {x} _ {t})\right) - \Pi_ {\mathcal {X}} \left(\mathbf {x} ^ {\star}\right) \right| \right| ^ {2} \\ \leq \left| \left| \mathbf {x} _ {t} - \eta_ {t} \nabla f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) - \mathbf {x} ^ {\star} \right| \right| ^ {2} \tag {B.1} \\ = | | \mathbf {x} _ {t} - \mathbf {x} ^ {\star} | | ^ {2} - 2 \eta_ {t} \langle \nabla f _ {i _ {t}} (\mathbf {x} _ {t}), \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \rangle + \eta_ {t} ^ {2} | | \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) | | ^ {2}. \\ \end{array}
$$
Dividing by  $2\eta_{t}$  and rearranging gives:
$$
\begin{array}{l} \left\langle \nabla f _ {i t} \left(\mathbf {x} _ {t}\right), \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right\rangle \\ \leq \frac {\left| \left| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right| \right| ^ {2}}{2 \eta_ {t}} - \frac {\left| \left| \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} \right| \right| ^ {2}}{2 \eta_ {t}} + \frac {\eta_ {t}}{2} \left| \left| \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) \right| \right| ^ {2} \tag {B.2} \\ = \frac {| | \mathbf {x} _ {t} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {t}} - \frac {| | \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {t + 1}} + \frac {| | \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {t + 1}} - \frac {| | \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {t}} + \frac {\eta_ {t}}{2} | | \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) | | ^ {2}. \\ \end{array}
$$
Summing from  $t = 0$  to  $t = T - 1$ , we get:
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \left\langle \nabla f _ {i _ {t}} \left(\mathbf {x} _ {t}\right), \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right\rangle \\ \leq \sum_ {t = 0} ^ {T - 1} \frac {| | \mathbf {x} _ {t} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {t}} - \frac {| | \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {t + 1}} + \frac {| | \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {t + 1}} - \frac {| | \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {t}} + \frac {\eta_ {t}}{2} | | \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) | | ^ {2} \\ \leq \frac {| | \mathbf {x} _ {0} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {0}} - \frac {| | \mathbf {x} _ {T} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {T}} + \frac {| | \mathbf {x} _ {T} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {T - 1}} - \frac {| | \mathbf {x} _ {T} - \mathbf {x} ^ {\star} | | ^ {2}}{2 \eta_ {T - 1}} + \sum_ {t = 0} ^ {T - 2} (\frac {1}{2 \eta_ {t + 1}} - \frac {1}{2 \eta_ {t}}) D ^ {2} + \sum_ {t = 0} ^ {T - 1} \frac {\eta_ {t}}{2} | | \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) | | ^ {2} \\ \leq \frac {\left\| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \right\| ^ {2}}{2 \eta_ {0}} - \frac {\left\| \mathbf {x} _ {T} - \mathbf {x} ^ {\star} \right\| ^ {2}}{2 \eta_ {T}} + \frac {\left\| \mathbf {x} _ {T} - \mathbf {x} ^ {\star} \right\| ^ {2}}{2 \eta_ {T}} + \frac {D ^ {2}}{2 \eta_ {T - 1}} + \sum_ {t = 0} ^ {T - 1} \frac {\eta_ {t}}{2} \left\| \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) \right\| ^ {2} \\ \end{array}
$$
$$
= \frac {\left\| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \right\| ^ {2}}{2 \eta_ {0}} + \frac {D ^ {2}}{2 \eta_ {T - 1}} + \sum_ {t = 0} ^ {T - 1} \frac {\eta_ {t}}{2} \| \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) \| ^ {2}, \tag {B.3}
$$
where in the second inequality, we use the decreasing property of the stepsize  $\eta_t$  which guarantees  $\frac{1}{2\eta_t} - \frac{1}{2\eta_{t-1}} \geq 0$ , and we use the fact that  $||\mathbf{x}_t - \mathbf{x}^*||^2 \leq D^2$  because of the projection step in (5). For clarity, we next separate the proof for AdaSPS and AdaSLS.
AdaSPS: We upper bound the last two terms by using Lemma 15 and we obtain:
$$
\sum_ {t = 0} ^ {T - 1} \frac {\eta_ {t}}{2} | | \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) | | ^ {2} \stackrel {(A. 1 6)} {\leq} \sum_ {t = 0} ^ {T - 1} \frac {f _ {i _ {t}} (\mathbf {x} _ {t}) - \ell_ {i _ {t}} ^ {\star}}{2 c _ {p} \sqrt {\sum_ {s = 0} ^ {t} f _ {i _ {s}} (\mathbf {x} _ {s}) - \ell_ {i _ {s}} ^ {\star}}} \stackrel {(A. 6)} {\leq} \frac {1}{c _ {p}} \sqrt {\sum_ {s = 0} ^ {T - 1} f _ {i _ {s}} (\mathbf {x} _ {s}) - \ell_ {i _ {s}} ^ {\star}}, \tag {B.4}
$$
$$
\frac {D ^ {2}}{2 \eta_ {T - 1}} \stackrel {(A. 1 6)} {\leq} c _ {p} L D ^ {2} \sqrt {\sum_ {s = 0} ^ {T - 1} f _ {i _ {s}} (\mathbf {x} _ {s}) - \ell_ {i _ {s}} ^ {\star}}. \tag {B.5}
$$
Using  $\frac{||\mathbf{x}_0 - \mathbf{x}^\star||^2}{2\eta_0} \leq \frac{D^2}{2\eta_{T-1}}$  and plugging (B.4) and (B.5) back to (B.3) gives:
$$
\sum_ {t = 0} ^ {T - 1} \left\langle \nabla f _ {i _ {t}} \left(\mathbf {x} _ {t}\right), \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right\rangle \leq \left(2 c _ {p} L D ^ {2} + \frac {1}{c _ {p}}\right) \sqrt {\sum_ {s = 0} ^ {T - 1} f _ {i _ {s}} \left(\mathbf {x} _ {s}\right) - \ell_ {i _ {s}} ^ {\star}}. \tag {B.6}
$$
Taking the expectation on both sides, we have:
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \langle \nabla f (\mathbf {x} _ {t}), \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \rangle ] \leq \left(2 c _ {p} L D ^ {2} + \frac {1}{c _ {p}}\right) \mathbb {E} \left[ \sqrt {\sum_ {s = 0} ^ {T - 1} f _ {i _ {s}} (\mathbf {x} _ {s}) - \ell_ {i _ {s}} ^ {\star}} \right] \tag {B.7} \\ = \left(2 c _ {p} L D ^ {2} + \frac {1}{c _ {p}}\right) \mathbb {E} \left[ \sqrt {\sum_ {s = 0} ^ {T - 1} f _ {i _ {s}} \left(\mathbf {x} _ {s}\right) - f _ {i _ {s}} \left(\mathbf {x} ^ {\star}\right) + f _ {i _ {s}} \left(\mathbf {x} ^ {\star}\right) - \ell_ {i _ {s}} ^ {\star}} \right]. \\ \end{array}
$$
Using the convexity assumption of  $f$  and applying Jensen's inequality to the square root function, we get:
$$
\sum_ {t = 0} ^ {T - 1} \mathbb {E} [ f (\mathbf {x} _ {t}) - f ^ {\star} ] \leq \left(2 c _ {p} L D ^ {2} + \frac {1}{c _ {p}}\right) \sqrt {\sum_ {s = 0} ^ {T - 1} \mathbb {E} [ f (\mathbf {x} _ {s}) - f ^ {\star} ] + \sigma_ {f , B} ^ {2} + \operatorname {e r r} _ {f , B} ^ {2}}, \tag {B.8}
$$
where  $\mathrm{err}_f^2,B = \mathbb{E}_{i_s}[f_{i_s}^\star -\ell_{i_s}^\star ]$  . Let  $\tau \coloneqq 2c_pLD^2 +\frac{1}{c_p}$  . Taking the square gives:
$$
\left(\sum_ {t = 0} ^ {T - 1} \mathbb {E} [ f (\mathbf {x} _ {t}) - f ^ {\star} ]\right) ^ {2} \leq \tau^ {2} \left(\sum_ {t = 0} ^ {T - 1} \mathbb {E} [ f (\mathbf {x} _ {t}) - f ^ {\star} ] + T \left(\sigma_ {f, B} ^ {2} + \operatorname {e r r} _ {f, B} ^ {2}\right)\right). \tag {B.9}
$$
We next apply Lemma 12 with  $x = \sum_{t=0}^{T-1} \mathbb{E}[f(\mathbf{x}_t) - f(\mathbf{x}^\star)]$ ,  $a = \tau^2$  and  $b = T(\sigma_{f,B}^2 + \mathrm{err}_{f,B}^2)$ :
$$
\sum_ {t = 0} ^ {T - 1} \mathbb {E} [ f (\mathbf {x} _ {t}) - f ^ {\star} ] \leq \tau^ {2} + \tau \sqrt {\sigma_ {f , B} ^ {2} + \operatorname {e r r} _ {f , B} ^ {2}} \sqrt {T}. \tag {B.10}
$$
We conclude by dividing both sides by  $T$  and using Jensen's inequality:
$$
\mathbb {E} \left[ f \left(\bar {\mathbf {x}} _ {T}\right) - f ^ {\star} \right] \leq \frac {\sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ f \left(\mathbf {x} _ {t}\right) - f ^ {\star} \right]}{T} \leq \frac {\tau^ {2}}{T} + \frac {\tau \sqrt {\sigma_ {f , B} ^ {2} + \operatorname {e r r} _ {f , B} ^ {2}}}{\sqrt {T}}. \tag {B.11}
$$
AdaSLS: The proof is almost the same as AdaSPS. We omit procedures with the same proof reasons for simplicity. We first use Lemma 16 to obtain:
$$
\sum_ {t = 0} ^ {T - 1} \frac {\eta_ {t}}{2} | | \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) | | ^ {2} \stackrel {(A. 1 7)} {\leq} \sum_ {t = 0} ^ {T - 1} \frac {\gamma_ {t} \| \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) \| ^ {2}}{2 c _ {l} \sqrt {\sum_ {s = 0} ^ {t} \gamma_ {s} \| \nabla f _ {i _ {s}} (\mathbf {x} _ {s}) \| ^ {2}}} \stackrel {(A. 6)} {\leq} \frac {1}{c _ {l}} \sqrt {\sum_ {s = 0} ^ {T - 1} \gamma_ {s} \| \nabla f _ {i _ {s}} (\mathbf {x} _ {s}) \| ^ {2}}, \tag {B.12}
$$
$$
\frac {D ^ {2}}{2 \eta_ {T - 1}} \stackrel {(A. 1 7)} {\leq} \frac {c _ {l} \sqrt {\sum_ {s = 0} ^ {T - 1} \gamma_ {s} \left\| \nabla f _ {i _ {s}} (\mathbf {x} _ {s}) \right\| ^ {2}} D ^ {2}}{2 \min  \left\{\frac {1 - \rho}{L}, \gamma_ {\max } \right\}} = \frac {\max  \left\{\frac {L}{1 - \rho} , \frac {1}{\gamma_ {\max }} \right\} c _ {l} D ^ {2}}{2} \sqrt {\sum_ {s = 0} ^ {T - 1} \gamma_ {s} \left\| \nabla f _ {i _ {s}} (\mathbf {x} _ {s}) \right\| ^ {2}}. \tag {B.13}
$$
Inequality (B.3) can then be further bounded by:
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \left\langle \nabla f _ {i _ {t}} \left(\mathbf {x} _ {t}\right), \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right\rangle \leq \left(\max  \left\{\frac {L}{1 - \rho}, \frac {1}{\gamma_ {\operatorname* {m a x}}} \right\} c _ {l} D ^ {2} + \frac {1}{c _ {l}}\right) \sqrt {\sum_ {s = 0} ^ {T - 1} \gamma_ {s} \| \nabla f _ {i _ {s}} \left(\mathbf {x} _ {s}\right) \| ^ {2}} \tag {B.14} \\ \leq \Big (\max  \Big \{\frac {L}{(1 - \rho) \sqrt {\rho}}, \frac {1}{\gamma_ {\max } \sqrt {\rho}} \Big \} c _ {l} D ^ {2} + \frac {1}{c _ {l} \sqrt {\rho}} \Big) \sqrt {\sum_ {s = 0} ^ {T - 1} f _ {i _ {s}} (\mathbf {x} _ {s}) - f _ {i _ {s}} ^ {\star}}. \\ \end{array}
$$
where we used line-search condition (4) and the fact that  $f_{i_s}(\mathbf{x}_t - \gamma_t\nabla f_{i_s}(\mathbf{x}_s))\geq f_{i_s}^\star$
Let  $\tau := \max \left\{\frac{L}{(1 - \rho)\sqrt{\rho}}, \frac{1}{\gamma_{\max} \sqrt{\rho}}\right\} c_l D^2 + \frac{1}{c_l \sqrt{\rho}}$ . We arrive at:
$$
\mathbb {E} \left[ f \left(\bar {\mathbf {x}} _ {T}\right) - f ^ {\star} \right] \leq \frac {\tau^ {2}}{T} + \frac {\tau \sigma_ {f , B}}{\sqrt {T}}. \tag {B.15}
$$
B.2 Full statement and proof for Lemma 4
Lemma 17 (Bounded iterates). Let each  $f_{i}$  be  $\mu$ -strongly convex and  $L$ -smooth. For any  $t \in \mathbb{N}$ , the iterates of SGD with AdaSPS or AdaSLS satisfy:
$$
\left| \left| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right| \right| ^ {2} \leq D _ {\max } := \max  \left\{\left| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \right| ^ {2}, \frac {2 \sigma_ {\max } ^ {2} + b}{\mu}, (2 \sigma_ {\max } ^ {2} + b) \eta_ {0} \right\}, \tag {B.16}
$$
where  $\sigma_{\mathrm{max}}^2 := \max_{i_t} \left\{f_{i_t}(\mathbf{x}^\star) - \ell_{i_t}^\star\right\}$ ,  $b := 1 / \left(4c_p^3 \sqrt{f_{i_0}(\mathbf{x}_0) - \ell_{i_0}^\star}\right)$  for AdaSPS and  $\sigma_{\mathrm{max}}^2 := \max_{i_t} \left\{f_{i_t}(\mathbf{x}^\star) - f_{i_t}^\star\right\}$ ,  $b := 1 / \left(4c_l^3 \rho^2 \sqrt{\gamma_0||\nabla f_{i_0}(\mathbf{x}_0)||^2}\right)$  for AdaSLS.
Proof. By strong convexity of  $f_{i_t}$ , the iterates generated by SGD satisfy:
$$
\begin{array}{l} \left| \left| \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} \right| \right| ^ {2} = \left| \left| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right| \right| ^ {2} - 2 \eta_ {t} \left\langle \nabla f _ {i _ {t}} (\mathbf {x} _ {t}), \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right\rangle + \eta_ {t} ^ {2} \left| \left| \nabla f _ {i _ {t}} (\mathbf {x} _ {t}) \right| \right| ^ {2} \\ \stackrel {(A. 2)} {\leq} \left\| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right\| ^ {2} - 2 \eta_ {t} \left(f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) - f _ {i _ {t}} \left(\mathbf {x} ^ {\star}\right) + \frac {\mu}{2} \left\| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right\| ^ {2}\right) + \eta_ {t} ^ {2} \| \nabla f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) \| ^ {2} \tag {B.17} \\ = \left(1 - \eta_ {t} \mu\right) \left\| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right\| ^ {2} - 2 \eta_ {t} \left(f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) - f _ {i _ {t}} \left(\mathbf {x} ^ {\star}\right)\right) + \eta_ {t} ^ {2} \left\| \nabla f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) \right\| ^ {2}. \\ \end{array}
$$
We next separate the proofs for clarity.
AdaSPS: Plugging in the upper bound of  $\eta_t$  in Lemma 15, we obtain:
$$
\begin{array}{l} \left| \left| \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} \right| \right| ^ {2} \\ \stackrel {\mathrm {(A . 1 6)}} {\leq} (1 - \eta_ {t} \mu) | | \mathbf {x} _ {t} - \mathbf {x} ^ {\star} | | ^ {2} - 2 \eta_ {t} (f _ {i _ {t}} (\mathbf {x} _ {t}) - f _ {i _ {t}} (\mathbf {x} ^ {\star})) + \eta_ {t} \frac {f _ {i _ {t}} (\mathbf {x} _ {t}) - \ell_ {i _ {t}} ^ {\star}}{c _ {p} \sqrt {\sum_ {s = 0} ^ {t} f _ {i _ {s}} (\mathbf {x} _ {t}) - \ell_ {i _ {s}} ^ {\star}}} \\ = (1 - \eta_ {t} \mu) | | \mathbf {x} _ {t} - \mathbf {x} ^ {\star} | | ^ {2} + 2 \eta_ {t} (f _ {i _ {t}} (\mathbf {x} ^ {\star}) - \ell_ {i _ {t}} ^ {\star}) - 2 \eta_ {t} (f _ {i _ {t}} (\mathbf {x} _ {t}) - \ell_ {i _ {t}} ^ {\star}) + \eta_ {t} \frac {f _ {i _ {t}} (\mathbf {x} _ {t}) - \ell_ {i _ {t}} ^ {\star}}{c _ {p} \sqrt {\sum_ {s = 0} ^ {t} f _ {i _ {s}} (\mathbf {x} _ {t}) - \ell_ {i _ {s}} ^ {\star}}} \\ \leq \left(1 - \eta_ {t} \mu\right) \left\| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right\| ^ {2} + 2 \eta_ {t} \sigma_ {\max } ^ {2} - 2 \eta_ {t} \underbrace {\left(f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) - \ell_ {i _ {t}} ^ {\star}\right)} _ {\geq 0} + \eta_ {t} \frac {f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) - \ell_ {i _ {t}} ^ {\star}}{c _ {p} \sqrt {\sum_ {s = 0} ^ {t} f _ {i _ {s}} \left(\mathbf {x} _ {t}\right) - \ell_ {i _ {s}} ^ {\star}}}, \tag {B.18} \\ \end{array}
$$
where  $\sigma_{\max}^2 \coloneqq \max_{i_t} \{f_{i_t}(\mathbf{x}^\star) - \ell_{i_t}^\star\}$ .
We now split the proof into two cases. Firstly, if  $c_p \sqrt{\sum_{s=0}^t f_{i_s}(\mathbf{x}_t) - \ell_{i_s}^\star} \leq \frac{1}{2}$ , then it follows that:
$$
f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) - \ell_ {i _ {t}} ^ {\star} \leq \left(\frac {1}{2 c _ {p}}\right) ^ {2} \quad \text {a n d} \quad \sum_ {s = 0} ^ {t} f _ {i _ {s}} \left(\mathbf {x} _ {t}\right) - \ell_ {i _ {s}} ^ {\star} \geq f _ {i _ {0}} \left(\mathbf {x} _ {0}\right) - \ell_ {i _ {0}} ^ {\star}. \tag {B.19}
$$
Plugging in the above bounds, we have:
$$
\left\| \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} \right\| ^ {2} \leq \left(1 - \eta_ {t} \mu\right) \left\| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right\| ^ {2} + \eta_ {t} \left(2 \sigma_ {\max } ^ {2} + \frac {1}{4 c _ {p} ^ {3} \sqrt {f _ {i _ {0}} (\mathbf {x} _ {0}) - \ell_ {i _ {0}} ^ {\star}}}\right). \tag {B.20}
$$
We conclude by applying Lemma 13 with  $z_{t} = ||\mathbf{x}_{t} - \mathbf{x}^{\star}||^{2}, a = \mu, b = (2\sigma_{\max}^{2} + \frac{1}{4c_{p}^{3}\sqrt{f_{i_{0}}(\mathbf{x}_{0}) - \ell_{i_{0}}^{\star}}})$ . Secondly, if instead  $c_{p}\sqrt{\sum_{s=0}^{t} f_{i_{s}}(\mathbf{x}_{t}) - \ell_{i_{s}}^{\star}} \geq \frac{1}{2}$ , then we have:
$$
- 2 \eta_ {t} \left(f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) - \ell_ {i _ {t}} ^ {\star}\right) + \eta_ {t} \frac {f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) - \ell_ {i _ {t}} ^ {\star}}{c _ {p} \sqrt {\sum_ {s = 0} ^ {t} f _ {i _ {s}} \left(\mathbf {x} _ {t}\right) - \ell_ {i _ {s}} ^ {\star}}} \leq 0, \tag {B.21}
$$
and consequently we can apply Lemma 13 with  $z_{t} = ||\mathbf{x}_{t} - \mathbf{x}^{\star}||^{2}, a = \mu, b = 2\sigma_{\max}^{2}$ .
AdaSLS: Similarly, by plugging the upper bound of  $\eta_t$  in Lemma 16, we obtain: