SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: Comparison of SBCGI, SBCGF, aR-IP-SeG, and DBGD-Sto for solving Problem (3)
(a) Lower-level gap
Figure 2: Comparison of SBCGI, SBCGF, aR-IP-SeG, and DBGD-Sto for solving Problem (5).
(c) Recovery rate
that after the initial stage, SBCGI increases slightly in terms of all the performance criteria, because SBCGI (1) only takes one sample per iteration and uses a decreasing step-size while SBCGF takes  $\sqrt{n}$  samples per iteration and uses a small constant stepsize, demonstrating a more robust performance.
Dictionary learning. To test our methods on problems with non-convex upper-level we consider problem (5) on a synthetic dataset with a similar setup to [12]. We first construct the true dictionary  $\tilde{\mathbf{D}}^{*} \in \mathbb{R}^{25 \times 50}$  comprising of 50 basis vectors in  $\mathbb{R}^{25}$ . All entries of these basis vectors are drawn from the standard Gaussian distribution and then normalized to have unit  $\ell_{2}$ -norm. We also generate two more dictionaries  $\mathbf{D}^{*}$  and  $\mathbf{D}'^{*}$  consisting of 40 and 20 basis vectors in  $\tilde{\mathbf{D}}^{*}$ , respectively (thus they share at least 10 bases). These two datasets  $\mathbf{A} = \{\mathbf{a}_{1}, \ldots, \mathbf{a}_{250}\}$  and  $\mathbf{A}' = \{\mathbf{a}_{1}', \ldots, \mathbf{a}_{250}'\}$  are constructed as  $\mathbf{a}_{i} = \mathbf{D}^{*}\mathbf{x}_{i} + \mathbf{n}_{i}$ , for  $i = 1, \ldots, 250$ , and  $\mathbf{a}_{k}' = \mathbf{D}'^{*}\mathbf{x}_{k}' + \mathbf{n}_{k}'$ , for  $k = 1, \ldots, 250$ , where  $\{\mathbf{x}_{i}\}_{i=1}^{250}$ ,  $\{\mathbf{x}_{k}'\}_{k=1}^{250}$  are coefficient vectors and  $\{\mathbf{n}_{i}\}_{i=1}^{250}$ ,  $\{\mathbf{n}_{k}'\}_{k=1}^{250}$  are random Gaussian noises. As neither  $\mathbf{A}$  nor  $\mathbf{A}'$  includes all the elements of  $\tilde{\mathbf{D}}^{*}$ , it is important to renew our dictionary by using the new dataset  $\mathbf{A}'$  while maintaining the knowledge from the old dataset  $\mathbf{A}$ .
In our experiment, we initially solve the standard dictionary learning problem employing dataset  $\mathbf{A}$ , achieving the initial dictionary  $\hat{\mathbf{D}}$  and coefficient vectors  $\{\hat{\mathbf{x}}\}_{i=1}^{250}$ . We define the lower-level objective as the reconstruction error on  $\mathbf{A}$  using  $\{\hat{\mathbf{x}}\}_{i=1}^{250}$ , and the upper-level objective as the error on new dataset  $\mathbf{A}'$ . We compare our algorithms with aR-IP-SeG and DBGD (stochastic version), measuring performance with the recovery rate of true basis vectors. Note that a basis vector  $\tilde{\mathbf{d}}_i^*$  in  $\tilde{\mathbf{D}}^*$  is considered as successfully recovered if there exists  $\tilde{\mathbf{d}}_j$  in  $\tilde{\mathbf{D}}$  such that  $|\langle \tilde{\mathbf{d}}_i^*, \tilde{\mathbf{d}}_j \rangle| > 0.9$  (for more details of the experiment setup see Appendix F). In Figure 2(a), we observe SBCGF converges faster than any other method regarding the lower-level objective. While SBCGI has the second-best performance in terms of the lower-level gap, aR-IP-SeG and DBGD-sto perform poorly compared with SBCGI and SBCGF. In Figures 2(b) and (c), we see that SBCGI, SBCGF, and DBGD-sto achieve good results in terms of the upper-level objective and the recovery rate. However, aR-IP-SeG still performs poorly in terms of both criteria, which matches the theoretical results in Table 1.
The research of J. Cao, R. Jiang, and A. Mokhtari is supported in part by NSF Grant 2127697 and the NSF AI Institute for Foundations of Machine Learning (IFML) at UT Austin. The research of N. Abolfazli and E. Yazdandoost Hamedani is supported by NSF Grant 2127696.
[37] Jiahao Xie, Zebang Shen, Chao Zhang, Boyu Wang, and Hui Qian. Efficient projection-free online methods with stochastic recursive gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6446-6453, 2020.
[38] Aryan Mokhtari, Asuman Ozdaglar, and Ali Jadbabaie. Escaping saddle points in constrained optimization. Advances in Neural Information Processing Systems, 31, 2018.
[39] Aleksandr Beznosikov, David Dobre, and Gauthier Gidel. Sarah frank-wolfe: Methods for constrained optimization with best rates and practical features. arXiv preprint arXiv:2304.11737, 2023.
[40] Alp Yurtsever, Suvrit Sra, and Volkan Cevher. Conditional gradient methods via stochastic path-integrated differential estimator. In International Conference on Machine Learning, pages 7282-7291. PMLR, 2019.
[41] Michael C Grant and Stephen P Boyd. Graph implementations for nonsmooth convex programs. In *Recent advances in learning and control*, pages 95–110. Springer, 2008.
[42] Michael Grant and Stephen Boyd. Cvx: Matlab software for disciplined convex programming, version 2.1, 2014.
[43] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[44] Iosif Pinelis. Optimum bounds for the distributions of martingales in banach spaces. The Annals of Probability, pages 1679-1706, 1994.
[45] Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael I Jordan. A short note on concentration inequalities for random vectors with subgaussian norm. arXiv preprint arXiv:1902.03736, 2019.
[46] Tong Zhang. Learning bounds for kernel regression using effective data dimensionality. Neural Computation, 17(9):2077-2098, 2005.
A Additional Motivating Examples
The bilevel optimization problem in (1) provides a versatile framework that covers a broad class of optimization problems. In addition to the motivating examples provided in the main body of the paper, here we also provide a generic example of stochastic convex constrained optimization that can be formulated as (1). We further present a more general form of the examples covered in the main body.
Stochastic convex optimization with many conic constraints: Consider the following convex optimization problem
$$
\min _ {\mathbf {x} \in \mathbb {R} ^ {d}} \mathbb {E} [ \tilde {f} (\mathbf {x}, \theta) ] \qquad \mathrm {s . t .} \quad h (\mathbf {x}, \xi) \in - \mathcal {K}, \forall \xi \in \Omega ,
$$
where  $\mathcal{K} \subseteq \mathbb{R}^d$  is a closed convex cone. This problem can be formulated as a special case of (1) by letting  $\tilde{g}(\mathbf{x},\xi) = \frac{1}{2} d_{-\mathcal{K}}^2(h(\mathbf{x},\xi))$  where  $d_{-\mathcal{K}}(\cdot) \triangleq \| \cdot - \mathcal{P}_{-\mathcal{K}}(\cdot)\|$  denotes the distance function and  $\mathcal{P}_{-\mathcal{K}}(\cdot)$  denotes the projection map. Our proposed framework provides an efficient method for solving this class of problems when the projections onto  $\mathcal{K}$  can be computed efficiently, while the projection onto the preimage  $h^{-1}(-\mathcal{K},\xi)$  is not practical, e.g., when  $\mathcal{K}$  is the positive semidefinite cone, computing a projection onto the preimage set requires solving a nonlinear SDP.
A.1 Lexicographic optimization
Example 1 (over-parameterized regression) can be generalized as a broader class of problem, which is known as lexicographic optimization [13] and uses the secondary loss to improve generalization. The problem can be formulated as the following stochastic simple bilevel optimization problem,
$$
\min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {d}} \mathcal {L} (\boldsymbol {\beta}) \quad \text {s . t .} \quad \boldsymbol {\beta} \in \underset {\theta \in \mathcal {Z}} {\arg \min } \ell_ {\operatorname {t r}} (\theta) = \mathbb {E} _ {\mathcal {D} _ {t r}} [ \ell (y, \hat {y} _ {\theta} (\mathbf {x})) ] \tag {17}
$$
In general, the lower-level problem could have multiple optimal solutions and be very sensitive to small perturbations. To tackle the issue, we use a secondary criterion  $\mathcal{L}(\cdot)$  to select some of the optimal solutions with our desired properties. For instance, we can find the optimal solutions with minimal  $\ell_2$ -norm by letting  $\mathcal{L}(\beta) = \| \beta \|^2$ , which is also known as Lexicographic  $\ell_2$  Regularization.
A.2 Lifelong learning
Example 2 (dictionary learning) is an instance of a popular framework known as lifelong learning, which can be formulated as follows,
$$
\min  _ {\boldsymbol {\beta}} \frac {1}{n ^ {\prime}} \sum_ {i = 1} ^ {n ^ {\prime}} \ell \left(\left\langle \mathbf {x} _ {i} ^ {\prime}, \boldsymbol {\beta} \right\rangle , y _ {i} ^ {\prime}\right) \quad \text {s . t .} \quad \sum_ {\left(\mathbf {x} _ {i}, y _ {i}\right) \in \mathcal {M}} \ell \left(\left\langle \mathbf {x} _ {i}, \boldsymbol {\beta} \right\rangle , y _ {i}\right) \leq \sum_ {\left(\mathbf {x} _ {i}, y _ {i}\right) \in \mathcal {M}} \ell \left(\left\langle \mathbf {x} _ {i}, \boldsymbol {\beta} ^ {(t - 1)} \right\rangle , y _ {i}\right) \tag {18}
$$
In this problem, the objective is the training loss on the current tasks  $\mathcal{D}_t = \{\left(\mathbf{x}_i', y_i'\right)\}_{i=1}^{n'}$ . While the constraint enforces that the model parameterized by  $\beta$  performs no worse than the previous one on the episodic memory  $\mathcal{M}$  (i.e., data samples from all the past tasks).
In the paper, we discuss a variant of the problem above, where we slightly change the constraint and ensure that the current model also minimizes the error on the past tasks. It can be formulated as the following finite-sum/stochastic simple bilevel optimization problem [12],
$$
\min  _ {\boldsymbol {\beta}} \frac {1}{n ^ {\prime}} \sum_ {i = 1} ^ {n ^ {\prime}} \ell \left(\left\langle \mathbf {x} _ {i} ^ {\prime}, \boldsymbol {\beta} \right\rangle , y _ {i} ^ {\prime}\right) \quad \text {s . t .} \quad \boldsymbol {\beta} \in \underset {\mathbf {z}} {\operatorname {a r g m i n}} \sum_ {\left(\mathbf {x} _ {i}, y _ {i}\right) \in \mathcal {M}} \ell \left(\left\langle \mathbf {x} _ {i}, \mathbf {z} \right\rangle , y _ {i}\right). \tag {19}
$$
B Supporting lemmas
Before we proceed to the proof for Lemma 4.1, we present the following technical lemma, which gives us an upper bound for a complex term appearing in the following analysis.
Lemma B.1. Define  $\rho_t = 1 / (t + 1)^\omega$  where  $\omega \in (0,1]$  and  $t \geq 1$ . For all  $t \geq 2$ , let  $\{s_t\}$  be a sequence of real numbers given by
$$
s _ {t} = \sum_ {\tau = 2} ^ {t} \left(\rho_ {\tau} \prod_ {k = \tau} ^ {t} (1 - \rho_ {k})\right) ^ {2}.
$$
$$
s _ {t} \leq \frac {1}{(t + 1) ^ {\omega}}. \tag {20}
$$
Proof. We prove the result by induction. For  $t = 2$ , we can verify that
$$
s _ {2} = \left(\frac {1}{3 ^ {\omega}} \cdot \frac {3 ^ {\omega} - 1}{3 ^ {\omega}}\right) ^ {2} \leq \frac {1}{3 ^ {2 \omega}} \leq \frac {1}{3 ^ {\omega}}.
$$
Now we suppose that the inequality in (20) holds when  $t = T$  for some  $T \geq 2$ , i.e.,
$$
s _ {T} = \sum_ {\tau = 2} ^ {T} \left(\rho_ {\tau} \prod_ {k = \tau} ^ {T} \left(1 - \rho_ {k}\right)\right) ^ {2} \leq \frac {1}{(t + 1) ^ {\omega}}.
$$
First note that the sequence  $\{s_t\}$  satisfies the following recurrence relation:
$$
\begin{array}{l} s _ {T + 1} = \sum_ {\tau = 2} ^ {T + 1} \left(\rho_ {\tau} \prod_ {k = \tau} ^ {T + 1} \left(1 - \rho_ {k}\right)\right) ^ {2} = \left(1 - \rho_ {T + 1}\right) ^ {2} \sum_ {\tau = 2} ^ {T + 1} \left(\rho_ {\tau} \prod_ {k = \tau} ^ {T} \left(1 - \rho_ {k}\right)\right) ^ {2} \\ = (1 - \rho_ {T + 1}) ^ {2} \left[ \sum_ {\tau = 2} ^ {T} \left(\rho_ {\tau} \prod_ {k = \tau} ^ {T} (1 - \rho_ {k})\right) ^ {2} + \rho_ {T + 1} ^ {2} \right] \\ = (1 - \rho_ {T + 1}) ^ {2} \left(s _ {T} + \rho_ {T + 1} ^ {2}\right). \\ \end{array}
$$
Moreover, since  $\omega \in (0,1]$ , we have  $(T + 2)^{\omega} - 1 \leq (t + 1)^{\omega}$ . Therefore, we obtain
$$
\begin{array}{l} s _ {T + 1} \leq \left(\frac {(T + 2) ^ {\omega} - 1}{(T + 2) ^ {\omega}}\right) ^ {2} \left(\frac {1}{(t + 1) ^ {\omega}} + \frac {1}{(T + 2) ^ {2 \omega}}\right) \\ \leq \frac {((T + 2) ^ {\omega} - 1) (t + 1) ^ {\omega}}{(T + 2) ^ {2 \omega}} \left(\frac {1}{(t + 1) ^ {\omega}} + \frac {1}{(T + 1) ^ {2 \omega}}\right) \\ = \frac {(T + 2) ^ {\omega} - 1}{(T + 2) ^ {2 \omega}} \frac {(T + 1) ^ {\omega} + 1}{(T + 1) ^ {\omega}} \\ = \frac {(T + 2) ^ {\omega} (t + 1) ^ {\omega} + (T + 2) ^ {\omega} - 1 - (t + 1) ^ {\omega}}{(T + 2) ^ {2 \omega} (t + 1) ^ {\omega}} \\ \leq \frac {(T + 2) ^ {\omega} (t + 1) ^ {\omega}}{(T + 2) ^ {2 \omega} (t + 1) ^ {\omega}} = \frac {1}{(T + 2) ^ {\omega}}. \\ \end{array}
$$
By induction, the inequality in (20) holds for all  $t \geq 2$ .
Now we proceed to prove Lemma 4.1.
Proof of Lemma 4.1. We show the proof of part (i) here. The proof of part (ii) is very similar to part (i). The first step is to reformulate  $\mathbf{e}_t = \widehat{\nabla} g_t - \nabla g(\mathbf{x}_t)$  as the sum of a martingale difference sequence. For  $t\geq 1$ , by unrolling the recurrence we have
$$
\begin{array}{l} \mathbf {e} _ {t} = \left(1 - \beta_ {t}\right) \mathbf {e} _ {t - 1} + \beta_ {t} \left(\nabla \tilde {g} \left(\mathbf {x} _ {t}, \xi_ {t}\right) - \nabla g \left(\mathbf {x} _ {t}\right)\right) \\ + \left(1 - \beta_ {t}\right) \left(\nabla \tilde {g} \left(\mathbf {x} _ {t}, \xi_ {t}\right) - \nabla \tilde {g} \left(\mathbf {x} _ {t - 1}, \xi_ {t}\right) - \left(\nabla g \left(\mathbf {x} _ {t}\right) - \nabla g \left(\mathbf {x} _ {t - 1}\right)\right) \right. \\ = \prod_ {k = 2} ^ {t} (1 - \beta_ {k}) \mathbf {e} _ {1} + \sum_ {\tau = 2} ^ {t} \prod_ {k = \tau} ^ {t} (1 - \beta_ {k}) (\nabla \tilde {g} (\mathbf {x} _ {\tau}, \xi_ {\tau}) - \nabla \tilde {g} (\mathbf {x} _ {\tau - 1}, \xi_ {\tau}) - (\nabla g (\mathbf {x} _ {\tau}) - \nabla g (\mathbf {x} _ {\tau - 1})) \\ + \sum_ {\tau = 2} ^ {t} \beta_ {\tau} \prod_ {k = \tau + 1} ^ {t} (1 - \beta_ {k}) (\nabla \tilde {g} (\mathbf {x} _ {\tau}, \xi_ {\tau}) - \nabla g (\mathbf {x} _ {\tau})). \tag {21} \\ \end{array}
$$
Thus, we can write  $\mathbf{e}_t$  as the sum  $\mathbf{e}_t = \sum_{\tau=1}^t \zeta_\tau$ , where we define  $\zeta_1 = \prod_{k=2}^t (1 - \beta_k) \mathbf{e}_1$  and
$$
\begin{array}{l} \zeta_ {\tau} = \prod_ {k = \tau} ^ {t} (1 - \beta_ {k}) (\nabla \tilde {g} (\mathbf {x} _ {\tau}, \xi_ {\tau}) - \nabla \tilde {g} (\mathbf {x} _ {\tau - 1}, \xi_ {\tau}) - (\nabla g (\mathbf {x} _ {\tau}) - \nabla g (\mathbf {x} _ {\tau - 1})) (22) \\ + \beta_ {\tau} \prod_ {k = \tau + 1} ^ {t} (1 - \beta_ {k}) (\nabla \tilde {g} (\mathbf {x} _ {\tau}, \xi_ {\tau}) - \nabla g (\mathbf {x} _ {\tau})) (23) \\ \end{array}
$$
for  $\tau > 1$ . Recall that  $\mathbf{e}_1 = \nabla \tilde{g}(\mathbf{x}_t, \zeta_1) - \nabla g(\mathbf{x}_1)$ . We observe that  $\mathbb{E}[\zeta_\tau | \mathcal{F}_{\tau-1}] = 0$  where  $\mathcal{F}_{\tau-1}$  is the  $\sigma$ -field generated by  $\{\mathbf{x}_1, \xi_1, \dots, \mathbf{x}_{\tau-1}, \xi_{\tau-1}\}$ . Therefore,  $\{\zeta_\tau\}_{\tau=1}^t$  is a martingale difference sequence.
Next, we derive upper bounds of  $\| \zeta_{\tau} \|$ . To begin with, we observe that for any  $\tau = 1,2,\ldots ,t$
$$
\prod_ {k = \tau} ^ {t} (1 - \beta_ {k}) = \prod_ {k = \tau} ^ {t} \left(1 - \frac {1}{(k + 1) ^ {\omega}}\right) = \prod_ {k = \tau} ^ {t} \frac {(k + 1) ^ {\omega} - 1}{(k + 1) ^ {\omega}} \leq \prod_ {k = \tau} ^ {t} \frac {k ^ {\alpha}}{(k + 1) ^ {\omega}} = \frac {\tau^ {\omega}}{(t + 1) ^ {\omega}}, \tag {24}
$$
where we used the fact that  $(k + 1)^{\omega} - 1\leq k^{\omega}$  in the last inequality. By using the above inequality, we can bound  $\| \zeta_1\|$  as follows:
$$
\| \zeta_ {1} \| = \prod_ {k = 2} ^ {t} (1 - \beta_ {k}) \| \mathbf {e} _ {1} \| \leq \frac {2 ^ {\omega}}{(t + 1) ^ {\omega}} \| \nabla \tilde {g} (\mathbf {x} _ {1}, \xi_ {1}) - \nabla g (\mathbf {x} _ {1}) \| = \frac {2 ^ {\omega} \sigma_ {1}}{(t + 1) ^ {\omega}} \frac {\| \nabla \tilde {g} (\mathbf {x} _ {1} , \xi_ {1}) - \nabla g (\mathbf {x} _ {1}) \|}{\sigma_ {1}}.
$$
Define  $c_{1} = \frac{2^{\omega}\sigma_{g}}{(T + 1)^{\omega}}$ , then by Assumption 2.3(ii) we have  $\mathbb{E}[\exp (\| \zeta_1\| ^2 /c_1^2)]\leq \exp (1)$ . Moreover, for  $\tau >1$ , by triangle inequality,  $\| \zeta_{\tau}\|$  can be bounded by
$$
\begin{array}{l} \| \zeta_ {\tau} \| \leq \prod_ {k = \tau} ^ {t} (1 - \beta_ {k}) (\| \nabla \tilde {g} (\mathbf {x} _ {\tau}, \xi_ {\tau}) - \nabla \tilde {g} (\mathbf {x} _ {\tau - 1}, \xi_ {\tau}) \| + \| (\nabla g (\mathbf {x} _ {\tau}) - \nabla g (\mathbf {x} _ {\tau - 1}) \|) \\ + \beta_ {\tau} \prod_ {k = \tau + 1} ^ {t} (1 - \beta_ {k}) \| \nabla \tilde {g} (\mathbf {x} _ {\tau}, \xi_ {\tau}) - \nabla g (\mathbf {x} _ {\tau}) \| \\ \leq 2 L _ {g} \| \mathbf {x} _ {\tau} - \mathbf {x} _ {\tau - 1} \| \prod_ {k = \tau} ^ {t} (1 - \beta_ {k}) + \| \nabla \tilde {g} (\mathbf {x} _ {\tau}, \xi_ {\tau}) - \nabla g (\mathbf {x} _ {\tau}) \| \beta_ {\tau} \prod_ {k = \tau + 1} ^ {t} (1 - \beta_ {k}) \\ = 2 L _ {g} \gamma_ {\tau} D \prod_ {k = \tau} ^ {t} (1 - \beta_ {k}) + \| \nabla \tilde {g} (\mathbf {x} _ {\tau}, \xi_ {\tau}) - \nabla g (\mathbf {x} _ {\tau}) \| \beta_ {\tau} \prod_ {k = \tau + 1} ^ {t} (1 - \beta_ {k}) \tag {25} \\ \leq 2 L _ {g} D \beta_ {\tau} \prod_ {k = \tau} ^ {t} (1 - \beta_ {k}) + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \| \nabla \tilde {g} (\mathbf {x} _ {\tau}, \xi_ {\tau}) - \nabla g (\mathbf {x} _ {\tau}) \| \beta_ {\tau} \prod_ {k = \tau} ^ {t} (1 - \beta_ {k}) \\ = \left(2 L _ {g} D + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \| \nabla \tilde {g} (\mathbf {x} _ {\tau}, \xi_ {\tau}) - \nabla g (\mathbf {x} _ {\tau}) \|\right) \beta_ {\tau} \prod_ {k = \tau} ^ {t} (1 - \beta_ {k}) \\ = \left(2 L _ {g} D + \frac {3 ^ {\omega} \sigma_ {g}}{3 ^ {\omega} - 1} \frac {\| \nabla \tilde {g} (\mathbf {x} _ {\tau} , \boldsymbol {\xi} _ {\tau}) - \nabla g (\mathbf {x} _ {\tau}) \|}{\sigma_ {g}}\right) \beta_ {\tau} \prod_ {k = \tau} ^ {t} (1 - \beta_ {k}) \\ \end{array}
$$
Define  $c_{\tau} = (2L_gD + \frac{3^\omega\sigma_g}{3^{\omega} - 1})\beta_\tau \prod_{k = \tau}^t (1 - \beta_k)$ . Note that if we have  $\mathbb{E}[\exp (X_1^2 /c_1^2)]\leq 1$  and  $\mathbb{E}[\exp (X_2^2 /c_2^2)]\leq 1$ , then we have  $\mathbb{E}[\exp ((X_1 + X_2)^2 /(c_1 + c_2)^2)]\leq 1$  [43]. Thus, we have  $\mathbb{E}[\exp (\| \zeta_{\tau}\|^{2} / c_{\tau}^{2})]\leq \exp (1)$  for all  $1\leq \tau \leq t$ . Hence by proposition E.2, with probability  $1 - \delta'$
$$
\left\| \mathbf {e} _ {t} \right\| \leq c \cdot \sqrt {\sum_ {\tau = 1} ^ {t} c _ {\tau} ^ {2} \log \frac {2 d}{\delta^ {\prime}}} \tag {26}
$$
where  $c$  is an absolute constant,  $d$  is the number of dimension, and  $\sum_{\tau=1}^{T} c_{\tau}^{2}$  can be bounded by Lemma B.1 as follows,
$$
\begin{array}{l} \sum_ {\tau = 1} ^ {t} c _ {\tau} ^ {2} = c _ {1} ^ {2} + \sum_ {\tau = 2} ^ {t} c _ {\tau} ^ {2} = \frac {2 ^ {2 \omega} \sigma_ {g} ^ {2}}{(T + 1) ^ {2 \omega}} + (2 L _ {g} D + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \sigma_ {g}) ^ {2} \sum_ {\tau = 2} ^ {T} (\beta_ {\tau} \prod_ {k = \tau} ^ {T} (1 - \beta_ {k})) ^ {2} \\ \leq \frac {2 ^ {2 \omega} \sigma_ {g} ^ {2}}{(T + 1) ^ {2 \omega}} + \frac {\left(2 L _ {g} D + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \sigma_ {g}\right) ^ {2}}{(t + 1) ^ {\omega}} \tag {27} \\ \leq \frac {((\sqrt {2}) ^ {\omega} \sigma_ {g}) ^ {2}}{(t + 1) ^ {\omega}} + \frac {(2 L _ {g} D + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \sigma_ {g}) ^ {2}}{(t + 1) ^ {\omega}} \\ \leq \frac {2 (2 L _ {g} D + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \sigma_ {g}) ^ {2}}{(t + 1) ^ {\omega}} \\ \end{array}
$$
where the last inequality follows from the fact that  $(\sqrt{2})^{\omega} \leq 3^{\omega} / (3^{\omega} - 1)$  for any  $\omega \in (0,1]$ . Combining (26) and (27), we have with probability at least  $1 - \delta'$ ,
$$
\left\| \nabla g \left(\mathbf {x} _ {t}\right) - \widehat {\nabla g} _ {t} \right\| \leq c \sqrt {2} \left(2 L _ {g} D + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \sigma_ {g}\right) (t + 1) ^ {- \omega / 2} \sqrt {\log \left(2 d / \delta^ {\prime}\right)} \stackrel {\text {d e f}} {=} K _ {1, t} \tag {28}
$$
Similarly with probability at least  $1 - \delta'$ ,
$$
\left| g \left(\mathbf {x} _ {t}\right) - \hat {g} _ {t} \right| \leq c \sqrt {2} \left(2 L _ {l} D + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \sigma_ {l}\right) (t + 1) ^ {- \omega / 2} \sqrt {\log \left(2 d / \delta^ {\prime}\right)} \stackrel {\text {d e f}} {=} K _ {0, t} \tag {29}
$$
and with probability at least  $1 - \delta'$ ,
$$
\left\| \nabla f \left(\mathbf {x} _ {t}\right) - \widehat {\nabla} f _ {t} \right\| \leq c \sqrt {2} \left(2 L _ {f} D + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \sigma_ {f}\right) (t + 1) ^ {- \omega / 2} \sqrt {\log \left(2 d / \delta^ {\prime}\right)} \stackrel {\text {d e f}} {=} K _ {2, t} \tag {30}
$$
where  $c$  is an absolute constant and  $d$  is the dimension of vectors. We can use union bound to obtain that these three inequalities hold for at least probability  $1 - 3\delta' = 1 - \delta$ . For simplicity, we define constant  $A_1^\omega$  and  $A_0^\omega$  such that,
$$
A _ {1} ^ {\omega} (t + 1) ^ {- \omega / 2} \sqrt {\log (6 d / \delta)} = K _ {1, t} \quad \text {a n d} \quad A _ {0} ^ {\omega} (t + 1) ^ {- \omega / 2} \sqrt {\log (6 d / \delta)} = K _ {0, t} \tag {31}
$$
and similarly  $A_2^\omega (t + 1)^{-\omega /2}\sqrt{\log(6d / \delta)} = K_{2,t}$
Proof. Let us define  $t_0 \triangleq \lfloor t / q \rfloor$  for any  $t \in \{0, \dots, T - 1\}$ , then whenever  $t = t_0q$  according to the Algorithm 2 a full batch of sample gradients are selected, hence,  $\widehat{\nabla g}_t = \nabla g(\mathbf{x}_t)$ ; otherwise, the error of computing a sample gradient can be expressed as follows
$$
\epsilon_ {t, i} = \frac {1}{S} (\nabla g _ {\mathcal {S} (i)} (\mathbf {x} _ {t}) - \nabla g _ {\mathcal {S} (i)} (\mathbf {x} _ {t - 1}) - \nabla g (\mathbf {x} _ {t}) + \nabla g (\mathbf {x} _ {t - 1})), \tag {32}
$$
where  $i$  is the index with  $\mathcal{S}(i)$  denoting the  $i$ -th random component function selected at iteration  $t$ . Furthermore, from the update rule of  $x_{t}$  we have  $\| \mathbf{x}_t - \mathbf{x}_{t - 1}\| = \gamma_t\| \mathbf{s}_{t - 1} - \mathbf{x}_{t - 1}\| \leq D\gamma$  for any  $t\geq 0$ , therefore,
$$
\begin{array}{l} \left\| \epsilon_ {t, i} \right\| \leq \frac {1}{S} \left(\left\| \nabla g _ {i} \left(\mathbf {x} _ {t}\right) - \nabla g _ {i} \left(\mathbf {x} _ {t - 1}\right) \right\| + \left\| \nabla g \left(\mathbf {x} _ {t}\right) - \nabla g \left(\mathbf {x} _ {t - 1}\right) \right\|\right) \\ \leq \frac {2 L _ {g}}{S} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| \leq \frac {2 L _ {g} D \gamma}{S}, \\ \end{array}
$$
for all  $t \in \{t_0 + 1, \dots, t_0 + q\}$  and  $i \in \{1, \dots, S\}$ . On the other hand, from the update of  $\widehat{\nabla g}_t$  and (32) we have that for any  $t \neq t_0q$ ,  $\widehat{\nabla g}_t - \nabla g(\mathbf{x}_t) = \widehat{\nabla g}_{t-1} - \nabla g(\mathbf{x}_{t-1}) + \sum_{i=1}^{S} \epsilon_{t,i}$ . Therefore, by continuing the recursive relation and taking the norm from both sides of the equality we obtain
$$
\begin{array}{l} \left\| \widehat {\nabla} g _ {t} - \nabla g \left(\mathbf {x} _ {t}\right) \right\| = \left\| \widehat {\nabla} g \left(\mathbf {x} _ {t _ {0}}\right) - \nabla g \left(\mathbf {x} _ {t _ {0}}\right) + \sum_ {j = t _ {0} + 1} ^ {t} \sum_ {i = 1} ^ {S} \epsilon_ {j, i} \right\| \tag {34} \\ = \| \sum_ {j = t _ {0} + 1} ^ {t} \sum_ {i = 1} ^ {S} \epsilon_ {j, i} \|, \\ \end{array}
$$
where the last equality follows from  $\widehat{\nabla g}(\mathbf{x}_{t_0}) = \nabla g(\mathbf{x}_{t_0})$ . Then by Proposition E.1, we have
$$
\mathbb {P} \left(\| \widehat {\nabla g} _ {t} - \nabla g (\mathbf {x} _ {t}) \| \geq \lambda\right) \leq 4 \exp \left(- \frac {\lambda^ {2}}{4 S \left(t - t _ {0}\right) \frac {4 L _ {g} ^ {2} D ^ {2} \gamma^ {2}}{S ^ {2}}}\right) \leq 4 \exp \left(- \frac {\lambda^ {2}}{1 6 L _ {g} ^ {2} D ^ {2} \gamma^ {2}}\right), \tag {35}
$$
where the last inequality follows from the fact  $S = \sqrt{n}$  and  $t - t_0 \leq q = \sqrt{n}$ . By setting  $\lambda = (4L_gD\gamma \sqrt{\log(4 / \delta')})$  for some  $\delta' \in (0,1)$ , we have with probability at least  $1 - \delta'$ ,
$$
\left\| \widehat {\nabla} g _ {t} - \nabla g \left(\mathbf {x} _ {t}\right) \right\| \leq 4 L _ {g} D \gamma \sqrt {\log \left(4 / \delta^ {\prime}\right)}. \tag {36}
$$
Similarly, with probability at least  $1 - \delta'$ ,