SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\widehat{\nabla g}_t$  and  $\hat{g}_t$  are gradient and function value estimators, respectively, that we formally define later. In the above expression, the addition of the term  $K_{t}$ , which is a sequence of constants converging to zero as  $t\to \infty$ , allows us to ensure that with high probability the random set  $\hat{\mathcal{X}}_t$  contains all optimal solutions of the lower-level problem. Choosing suitable values for the sequence  $K_{t}$  is a crucial task. If we select a large value for  $K_{t}$  then the probability of  $\hat{\mathcal{X}}_t$  containing  $\mathcal{X}_g^*$  goes up at the price allowing points with larger values  $g$  in the set. As a result, once we perform an update similar to the one in (6), the lower-level function value could increase significantly. On the other hand, selecting small values for  $K_{t}$  would allow us to show that the lower level objective function is not growing, while the probability of  $\hat{\mathcal{X}}_t$  containing  $\mathcal{X}_g^*$  becomes smaller which could even lead to a case that the set becomes empty and the bilevel problem becomes infeasible.
Remark 3.1. How to compute  $g(\mathbf{x}_0)$ ? In the finite sum setting, we can accurately compute  $g(\mathbf{x}_0)$ , and the additional cost of  $n$  function evaluations will be dominated by the overall complexity. In the stochastic setting, we could use a large batch of samples to compute  $g(\mathbf{x}_0)$  with high precision at the beginning of the process. This additional operation will not affect the overall sample complexity of the proposed method, as the additional cost is negligible compared to the overall sample complexity. Specifically, we need to take a batch size of  $b = \tilde{\mathcal{O}} (\epsilon^{-2})$  to estimate  $\hat{g} (\mathbf{x}_0)$ . Using the Hoeffding inequality for subgaussian random variables, we have the following bound:  $|\hat{g} (\mathbf{x}_0) - g(\mathbf{x}_0)|\leq \sqrt{2}\sigma_l(T + 1)^{-\omega /2}\sqrt{\log(2 / \delta)}$ , with a probability of at least  $1 - \delta$ , where  $T$  is the maximum number of iterations. Comparing this with Lemma 4.1.3, we can further derive:  $|\hat{g} (\mathbf{x}_0) - g(\mathbf{x}_0)|\leq \sqrt{2}\sigma_l(T + 1)^{-\omega /2}\sqrt{\log(2 / \delta)}\leq \sqrt{2}(2L_lD + \frac{3^\omega}{3^\omega - 1}\sigma_l)(t + 1)^{-\omega /2}\sqrt{\log(6 / \delta)}$ , with a probability of at least  $1 - \delta$  for all  $0\le t\le T$ . Consequently, the introduced error term would be absorbed in  $K_{0,t}$  and will not affect any parts of the analysis.
Variance reduced estimators. As mentioned above, a key point in the design of our stochastic bilevel algorithms is to select  $K_{t}$  properly such that  $\hat{\mathcal{X}}_t$  contains  $\mathcal{X}_g^*$  with high probability, for all  $t \geq 0$ . To
Algorithm 1 SBCGI
1: Input: Target accuracy:  $\epsilon_f, \epsilon_g > 0$ , probability  $\delta > 0$ , step size:  $\alpha_t, \beta_t, \rho_t, \gamma_t > 0$   
2: Initialization: Initialize  $\mathbf{x}_0 \in \mathcal{Z}$  such that  $g(\mathbf{x}_0) - g^* \leq \epsilon_g / 2$   
3: for  $t = 0, \dots, T$  do  
4: if  $t = 0$  then  
5:  $\widehat{\nabla f}_t = \nabla \tilde{f}(\mathbf{x}_t, \theta_t), \widehat{\nabla g}_t = \nabla \tilde{g}(\mathbf{x}_t, \xi_t), \hat{g}_t = \tilde{g}(\mathbf{x}_t, \xi_t)$   
6: else  
7: Update the estimate of  $\nabla f, \widehat{\nabla f}_t = (1 - \alpha_t) \widehat{\nabla f}_{t-1} + \nabla \tilde{f}(\mathbf{x}_t, \theta_t) - (1 - \alpha_t) \nabla \tilde{f}(\mathbf{x}_{t-1}, \theta_t)$   
8: Update the estimate of  $\nabla g, \widehat{\nabla g}_t = (1 - \beta_t) \widehat{\nabla g}_{t-1} + \nabla \tilde{g}(\mathbf{x}_t, \xi_t) - (1 - \beta_t) \nabla \tilde{g}(\mathbf{x}_{t-1}, \xi_t)$   
9: Update the estimate of  $g, \hat{g}_t = (1 - \rho_t) \hat{g}_{t-1} + \tilde{g}(\mathbf{x}_t, \xi_t) - (1 - \rho_t) \tilde{g}(\mathbf{x}_{t-1}, \xi_t)$   
10: end if  
11: Compute  $\mathbf{s}_t \in \arg \min_{\mathbf{s} \in \mathcal{X}_t} \{\widehat{\nabla f}_t^\top \mathbf{s}\}$  where  $\mathcal{X}_t = \{\mathbf{s} \in \mathcal{Z}: \langle \widehat{\nabla g}_t, \mathbf{s} - \mathbf{x}_t \rangle \leq g(\mathbf{x}_0) - \hat{g}_t + K_t\}$   
12: Update the variable  $\mathbf{x}_{t+1} = (1 - \gamma_{t+1})\mathbf{x}_t + \gamma_{t+1}\mathbf{s}_t$   
13: end for
achieve such a guarantee, we first need to characterize the error of our gradient and function value estimators. More precisely, suppose that for our function estimator we have that  $P(|\hat{g}_t - g(\mathbf{x}_t)| \leq K_{0,t}) \geq 1 - \delta'$  and for the gradient estimator we have  $P(\| \widehat{\nabla} g_t - \nabla g(\mathbf{x}_t) \| \leq K_{1,t}) \geq 1 - \delta'$ , for some  $\delta' \in (0,1)$ . Then, by setting  $K_t = K_{0,t} + DK_{1,t}$ , we can guarantee that the conditions required for the inequalities in (7) hold with probability at least  $(1 - 2\delta')$ .
Using simple sample average estimators would not allow for the selection of a diminishing  $K_{t}$ , as the variance is not vanishing, but by using proper variance-reduced estimators the variance of the estimators vanishes over time and eventually, we can send  $K_{t}$  to zero. In this section, we focus on two different variance reduction estimators. For the stochastic setting in (1) we use the STochastic Recursive Momentum estimator (STORM), proposed in [35], and for the finite-sum setting, we utilize the Stochastic Path-Integrated Differential Estimator (SPIDER) proposed in [36]. If  $\mathbf{v}_{t - 1}$  is the gradient estimator of STORM at time  $t - 1$ , the next estimator is computed as
$$
\mathbf {v} _ {t} = \left(1 - \alpha_ {t}\right) \mathbf {v} _ {t - 1} + \nabla \tilde {f} (\mathbf {x} _ {t}, \theta_ {t}) - \left(1 - \alpha_ {t}\right) \nabla \tilde {f} (\mathbf {x} _ {t - 1}, \theta_ {t}), \tag {10}
$$
where  $\nabla \tilde{f}(\mathbf{x},\theta)$  is the stochastic gradient evaluated at  $\mathbf{x}$  with sample  $\theta$ . The main advantage of the above estimator is that it can be implemented even with one sample per iteration. Unlike STORM, for the SPIDER estimator, we need a larger batch of samples per update. More precisely, if we consider  $\mathbf{v}_{t-1}$  as the estimator of SPIDER for  $\nabla f(\mathbf{x}_t)$ , it is updated according to
$$
\mathbf {v} _ {t} = \nabla f _ {\mathcal {S}} (\mathbf {x} _ {t}) - \nabla f _ {\mathcal {S}} (\mathbf {x} _ {t - 1}) + \mathbf {v} _ {t - 1}, \tag {11}
$$
where  $\nabla f_{S}(\mathbf{x}) = (1 / S)\sum_{i\in S}\nabla f(\mathbf{x},\theta_{i})$  is the average sub-sampled stochastic gradient computed using samples that are in the set  $S$ . As we will discuss later, in the finite sum case that we use SPIDER, the size of batch  $S$  depends on  $n$  which is the number of component functions. We delay establishing a high probability error bound for these estimators to section 4.1.
3.1 Conditional gradient algorithms with random sets: stochastic and finite-sum
Next, we present our Stochastic Bilevel Conditional Gradient method for Infinite sample case abbreviated by SBCGI for solving (1) and its finite sum variant denoted by SBCGF. In both cases, we first find a point  $\mathbf{x}_0$  that satisfies  $g(\mathbf{x}_0) - g^* \leq \epsilon_g / 2$ , for some accuracy  $\epsilon_g$ . The cost of finding such a point is negligible compared to the cost of the main algorithm as we discuss later. At each iteration  $t$ , we first update the gradient estimator of the upper-level and the function and gradient estimators of the lower-level problem. In SBCGI, we follow the STORM idea as described in steps 4-6 of Algorithm 1, while in SBCGF, we use the SPIDER technique as presented in steps 7-10 of Algorithm 2. In the case of SBCGF, we need to compute the exact gradient and function values once every  $q$  iteration as presented in steps 4-6 of Algorithm 2. Once the estimators are updated, we can define the random set  $\hat{\mathcal{X}}_t$  as in (9) and solve the following subproblem over the set  $\hat{\mathcal{X}}_t$ ,
$$
\mathbf {s} _ {t} = \underset {\mathbf {s} \in \hat {\mathcal {X}} _ {t}} {\arg \min } \langle \widehat {\nabla f} _ {t}, \mathbf {s} \rangle , \tag {12}
$$
Algorithm 2 SBCGF
1: Input: Target accuracy:  $\epsilon_{f},\epsilon_{g} > 0$  , probability accuracy:  $\delta_0,\delta_1 > 0$  , step size:  $\gamma_t > 0$
2: Initialization: Initialize  $\mathbf{x}_0\in \mathcal{Z}$  such that  $g(\mathbf{x}_0) - g^*\leq \epsilon_g / 2$
3: for  $t = 0, \dots, T$  do
4: if  $\mathrm{mod}(t,q) = 0$  then
5: Set  $\widehat{\nabla f}_t = \nabla f(\mathbf{x}_t), \widehat{\nabla g}_t = \nabla g(\mathbf{x}_t), \hat{g}_t = g(\mathbf{x}_t)$
6: else
7: Draw  $S$  samples
8: Update the estimate of  $\nabla f$  as  $\widehat{\nabla f}_t = \widehat{\nabla f}_{t - 1} + \nabla f_S(\mathbf{x}_t) - \nabla f_S(\mathbf{x}_{t - 1})$
9: Update the estimate of  $\nabla g$  as  $\widehat{\nabla g}_t = \widehat{\nabla g}_{t - 1} + \nabla g_{\mathcal{S}}(\mathbf{x}_t) - \nabla g_{\mathcal{S}}(\mathbf{x}_{t - 1})$
10: Update the estimate of  $g$  as  $\hat{g}_t = \hat{g}_{t - 1} + g_S(\mathbf{x}_t) - g_S(\mathbf{x}_{t - 1})$
1: end if
2: Compute  $\mathbf{s}_t\in \arg \min_{\mathbf{s}\in \mathcal{X}_t}\{\widehat{\nabla f}_t^\top \mathbf{s}\}$  where  $\mathcal{X}_t = \{\mathbf{s}\in \mathcal{Z}:\langle \widehat{\nabla g}_t,\mathbf{s} - \mathbf{x}_t\rangle \leq g(\mathbf{x}_0) - \hat{g}_t + K_t\}$
3: Update the variable  $\mathbf{x}_{t + 1} = (1 - \gamma_{t + 1})\mathbf{x}_t + \gamma_{t + 1}\mathbf{s}_t$
4: end for
where  $\widehat{\nabla f}_t$  is the unbiased estimator of  $\nabla f(\mathbf{x}_t)$ . Note that we implicitly assume that we have access to a linear optimization oracle that returns a solution of the subproblem in (12), which is standard for projection-free methods [32, 33]. In particular, if  $\mathcal{Z}$  can be described by a system of linear inequalities, then problem (12) corresponds to a linear program and can be solved by a standard solver as we show in our experiments. Once,  $\mathbf{s}_t$  is calculated we simply update the iterate
$$
\mathbf {x} _ {t + 1} = \left(1 - \gamma_ {t + 1}\right) \mathbf {x} _ {t} + \gamma_ {t + 1} \mathbf {s} _ {t} \tag {13}
$$
with stepsize  $\gamma_{t + 1}\in [0,1]$ . The only missing part for the implementation of our methods is the choice of  $K_{t}$  in the random set and the stepsize parameters. We address these points in the next section.
Remark 3.2. SBCGI can be implemented with a batch size as small as  $S = 1$ . However, this does not imply that the batch size "has to be"  $S = 1$ . In other words, the main advantage of SBCGI, compared to SBCGF, is its capability to be implemented with any mini-batch size, even as small as  $S = 1$ . Therefore, for SBCGI, the batch size can be set arbitrarily, whereas for SBCGF, it must be  $\sqrt{n}$ .
Remark 3.3. In the finite-sum setting, if the numbers of functions in the upper- and lower-level losses are different, we could simply modify SBCGF 2 by choosing  $S_{u} = q_{u} = \sqrt{n_{u}}$  and  $S_{l} = q_{l} = \sqrt{n_{l}}$ , where  $n_u$  and  $n_l$  are the number of functions in the upper- and lower-level, respectively.
In this section, we characterize the sample complexity of our methods for stochastic and finite-sum settings. Before stating our results, we first characterize a high probability bound for the estimators of our algorithms, which are crucial in the selection of parameter  $K_{t}$  and the overall sample complexity.
4.1 High probability bound for the error terms
To achieve a high probability bound, it is common to assume that the noise of gradient or function is uniformly bounded as in [36, 37], but such assumptions may not be realistic for most machine learning applications. Hence, in our analysis, we consider a milder assumption and assume the noise of function and gradient are sub-Gaussian as in Assumptions 2.2 and 2.3, respectively. Given these assumptions, we next establish a high probability error bound for the estimators in SBCGI.
Lemma 4.1. Consider SBCGI in Algorithm 1 with parameters  $\alpha_{t} = \beta_{t} = \rho_{t} = \gamma_{t} = 1 / (t + 1)^{\omega}$  where  $\omega \in (0,1]$ . If Assumptions 2.1, 2.2, and 2.3 are satisfied, for any  $t\geq 1$  and  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ , for some absolute constant  $c$ , ( $d$  is the number of dimension), we have
$$
\left\| \widehat {\nabla f} _ {t} - \nabla f (\mathbf {x} _ {t}) \right\| \leq c \sqrt {2} \left(2 L _ {f} D + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \sigma_ {f}\right) (t + 1) ^ {- \omega / 2} \sqrt {\log (6 d / \delta)}, \tag {14}
$$
$$
\left\| \widehat {\nabla} g _ {t} - \nabla g \left(\mathbf {x} _ {t}\right) \right\| \leq c \sqrt {2} \left(2 L _ {g} D + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \sigma_ {g}\right) (t + 1) ^ {- \omega / 2} \sqrt {\log (6 d / \delta)}, \tag {15}
$$
$$
\left| \hat {g} _ {t} - g \left(\mathbf {x} _ {t}\right) \right| \leq c \sqrt {2} \left(2 L _ {l} D + \frac {3 ^ {\omega}}{3 ^ {\omega} - 1} \sigma_ {l}\right) (t + 1) ^ {- \omega / 2} \sqrt {\log (6 / \delta)}. \tag {16}
$$
Lemma 4.1 shows that for any  $\omega \in (0,1]$ , if we set  $\alpha_{t} = \beta_{t} = \rho_{t} = \gamma_{t} = 1 / (t + 1)^{\omega}$ , then with high probability the gradient and function approximation errors converge to zero at a sublinear rate of  $\tilde{\mathcal{O}}(1/t^{\omega/2})$ . Moreover, the above result characterizes the choice of  $K_{t}$ . Specifically, if we define  $K_{1,t}$  as the upper bound in (15) and  $K_{0,t}$  as the upper bound in (16), by setting  $K_{t} = K_{0,t} + DK_{1,t}$ , then with probability  $(1 - \delta)$  the random set  $\hat{\mathcal{X}}_t$  contains  $\mathcal{X}_g^*$ . Later, we show that  $\omega = 1$  leads to the best complexity bound for the convex setting and  $\omega = 2/3$  is the best choice for the nonconvex setting.
Next, we establish a similar result for the estimators in SBCGF.
Lemma 4.2. Consider SBCGF with stepsize  $\gamma$  and  $S = q = \sqrt{n}$ . If Assumptions 2.1-2.3 hold, for any  $t \geq 1$  and  $\delta \in (0,1)$ , with probability  $1 - \delta$  we have  $\| \widehat{\nabla f}_t - \nabla f(\mathbf{x}_t) \| \leq 4L_g D\gamma \sqrt{\log(12 / \delta)}$ ,  $\| \widehat{\nabla g}_t - \nabla g(\mathbf{x}_t) \| \leq 4L_g D\gamma \sqrt{\log(12 / \delta)}$ , and  $|\hat{g}_t - g(\mathbf{x}_t)| \leq 4L_l D\gamma \sqrt{\log(12 / \delta)}$ .
Similarly, for SBCGF, we set  $K_{1,t} = 4L_gD\gamma \sqrt{\log(12 / \delta)}$  and  $K_{0,t} = 4L_lD\gamma \sqrt{\log(12 / \delta)}$  and choose  $K_{t} = K_{0,t} + DK_{1,t}$ , then the random set  $\hat{\mathcal{X}}_t$  contains  $\mathcal{X}_g^*$  with probability  $1 - \delta$ .
Next, we formalize our claim about the random set with the above choice of  $K_{t}$ .
Lemma 4.3. If  $\mathcal{X}_g^*$  is the solution set of the lower-level problem and  $\hat{\mathcal{X}}_t$  is the feasible set constructed by cutting plane at iteration  $t$ , then for any  $t \geq 0$  and  $\delta \in (0,1)$ , we have  $\mathbf{P}(\mathcal{X}_g^* \subseteq \hat{\mathcal{X}}_t) \geq 1 - \delta$ .
This lemma shows all  $\mathcal{X}_g^*$  is a subset of the constructed feasible set  $\hat{\mathcal{X}}_t$  with a high probability of  $1 - \delta$ . Indeed, using a union bound one can show that the above statement holds for all iterations up to time  $t$  with probability  $1 - t\delta$ .
4.2 Convergence and complexity results for the stochastic setting
Next, we characterize the iteration and sample complexity of the proposed method in SBCGI for the stochastic setting. First, we present the result for the case that  $f$  is convex.
Theorem 4.4 (Stochastic setting with convex upper-level). Consider SBCGI in Algorithm 1 for solving problem (1). Suppose Assumptions2.1, 2.2, and 2.3 hold and  $f$  is convex. If the step sizes of SBCGI are selected as  $\alpha_{t} = \beta_{t} = \rho_{t} = \gamma_{t} = (t + 1)^{-1}$ , and the cutting plane parameter is  $K_{t} = c((2L_{l}D + \frac{3}{2}\sigma_{l})\sqrt{2\log(6t / \delta)} + D(2L_{g}D + \frac{3}{2}\sigma_{g})\sqrt{2\log(6td / \delta))}(t + 1)^{-1 / 2}$ , then after  $T$  iterations,
$$
g (\mathbf {x} _ {T}) - g ^ {*} \leq \frac {C _ {1} \zeta}{\sqrt {T}} + \frac {L _ {g} D ^ {2} \log T}{T} + \frac {\epsilon_ {g}}{2}, \quad f (\mathbf {x} _ {T}) - f ^ {*} \leq \frac {C _ {2} \zeta}{\sqrt {T}} + \frac {f (\mathbf {x} _ {0}) - f ^ {*} + L _ {f} D ^ {2} \log T}{T}.
$$
with probability  $1 - \delta$  for some absolute constants  $C_1$  and  $C_2$  and  $\zeta \coloneqq \sqrt{\log(6Td / \delta)}$ .
Theorem 4.4 shows a convergence rate of  $\mathcal{O}(\sqrt{\log(Td / \delta)} /\sqrt{T})$ . As a corollary, SBCGI returns an  $(\epsilon_{f},\epsilon_{g})$ -optimal solution with probability  $1 - \delta$  after  $\mathcal{O}(\log (d / \delta \epsilon) / \epsilon^2)$  iterations, where  $\epsilon = \min \{\epsilon_f,\epsilon_g\}$ . Since we use one sample per iteration, the overall sample complexity is also  $\mathcal{O}(\log (d / \delta \epsilon) / \epsilon^2)$ . Note that the iteration complexity and sample complexity of our method outperform the ones in [16], as they require  $\mathcal{O}(1 / \epsilon^4)$  iterations and sample to achieve the same guarantee.
Remark 4.1. The task of finding  $\mathbf{x}_0$  which is equivalent to a single-level stochastic optimization problem requires  $\mathcal{O}(1 / \epsilon_g^2)$  iterations and samples. As a result, this additional cost does not affect the overall complexity of our method. The same argument also holds in the non-convex case.
Theorem 4.5 (Stochastic setting with non-convex upper level). Consider SBCGI for solving problem (1). Suppose Assumptions 2.1-2.3 hold,  $f$  is nonconvex, and define  $\underline{f} = \min_{\mathbf{x} \in \mathcal{Z}} f(\mathbf{x})$ . If the stepsizes of SBCGI are selected as  $\alpha_t = \beta_t = \rho_t = (t + 1)^{-2/3}$ ,  $\gamma_t = (T + 1)^{-2/3}$ , and the cutting plane parameter is  $K_t = c((2L_lD + \frac{3^{2/3}}{3^{2/3} - 1}\sigma_l)\sqrt{2\log(6T/\delta)} + D(2L_gD + \frac{3^{2/3}}{3^{2/3} - 1}\sigma_g)\sqrt{2\log(6Td/\delta)})(t + 1)^{-1/3}$ , then after  $T$  iterations, there exists an iterate  $\mathbf{x}_{t^*}$  in the set  $\in \{\mathbf{x}_0, \mathbf{x}_1, \dots, \mathbf{x}_{T-1}\}$  for which
$$
g \left(\mathbf {x} _ {t ^ {*}}\right) - g ^ {*} \leq \frac {C _ {3} \zeta + L _ {g} D ^ {2}}{(T + 1) ^ {1 / 3}} + \frac {\epsilon_ {g}}{2}, \quad \mathcal {G} \left(\mathbf {x} _ {t ^ {*}}\right) \leq \frac {f \left(\mathbf {x} _ {0}\right) - \underline {{f}} + C _ {4} \zeta + L _ {f} D ^ {2}}{(T + 1) ^ {1 / 3}}
$$
with probability  $1 - \delta$  for some absolute constants  $C_3$  and  $C_4$  and  $\zeta \coloneqq \sqrt{\log(6Td / \delta)}$ .
As a corollary of Theorem 4.5, the number of iterations required to find an  $(\epsilon_f,\epsilon_g)$ -optimal solution can be upper bounded by  $\mathcal{O}(\log (d / \delta \epsilon)^{3 / 2} / \epsilon^3)$ , where  $\epsilon = \min \{\epsilon_f,\epsilon_g\}$ . We note that the dependence on the upper-level accuracy  $\epsilon_{f}$  also matches that in the standard CG method for a single-level nonconvex problem [33, 38]. Moreover, as we only need one stochastic oracle query per iteration, SBCGI only requires  $\mathcal{O}(\log (d / \delta \epsilon)^{3 / 2} / \epsilon^3)$  stochastic oracle queries to find an  $(\epsilon_f,\epsilon_g)$ -optimal.
4.3 Convergence and complexity results for the finite-sum setting
Similarly, we present iteration and sample complexity for algorithm 2 under the finite-sum setting.
Theorem 4.6 (Finite-sum setting with convex upper-level). Consider SBCGF presented in Algorithm 2 for solving the finite-sum version of (1). Suppose Assumptions 2.1, 2.2, and 2.3 hold and  $f$  is convex. If we set the step sizes of SBCGF as  $\gamma = \log T / T$ ,  $S = q = \sqrt{n}$ , and the cutting plane parameter as  $K_{t} = 4D(L_{l}\sqrt{\log(12T / \delta)} + L_{g}D\sqrt{\log(12T / \delta)})\log T / T$ , then after  $T$  iterations,
$$
g (\mathbf {x} _ {T}) - g ^ {*} \leq \frac {(C _ {5} \zeta^ {\prime} + L _ {g} D ^ {2}) \log T}{T} + \frac {\epsilon_ {g}}{2}, f (\mathbf {x} _ {T}) - f ^ {*} \leq \frac {f (\mathbf {x} _ {0}) - f ^ {*} + C _ {6} \zeta^ {\prime} \log T}{T},
$$
with probability at least  $1 - \delta$ , for some absolute constant  $C_5$  and  $C_6$ , and  $\zeta' = \sqrt{\log(12T / \delta)}$ .
Theorem 4.6 implies the number of stochastic oracle queries is  $\mathcal{O}(\log (1 / \delta \epsilon)^{3 / 2}\sqrt{n} /\epsilon)$ , where  $\epsilon = \min \{\epsilon_f,\epsilon_g\}$ , which matches the optimal sample complexity of single-level problems [39].
Theorem 4.7 (Finite-sum setting with non-convex upper-level). Consider SBCGF presented in Algorithm 2 for solving the finite-sum version of (1). Suppose Assumptions2.1-2.3 hold, and  $f$  is non-convex. Define  $\underline{f} = \min_{\mathbf{x} \in \mathcal{Z}} f(\mathbf{x})$ . If the parameters of SBCGF are selected as  $\gamma = 1 / \sqrt{T}$ ,  $S = q = \sqrt{n}$ , and the cutting plane parameter is  $K_t = 4D(L_l \sqrt{\log(12T / \delta)} + L_g D \sqrt{\log(12T / \delta)}) / \sqrt{T}$ , then after  $T$  iterations, there exists an iterate  $\mathbf{x}_{t^*} \in \{\mathbf{x}_0, \mathbf{x}_1, \ldots, \mathbf{x}_{T - 1}\}$  for which,
$$
g (\mathbf {x} _ {t ^ {*}}) - g ^ {*} \leq \frac {C _ {7} \zeta^ {\prime} + L _ {g} D ^ {2}}{T ^ {1 / 2}} + \frac {\epsilon_ {g}}{2}, \quad \mathcal {G} (\mathbf {x} _ {t ^ {*}}) \leq \frac {f (\mathbf {x} _ {0}) - \underline {{f}} + C _ {8} \zeta^ {\prime}}{T ^ {1 / 2}}
$$
with probability at least  $1 - \delta$ , for some absolute constants  $C_7$  and  $C_8$ , and  $\zeta' = \sqrt{\log(12T / \delta)}$ .
As a corollary of Theorem 4.7, the number of stochastic oracle queries is  $\mathcal{O}(\log (1 / \delta \epsilon)\sqrt{n} /\epsilon^2)$ , where  $\epsilon = \min \{\epsilon_f,\epsilon_g\}$ , which matches the state-of-the-art single-level result  $\mathcal{O}(\sqrt{n} /\epsilon^2)$  in [40]. SBCGF also improves the number of linear minimization oracle queries of SBCGI from  $\mathcal{O}(1 / \epsilon^2)$  to  $\mathcal{O}(1 / \epsilon)$  for convex upper-level and from  $\mathcal{O}(1 / \epsilon^3)$  to  $\mathcal{O}(1 / \epsilon^2)$  for non-convex upper-level.
In this section, we test our methods on two different stochastic bilevel optimization problems with real and synthetic datasets and compare them with other existing stochastic methods in [16] and [13].
Over-parameterized regression. We consider the bilevel problem corresponding to sparse linear regression introduced in (3). We apply the Wikipedia Math Essential dataset [30] which composes of a data matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  with  $n = 1068$  samples and  $d = 730$  features and an output vector  $\mathbf{b} \in \mathbb{R}^n$ . To ensure the problem is over-parameterized, we assign  $1/3$  of the dataset as the training set  $(\mathbf{A}_{\mathrm{tr}}, \mathbf{b}_{\mathrm{tr}})$ ,  $1/3$  as the validation set  $(\mathbf{A}_{\mathrm{val}}, \mathbf{b}_{\mathrm{val}})$  and the remaining  $1/3$  as the test set  $(\mathbf{A}_{\mathrm{test}}, \mathbf{b}_{\mathrm{test}})$ . For both upper- and lower-level loss functions we use the least squared loss, and we set  $\lambda = 10$ . We compare the performance of our methods with the aR-IP-SeG method by [16] and the stochastic version of DBGD introduced by [13]. We employ CVX [41, 42] to solve the lower-level problem and the reformulation of the bilevel problem to obtain  $g^*$  and  $f^*$ , respectively. We also include the additional cost of finding  $\mathbf{x}_0$  in SBCGI and SBCGF in our comparisons.
In Figure 1(a)(b), we observe that SBCGF maintains a smaller lower-level gap than other methods and converges faster than the rest in terms of upper-level error. SBCGI has the second-best performance in terms of lower- and upper-level gaps, while aR-IP-SeG performs poorly in terms of both lower- and upper-level objectives. The performance of DBGD-sto for the upper-level objective is well, however, it underperforms in terms of lower-level error. In Figure 1(c), SBCGF, SBCGI, and DBGD-sto achieve almost equally small test errors, while aR-IP-SeG fails to achieve a low test error. Note
(a) Lower-level gap
(b) Upper-level gap
(c) Test error