SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By Lemma 4.2, we have  $\| \nabla g(\mathbf{x}_t) - \widehat{\nabla g}_t\| \leq 4L_gD\gamma \sqrt{\log(12 / \delta')}$  and  $\| g(\mathbf{x}_t) - \hat{g}_t\| \leq 4L_lD\gamma \sqrt{\log(12 / \delta')}$  with probability  $1 - \delta'$ . Let  $C_7 = 8D(DL_g + L_l)$  and  $\delta = T\delta'$ . Plug them in inequality above and let  $\gamma_t = 1 / \sqrt{T}$  to obtain,
$$
\begin{array}{l} g \left(\mathbf {x} _ {t + 1}\right) - g \left(\mathbf {x} _ {0}\right) \leq \left(1 - \frac {1}{T ^ {1 / 2}}\right) \left(g \left(\mathbf {x} _ {t}\right) - g \left(\mathbf {x} _ {0}\right)\right) + \frac {C _ {7} \sqrt {\log \left(1 2 / \delta^ {\prime}\right)}}{T} + \frac {L _ {g} D ^ {2}}{2 T} \tag {74} \\ \leq \left(1 - \frac {1}{T ^ {1 / 2}}\right) \left(g (\mathbf {x} _ {t}) - g (\mathbf {x} _ {0})\right) + \frac {C _ {7} \sqrt {\log (1 2 / \delta^ {\prime})} + L _ {g} D ^ {2} / 2}{T} \\ \end{array}
$$
with probability  $1 - \delta / T$ . Sum up the inequalities for all  $t \geq 1$  to get,
$$
\begin{array}{l} g \left(\mathbf {x} _ {t + 1}\right) - g \left(\mathbf {x} _ {0}\right) = \left(1 - \frac {1}{T ^ {1 / 2}}\right) ^ {t} \mathbb {E} \left[ g \left(\mathbf {x} _ {0}\right) - g \left(\mathbf {x} _ {0}\right) \right] + \frac {\left(C _ {7} \sqrt {\log \left(1 2 / \delta^ {\prime}\right)} + L _ {g} D ^ {2} / 2\right)}{T} \sum_ {k = 1} ^ {t} \left(1 - \frac {1}{T ^ {1 / 2}}\right) ^ {k} \\ \leq \frac {C _ {7} \sqrt {\log (1 2 T / \delta)} + L _ {g} D ^ {2} / 2}{T ^ {1 / 2}} \tag {75} \\ \end{array}
$$
$$
\gamma_ {t + 1} \mathcal {G} (\mathbf {x} _ {t}) \leq f (\mathbf {x} _ {t}) - f (\mathbf {x} _ {t + 1}) + \gamma_ {t + 1} D \| \nabla f (\mathbf {x} _ {t}) - \widehat {\nabla f} _ {t} \| + \frac {L _ {f} \gamma_ {t + 1} ^ {2} D ^ {2}}{2} \tag {76}
$$
By Lemma 4.2, we have  $\| \nabla f(\mathbf{x}_t) - \widehat{\nabla f}_t\| \leq 4L_fD\gamma \sqrt{\log(12 / \delta')}$  with probability  $1 - \delta'$ . Plug it and  $\gamma_{t + 1} = 1 / \sqrt{T}$  in inequality above to obtain,
$$
\begin{array}{l} \frac {1}{\sqrt {T}} \sum_ {t = 0} ^ {T - 1} \mathcal {G} (\mathbf {x} _ {t}) \leq f (\mathbf {x} _ {0}) - f (\mathbf {x} _ {T}) + D \sum_ {t = 0} ^ {T - 1} \gamma_ {t + 1} \| \nabla f (\mathbf {x} _ {t}) - \widehat {\nabla f} _ {t} \| + \frac {L _ {f} D ^ {2}}{2} \sum_ {t = 0} ^ {T - 1} \gamma_ {t + 1} ^ {2} \tag {77} \\ \leq f (\mathbf {x} _ {0}) - f (\mathbf {x} _ {T}) + L _ {f} D ^ {2} (4 \sqrt {\log (1 2 \delta^ {\prime})} + 1 / 2) \\ \end{array}
$$
Divide both sides by  $\sqrt{T}$ , we can get, Let  $\mathbf{x}_{t^*} \triangleq \arg \min_{1 \leq t \leq T} \mathcal{G}(\mathbf{x}_t)$ , then
$$
\mathcal {G} \left(\mathbf {x} _ {t ^ {*}}\right) \leq \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {G} \left(\mathbf {x} _ {t}\right) \leq \frac {f \left(\mathbf {x} _ {0}\right) - \underline {{f}} + L _ {f} D ^ {2} \left(4 \sqrt {\log (1 2 T / \delta)} + 1 / 2\right)}{T ^ {1 / 2}} \tag {78}
$$
with probability  $1 - \delta$ . By letting  $C_8 = 5L_fD^2$ , the theorem is obtained.
E Azuma-Hoeffding-type inequalities
In this section, we present two useful vector versions of Azuma-Hoeffding-type concentration inequalities with uniform bound assumption or sub-gaussian assumption. They are crucial in our high probability analysis.
Proposition E.1. (Pinelis and other 1994 [44], Theorem 3.5) Let  $\zeta_1, \ldots, \zeta_t \in \mathbb{R}^d$  be a vector-valued martingale difference sequence w.r.t. a filtration  $\{\mathcal{F}_t\}$ , i.e. for each  $\tau \in 1, \ldots, t$ , we have  $\mathbb{E}[\zeta_{\tau}|\mathcal{F}_{\tau -1}] = 0$ . Suppose that  $\| \zeta_{\tau}\| \leq c_{\tau}$  almost surely. Then  $\forall t \geq 1$ ,
$$
P \left(\| \sum_ {\tau = 1} ^ {T} \zeta_ {\tau} \| \geq \lambda\right) \leq 4 \exp \left(- \frac {\lambda^ {2}}{4 \sum_ {\tau = 1} ^ {T} c _ {\tau} ^ {2}}\right) \tag {79}
$$
Proposition E.2. (Jin et al. [45], Corollary 7) Let  $\zeta_1, \ldots, \zeta_t \in \mathbb{R}^d$  be a vector-valued martingale difference sequence w.r.t. a filtration  $\{\mathcal{F}_t\}$ , i.e. for each  $\tau \in 1, \ldots, t$ , we have  $\mathbb{E}[\zeta_{\tau} | \mathcal{F}_{\tau - 1}] = 0$ . Suppose that  $\mathbb{E}[\exp (\| \zeta_{\tau}\|^{2} / c_{\tau}^{2})] \leq \exp(1)$ . Then there exists a absolute constant  $c$  such that, for any  $\delta > 0$ , with probability at least  $1 - \delta$ ,
$$
\left\| \sum_ {\tau = 1} ^ {T} \zeta_ {\tau} \right\| \leq c \cdot \sqrt {\sum_ {\tau = 1} ^ {T} c _ {\tau} ^ {2} \log \frac {2 d}{\delta}} \tag {80}
$$
This proposition was also used in previous literature including [46] and [37]. It is common to use such martingale inequality to obtain some high-probability results recently.
In this section, we include more details about the numerical experiments in Section 5. For completeness, we briefly introduce the update rules of aR-IP-SeG in [16] and DBGD in [13]. In the following, we use the notation  $\Pi_{\mathcal{Z}}(\cdot)$  to denote the Euclidean projection onto the set  $\mathcal{Z}$ .
The aR-IP-SeG algorithm is given by,
$$
\mathbf {y} _ {t + 1} = \Pi_ {\Xi} (\mathbf {x} _ {t} - \gamma_ {t} (\nabla \tilde {f} (\mathbf {x} _ {t}, \boldsymbol {\theta} _ {t})) + \rho_ {t} \nabla \tilde {g} (\mathbf {x} _ {t}, \boldsymbol {\xi} _ {t}))
$$
$$
\mathbf {x} _ {t + 1} = \Pi_ {\tilde {Z}} \left(\mathbf {x} _ {t} - \gamma_ {t} (\nabla \tilde {f} (\mathbf {y} _ {t}, \theta_ {t} ^ {\prime})) + \rho_ {t} \nabla \tilde {g} (\mathbf {y} _ {t}, \xi_ {t} ^ {\prime})\right)
$$
$$
\Gamma_ {t + 1} = \Gamma_ {t} + \left(\gamma_ {t} \rho_ {t}\right) ^ {r} \tag {81}
$$
$$
\bar {\mathbf {y}} _ {t + 1} = \frac {\Gamma_ {t} \bar {\mathbf {y}} _ {t} + (\gamma_ {t} \rho_ {t}) ^ {r} \mathbf {y} _ {t + 1}}{\Gamma_ {t + 1}}
$$
where  $\gamma_{t}$  is the stepsize,  $\rho_{t}$  is the regularization parameter, and  $\bar{\mathbf{y}}_T$  is the output of the algorithm. In this experiment, we choose  $\gamma_{t} = \gamma_{0} / (t + 1)^{3 / 4}$  and  $\rho_{t} = \rho_{0}(t + 1)^{1 / 4}$  for some constants  $\gamma_0$  and  $\rho_0$ . The DBGD-sto is a stochastic version of DBGD, which simply replaces the gradients in DBGD with stochastic gradients. Although the stochastic version of DBGD does not have a theoretical guarantee, it has been used to solve stochastic simple bilevel optimization problems in [13], which worked pretty well empirically. Hence, we use it as a baseline for solving stochastic simple bilevel problems and compare it with our proposed algorithms. The DBGD algorithm is given by
$$
\mathbf {x} _ {k + 1} = \mathbf {x} _ {k} - \gamma_ {k} (\nabla f (\mathbf {x} _ {k}) + \lambda_ {k} \nabla g (\mathbf {x} _ {k}))
$$
where  $\gamma_{k}$  is the stepsize and we set  $\lambda_{k}$  as
$$
\lambda_ {k} = \max  \left\{\frac {\phi (\mathbf {x} _ {k}) - \langle \nabla f (\mathbf {x} _ {k}) , \nabla g (\mathbf {x} _ {k}) \rangle}{\| \nabla g (\mathbf {x} _ {k}) \| ^ {2}}, 0 \right\} \quad \mathrm {a n d} \quad \phi (\mathbf {x}) = \min  \left\{\alpha (g (\mathbf {x}) - \hat {g}), \beta \| \nabla g (\mathbf {x}) \| ^ {2} \right\}
$$
where  $\alpha$  and  $\beta$  are hyperparameters and  $\hat{g}$  is a lower bound of  $g^{*}$ . In this experiment, we choose  $\hat{g} = 0$ . We also note that [13] only considered unconstrained simple bilevel optimization, i.e.  $\mathcal{Z} = \mathbb{R}^d$ . We further project  $\mathbf{x}_t$  onto  $\mathcal{Z}$  for each iteration to ensure the constraints are satisfied.
F.1 Over-parameterized regression
Dataset generation. The original Wikipedia Math Essential dataset [30] composes of a data matrix of size  $1068 \times 731$ . We randomly select one of the columns as the outcome vector  $\mathbf{b} \in \mathbb{R}^{1068}$  and the rest to be a new matrix  $\mathbf{A} \in \mathbb{R}^{1068 \times 731}$ . We set constraint parameter  $\lambda = 10$  in this experiment.
Initialization. We run the algorithm, SPIDER-FW [40], with stepsize chosen as  $\gamma_t = 0.1 / (t + 1)$  on the lower-level problem in (1). We terminate the process to get  $\mathbf{x}_0$  as the initial point for both SBCGI 1 and SBCGF 2 after  $10^5$  stochastic oracle queries.
Implementation details. We query stochastic oracle  $9 \times 10^{5}$  times with stepsize  $\gamma_{t} = 0.01 / (t + 1)$  and  $\gamma = 10^{-5}$  for SBCGI 1 and SBCGF 2 with  $K_{t} = 10^{-4} / \sqrt{t + 1}$ , respectively. In each iteration, we need to solve the following subproblem induced by the methods,
$$
\min  _ {\mathbf {s}} \left\langle \nabla f \left(\boldsymbol {\beta} _ {k}\right), \mathbf {s} \right\rangle \quad \text {s . t .} \quad \| \mathbf {s} \| _ {1} \leq \lambda , \left\langle \nabla g \left(\boldsymbol {\beta} _ {k}\right), \mathbf {s} - \boldsymbol {\beta} _ {k} \right\rangle \leq g \left(\boldsymbol {\beta} _ {0}\right) - g \left(\boldsymbol {\beta} _ {k}\right). \tag {82}
$$
Introduce  $\mathbf{s}^{+},\mathbf{s}^{-}\geq 0$  such that  $\mathbf{s} = \mathbf{s}^{+} - \mathbf{s}^{-}$  . Then we can reformulate the problem above as follows,
$$
\min  _ {\mathbf {s} ^ {+}, \mathbf {s} ^ {-}} \left\langle \nabla f \left(\boldsymbol {\beta} _ {k}\right), \mathbf {s} ^ {+} - \mathbf {s} ^ {-} \right\rangle \tag {83}
$$
$$
\mathrm {s . t .} \mathbf {s} ^ {+}, \mathbf {s} ^ {-} \geq 0, \left\langle \mathbf {s} ^ {+}, \mathbf {1} \right\rangle + \left\langle \mathbf {s} ^ {-}, \mathbf {1} \right\rangle \leq \lambda , \left\langle \nabla g \left(\boldsymbol {\beta} _ {k}\right), \mathbf {s} ^ {+} - \mathbf {s} ^ {-} - \boldsymbol {\beta} _ {k} \right\rangle \leq g \left(\boldsymbol {\beta} _ {0}\right) - g \left(\boldsymbol {\beta} _ {k}\right),
$$
where  $1\in \mathbb{R}^d$  is the all-one vector.
For aR-IP-SeG, we choose  $\gamma_0 = 10^{-7}$  and  $\rho_0 = 10^3$ . For DBGD, we set  $\alpha = \beta = 1$  and  $\gamma_t = 10^{-6}$ .
F.2 Dictionary learning
Dataset generation. We generate 500 sparse coefficient vectors  $\{\mathbf{x}_i\}_{i=1}^{250}$  and  $\{\mathbf{x}_k'\}_{k=1}^{250}$  with 5 random nonzero entries, whose absolute values are drawn uniformly from  $[0.2, 1]$ . The entries of the random noise vectors  $\{\mathbf{n}_i\}_{i=1}^{250}$  and  $\{\mathbf{n}_k'\}_{k=1}^{250}$  are drawn from i.i.d. Gaussian distribution with mean 0 and standard deviation 0.01.
Initialization. We use a similar initialization procedure as [12], which consists of two phases. In the first phase, we run the standard Frank-Wolfe algorithm on both the variables  $\mathbf{D} \in \mathbb{R}^{25 \times 40}$  and  $\mathbf{X} \in \mathbb{R}^{40 \times 250}$  for  $10^{4}$  iterations with the stepsize  $\gamma_{t} = 1 / \sqrt{t + 1}$ . Next, in the second phase, we fix the variable  $\mathbf{X}$  and only update  $\mathbf{D}$  using the Frank-Wolfe algorithm with exact line search for additional  $10^{4}$  iterations to obtain  $\hat{\mathbf{D}}$  and  $\hat{\mathbf{X}}$  as the initial point for the full bilevel problem.
Implementation Details. We choose  $\delta = 3$  in both problems (5). To be fair, all four algorithms start from the same initial point. We slightly modify the initial point by letting  $\tilde{\mathbf{D}}\in \mathbb{R}^{25\times 50}$  be the concatenation of  $\hat{\mathbf{D}}\in \mathbb{R}^{25\times 40}$  and 10 columns of all zeros vectors. Furthermore, we initialize another variable  $\tilde{\mathbf{X}}$  randomly by choosing its entries from a standard Gaussian distribution and then normalizing each column to have a  $\ell_1$ -norm of  $\delta$ . We choose the stepsize as  $\gamma_t = 0.1 / (t + 1)^{2 / 3}$  and  $\gamma = 10^{-3}$  for our SBCGI 1 and SBCGF2 with  $K_{t} = 0.01 / (t + 1)^{1 / 3}$ , respectively. Empirically, we observe that taking one sample per iteration leads to a very unstable process in this problem. In this case, we choose a mini-batch of size 8 for SBCGI, aR-IP-SeG, and the stochastic version of DBGD. For each iteration, we will solve the following subproblem,
$$
\min  _ {\tilde {\mathbf {D}}} \left\langle \nabla f _ {\tilde {\mathbf {D}}} \left(\tilde {\mathbf {D}} _ {k}, \tilde {\mathbf {X}} _ {k}\right), \tilde {\mathbf {D}} \right\rangle \quad \text {s . t .} \quad \left\| \tilde {\mathbf {d}} _ {i} \right\| _ {2} \leq 1, \left\langle \nabla g \left(\tilde {\mathbf {D}} _ {k}\right), \tilde {\mathbf {D}} - \tilde {\mathbf {D}} _ {k} \right\rangle \leq g \left(\tilde {\mathbf {D}} _ {0}\right) - g \left(\tilde {\mathbf {D}} _ {k}\right) \tag {84}
$$
The above problem can be reformulated by using the KKT condition, which is equivalent to get a root of the following one-dimensional nonlinear equation involving  $\lambda \geq 0$ :
$$
\tilde {\mathbf {D}} = \Pi_ {\mathcal {Z}} \left(\nabla f _ {\tilde {\mathbf {D}}} \left(\tilde {\mathbf {D}} _ {k}, \tilde {\mathbf {X}} _ {k}\right) + \lambda \nabla g \left(\tilde {\mathbf {D}} _ {k}\right)\right), \quad \left\langle \nabla g \left(\tilde {\mathbf {D}} _ {k}\right), \tilde {\mathbf {D}} - \tilde {\mathbf {D}} _ {k} \right\rangle = g \left(\tilde {\mathbf {D}} _ {0}\right) - g \left(\tilde {\mathbf {D}} _ {k}\right) \tag {85}
$$
Figure 3: Comparison of SBCGI, SBCGF, aR-IP-SeG, and DBGD-Sto for the over-parameterized regression problem
(a) Lower-level gap
Figure 4: Comparison of SBCGI, SBCGF, aR-IP-SeG, and DBGD-Sto for solving the dictionary learning problem.
where the projection onto  $\mathcal{Z} = \left\{\tilde{\mathbf{D}}\in \mathbb{R}^{25\times 50}:\left\| \tilde{\mathbf{d}}_i\right\| _2\leq 1,i = 1,\ldots ,50\right\}$  is equivalent to project each column on the Euclidean ball. In practice, the reformulated problem can be solved efficiently by MATLAB's root-finding solver.
For aR-IP-SeG, we choose  $\gamma_0 = 10^{-4}$  and  $\rho_0 = 1$ . For the stochastic version of DBGD, we set  $\alpha = \beta = 100$  and  $\gamma_t = 5 \times 10^{-3}$ .
Additional plots illustrating the comparison of the studied methods in terms of runtime rather than the number of sample used are provided in Figure 3 and Figure 4.
F.3 Experiments with different random seeds
We further repeat the experiment 10 times with different random seeds to see more realizations of the stochastic algorithms. The results are reported in Figure 5 and Figure 6. The solid lines denote the average statistics over 10 trials of the algorithms. While the shaded regions surrounding each line reflect the span of all the random instances involved, Figure 5 and Figure 6 present similar results as Figure 1 and Figure 2, which eliminates the possibility of choosing a particularly good instance.
(a) Lower-level gap
Figure 5: Comparison of SBCGI, SBCGF, aR-IP-SeG, and DBGD-Sto for solving Problem (3) with 10 different random seeds
(b) Upper-level gap
Figure 6: Comparison of SBCGI, SBCGF, aR-IP-SeG, and DBGD-Sto for solving Problem (5) with 10 different random seeds
(c) Recovery rate
Figure 7: Comparison of SBCGI, SBCGF, SBCGI-M, SBCGF-M, STORM-FW, and SPIDER-FW for solving Problem (5).
F.4 Importance of the right cutting plane
In this section, we numerically illustrate the importance of choosing the right cutting plane on Example 2 (dictionary learning). Specifically, we compare our proposed methods with the ones without a cutting plane and with an unregularized cutting plane (without additional term  $K_{t}$ ).
If we replace the stochastic cutting plane (9) with the unregularized cutting plane (8) in SBCGI 1 and SBCGF 2, then the algorithm usually fails at some point in the process, depending on the datasets and parameters chosen, based on our experimental observations. More specifically, algorithms' failure means that the subproblem of dictionary learning (85) is infeasible. So we slightly modify it by adding a checkpoint before solving the subproblem. If the subproblem is infeasible at the current iteration, then we choose the update direction  $\mathbf{s}_t = \widehat{\nabla g_t}$ . This adjustment prevents unnecessary interruptions during the process and enforce the algorithms to focus only on the lower-level problem when the subproblem is infeasible. We denote the modified algorithms SBCGI-M and SBCGF-M. Moreover, we also take SBCGI and SBCGF without cutting planes into consideration, denoted as STORM-FW and SPIDER-FW. In fact, in this case, the bilevel algorithms degenerate to single-level projection-free algorithms similar to algorithms in [37] and [40].
Figure 7 (a) indicates that SBCGI-M and SBCGF-M focus more on the lower-level problem due to the design of the algorithms and extremely unstable as we can see in Figure 7 (b)(c). While STORM-FW and SPIDER-FW only focus on the upper-level problem, which leads to terrible results on the lower-level gap and recovery rate.