SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Estimating  $\nabla F(x)$  via DL-SGD. For any given  $K$  and  $N$ , we construct the DL-SGD estimator for the gradient of  $F$  by using the following procedure: (i) generate a sample  $\xi$  from  $\mathbb{P}_{\xi}$ , (ii) generate i.i.d. samples  $\eta'$  and  $\eta''$  from the conditional distribution  $\mathbb{P}_{\eta|\xi}$ , (iii) run EpochsGD as described in Algorithm 1 with an arbitrary initial iterate  $y_1^0$  to obtain  $y_{K+1}^0$ , and (iv) construct  $\widehat{\Lambda}(x, y_{K+1}^0; \xi)$  as in (5). Using these ingredients, we can now construct the DL-SGD gradient estimator as
$$
\widehat {v} ^ {K} (x) := \nabla_ {1} f (x, y _ {K + 1} ^ {0}; \eta^ {\prime \prime}, \xi) - \nabla_ {1 2} ^ {2} g (x, y _ {K + 1} ^ {0}; \eta^ {\prime}, \xi) \widehat {\Lambda} (x, y _ {K + 1} ^ {0}; \xi) \nabla_ {2} f (x, y _ {K + 1} ^ {0}; \eta^ {\prime \prime}, \xi). \tag {6}
$$
Algorithm 2 SGD Framework
Input:  $\sharp$  of iterations  $T$ , stepsizes  $\{\alpha_t\}_{t=1}^T$ , initial iterate  $x_1$ .
1: for  $t = 1$  to  $T$  do
2: Construct a gradient estimator  $v(x_{t})$  and update  $x_{t + 1} = x_{t} - \alpha_{t}v(x_{t})$ .
3: end for
Output:  $\widehat{x}_T$  uniformly sampled from  $\{x_1,\dots,x_T\}$ .
In Lemma 2 below, we will analyze the bias and variance as well as the sampling and computational costs of the DL-SGD gradient estimator. We will see that a small bias  $\| \mathbb{E}[\widehat{v}^K](x) - \nabla F(x)\| \leq \epsilon$  can be ensured by setting  $K = \mathcal{O}(\log (\epsilon^{-1}))$ , in which case Epochs SGD computes  $\mathcal{O}(\epsilon^{-2})$  stochastic gradients of  $g$ . From now on, we refer to Algorithm 2 with  $v(x) = \widehat{v}^{K}(x)$  as the DL-SGD algorithm.
2.1 RT-MLMC Gradient Estimator
The bottleneck of evaluating the DL-SGD gradient estimators is the computation of  $y_{K + 1}^{0}$ . The computational costs can be reduced, however, by exploiting the telescoping sum property
$$
\begin{array}{l} \widehat {v} ^ {K} (x) = \widehat {v} ^ {1} (x) + \sum_ {k = 1} ^ {K} [ \widehat {v} ^ {k + 1} (x) - \widehat {v} ^ {k} (x) ] \\ = \widehat {v} ^ {1} (x) + \sum_ {k = 1} ^ {K} p _ {k} \frac {\widehat {v} ^ {k + 1} (x) - \widehat {v} ^ {k} (x)}{p _ {k}} = \widehat {v} ^ {1} (x) + \mathbb {E} _ {\widehat {k} \sim \mathbb {P} _ {\widehat {k}}} \Big [ \frac {\widehat {v} ^ {\widehat {k} + 1} (x) - \widehat {v} ^ {\widehat {k}} (x)}{p _ {\widehat {k}}} \Big ], \\ \end{array}
$$
where  $\widehat{v^k}$  is defined as in (6) with  $k = 1,\dots ,K$  replacing  $K$ , and where  $\mathbb{P}_{\widehat{k}}$  is a truncated geometric distribution with  $\mathbb{P}_{\widehat{k}}(\widehat{k} = k) = p_k\propto 2^{-k}$  for every  $k = 1,\ldots ,K$ . This observation prompts us to construct the RT-MLMC gradient estimator as
$$
\widehat {v} (x) = \widehat {v} ^ {1} (x) + p _ {\widehat {k}} ^ {- 1} \left(\widehat {v} ^ {\widehat {k} + 1} (x) - \widehat {v} ^ {\widehat {k}} (x)\right). \tag {7}
$$
The RT-MLMC gradient estimator has three key properties:
- It is an unbiased estimator for the DL-SGD gradient estimator, i.e.,  $\mathbb{E}_{\widehat{k} \sim \mathbb{P}_{\widehat{k}}}[\widehat{v}(x)] = \widehat{v}^K(x)$ .
- Evaluating  $\widehat{v}(x)$  requires computing  $y_{k+1}^{0}(x, \xi)$  with probability  $p_k$ , which decays exponentially with  $k$ . To ensure a small bias, we need to set  $K = \mathcal{O}(\log(\epsilon^{-1}))$ , and thus  $p_K = \mathcal{O}(\epsilon)$ . Hence, most of the time, EpochsGD only needs to run over  $k \ll K$  epochs. As a result, the average sampling and computational costs are markedly smaller for RT-MLMC than for DL-SGD.
- Since  $\widehat{v}^{k + 1}(x)$  and  $\widehat{v}^k (x)$  differ only in  $y_{k + 1}^0$  and  $y_{k}^{0}$ , both of which are generated by EpochsSGD and are thus highly correlated,  $\widehat{v}^{k + 1}(x) - \widehat{v}^k (x)$  has a small variance thanks to a control variate effect [Nelson, 1990]. Hence, the variance of RT-MLMC is well-controlled, as shown in Lemma 2.
In Lemma 2 below, we will analyze the bias and variance as well as the sampling and computational costs of the RT-MLMC gradient estimator. We will see that it requires only  $\mathcal{O}(1)$  samples to ensure that the bias drops to  $\mathcal{O}(\epsilon)$ . This is in stark contrast to the DL-SGD estimator, which needs  $\mathcal{O}(\epsilon^{-2})$  samples. The lower sample complexity and the corresponding lower computational cost come at the expense of an increased variance of the order  $\mathcal{O}(\log (\epsilon^{-1}))$ . The construction of the RT-MLMC gradient estimator is detailed in Algorithm 3. From now on, we refer to Algorithm 2 with  $v(x) = \widehat{v} (x)$  as the RT-MLMC algorithm.
2.2 Memory and Arithmetic Operational Costs
The per-iteration memory and arithmetic operational cost of DL-SGD as well as RT-MLMC is dominated by the cost of computing the matrix-vector product
$$
\widehat {c} (x, y; \xi) := \nabla_ {1 2} ^ {2} g (x, y; \eta^ {\prime}, \xi) \widehat {\Lambda} (x, y; \xi) \nabla_ {2} f (x, y; \eta^ {\prime \prime}, \xi). \tag {8}
$$
By (5),  $\widehat{\Lambda} (x,y;\xi)$  is a product of  $\widehat{N}$  matrices of the form  $I - 1 / (2L_{g,1})\nabla_{22}g(x,y;\eta_n,\xi)$ , and the  $n$ -th matrix coincides with the gradient of  $(y - 1 / (2L_{g,1})\nabla_2g(x,y;\eta_n,\xi))$  with respect to  $y$ . We can thus compute (8) recursively as follows. We first set  $v = \nabla_2f(x,y;\eta'',\xi)$ . Next, we update  $v$  by
Algorithm 3 RT-MLMC Gradient Estimator for Conditional Bilevel Optimization
Input: Iterate  $x$ , largest epoch number  $K$ , initialization  $y_1^0$ , approximation number  $N$
1: Sample  $\xi$  from  $\mathbb{P}_{\xi}$  and sample  $\widehat{N}$  uniformly from  $\{0,\dots ,N - 1\}$ .
2: Sample  $\bar{k}$  from the truncated geometric distribution  $\mathbb{P}_{\bar{k}}$
3: Run Epochs SGD  $(\widehat{k}, x, \xi, y_1^0)$  and obtain  $y_{\widehat{k} + 1}^0$ ,  $y_{\widehat{k}}^0$ , and  $y_1^0$ .
4: Construct  $\widehat{v}^{\widehat{k} +1}(x),\widehat{v}^{\widehat{k}}(x)$  , and  $\widehat{v}^1 (x)$  according to (6) and compute
$$
\widehat {v} (x) = \widehat {v} ^ {1} (x) + p _ {\widehat {k}} ^ {- 1} (\widehat {v} ^ {\widehat {k} + 1} (x) - \widehat {v} ^ {\widehat {k}} (x)).
$$
Output:  $\widehat{v} (x)$
setting it to the gradient of  $(y - 1 / (2L_{g,1})\nabla_2g(x,y;\eta_{\widehat{N}},\xi))^{\top}v$  with respect to  $y$ , which is computed via automatic differentiation. This yields  $v = (I - 1 / (2L_{g,1})\nabla_{22}g(x,y;\eta_{\widehat{N}},\xi))\nabla_2f(x,y;\eta^{\prime \prime},\xi)$ . By using a backward recursion with respect to  $n$ , we can continue to multiply  $v$  from the left with the other matrices in the expansion of  $\widehat{\Lambda} (x,y;\xi)$ . This procedure is highly efficient because the memory and arithmetic operational cost of computing the product of a Hessian matrix with a constant vector via automatic differentiation is bounded—up to a universal constant—by the cost of computing the gradient of the same function [Rajeswaran et al., 2019]. See Algorithm 4 in Appendix C for details. The expected arithmetic operational costs of Algorithm 4 is  $\mathcal{O}(Nd)$  and the memory cost is  $\mathcal{O}(d)$ .
3 Complexity Bounds
In this section we derive the sample and gradient complexities of the proposed algorithms. We first analyze the error of the general SGD framework detailed in Algorithm 2.
Lemma 1 (Error Analysis of Algorithm 2). If Algorithm 2 is used to minimize an  $L_{\psi}$ -Lipschitz continuous and  $S_{\psi}$ -Lipschitz smooth function  $\psi(x)$  and if  $\alpha \leq 1/(2S_{\psi})$ , then we have
$$
\mathbb {E} \| \nabla \psi (\widehat {x} _ {T}) \| ^ {2} \leq \frac {2 A _ {1}}{\alpha T} + \frac {2}{T} \sum_ {t = 1} ^ {T} \left[ L _ {\psi} \| \mathbb {E} [ v (x _ {t}) - \nabla \psi (x _ {t}) ] \| + S _ {\psi} \alpha \mathbb {E} \| v (x _ {t}) - \nabla \psi (x _ {t}) \| ^ {2} \right],
$$
where  $A_{1}\coloneqq \psi (x_{1}) - \min_{x}\psi (x)$
Lemma 1 sightly generalizes [Rakhlin et al., 2012, Ghadimi et al., 2016, Bottou et al., 2018]. We defer the proof to Appendix A. Thus, to prove convergence to a stationary point, we need to characterize the bias, variance, and computational costs of the DL-SGD and the RT-MLMC gradient estimators.
Lemma 2 (Bias, Variance, Sampling Cost and Computational Cost). We have the following results.
- The biases of the DL-SGD and RT-MLMC estimators match and satisfy
$$
\| \mathbb {E} \widehat {v} ^ {K} (x) - \nabla F (x) \| = \| \mathbb {E} \widehat {v} (x) - \nabla F (x) \| \leq \mu_ {g} ^ {- 1} \left(1 - \mu_ {g} / \left(2 L _ {g, 1}\right)\right) ^ {N} + \mathcal {O} \left(N ^ {2} 2 ^ {- K / 2}\right),
$$
and the corresponding variances satisfy  $\operatorname{Var}(\widehat{v}^K(x)) = \mathcal{O}(N^2)$  and  $\operatorname{Var}(\widehat{v}(x)) = \mathcal{O}(KN^4)$ .
- The numbers of samples and iterations needed by EpochsGD to build a DL-SGD estimator are bounded by  $N + 2^{K + 1} - 1$  and  $2^{K + 1} - 1$ , respectively. The expected numbers of samples and iterations needed for an RT-MLMC estimator are bounded by  $N + 3K$  and  $3K$ , respectively.
Lemma 2 implies that setting  $N = \mathcal{O}(\log (\epsilon^{-1}))$  and  $K = \mathcal{O}(\log (\epsilon^{-1}))$  reduces the bias to  $\mathcal{O}(\epsilon)$ . In this case the RT-MLMC estimators have higher variances than the DL-SGD estimators, but their variances are still of the order  $\mathcal{O}(\log (\epsilon^{-1}))$ . On the other hand, using RT-MLMC estimators reduces the per-iteration sampling and computational costs from  $\mathcal{O}(2^K) = \mathcal{O}(\epsilon^{-2})$  to  $\mathcal{O}(K) = \mathcal{O}(\log (\epsilon^{-1}))$ .
Note that Hu et al. [2021] characterize the properties of general MLMC estimators and derive their complexity bounds. However, the proposed RT-MLMC estimators for CSBO problems are the first of their kind. In addition, as we need to estimate the Hessian inverse  $\Lambda (x,y^{*}(x;\xi);\xi)$ , our analysis is more involved. In contrast to Asi et al. [2021], who use MLMC techniques for estimating projections and proximal points, we use MLMC techniques for estimating gradients in bilevel optimization. The following main theorem summarizes our complexity bounds.
Figure 1: Test error of DL-SGD (left figure), RT-MLMC (right figure), and MAML against upper-level iterations on meta-learning.  $K$  represents how accurately we solve the lower-level problem.
Theorem 1 (Complexity Bounds). If Assumption 2.1 holds, then Algorithm 2 based on the RT-MLMC or the DL-SGD estimator outputs an  $\epsilon$ -stationary point of  $F$  provided that  $N = \mathcal{O}(\log (\epsilon^{-1}))$ ,  $K = \mathcal{O}(\log (\epsilon^{-1}))$ ,  $\alpha = \mathcal{O}(\epsilon^2)$  and  $T = \mathcal{O}(\epsilon^{-4})$ . When using the RT-MLMC estimator, the sample complexities of  $\xi$  and  $\eta$  as well as the gradient complexities of  $g$  and  $f$  are  $\widetilde{\mathcal{O}} (\epsilon^{-4})$ . When using the DL-SGD estimator, the sample complexity of  $\eta$  and the gradient complexity of  $g$  are  $\widetilde{\mathcal{O}} (\epsilon^{-6})$ , while and the sample complexity of  $\xi$  and the gradient complexity of  $f$  are  $\widetilde{\mathcal{O}} (\epsilon^{-4})$ .
Remark. Theorem 1 asserts that the sample complexity of  $\eta$  and the gradient complexity of  $g$  are much smaller for RT-MLMC than for DL-SGD, while the gradient complexities of  $f$  are comparable. When specialized to SBO or CSO problems, the complexity bounds of RT-MLMC match those of the state-of-the-art algorithms ALEST for SBO problems [Chen et al., 2021] and MLMC-based methods for CSO problems [Hu et al., 2021]. When restricted to classical stochastic nonconvex optimization, the complexity bounds of RT-MLMC match the existing lower bounds [Arjevani et al., 2023]. These observations further highlight the effectiveness of RT-MLMC across various settings.
4 Applications and Numerical Experiments
4.1 Meta-Learning
Optimization-based meta-learning [Finn et al., 2017, Rajeswaran et al., 2019] aims to find a common shared regularization parameter for multiple similar yet different machine learning tasks in order to avoid overfitting when training each task separately on their datasets that each only processes a few data points. Recall Equation (4), the objective function of the optimization-based meta-learning [Rajeswaran et al., 2019],
$$
\min _ {x} \mathbb {E} _ {i \sim \mu} \mathbb {E} _ {D _ {i} ^ {t e s t} \sim \rho_ {i}} \left[ l _ {i} (y _ {i} ^ {*} (x), D _ {i} ^ {t e s t}) \right]
$$
$$
\text {w h e r e} y _ {i} ^ {*} (x) = \operatorname {a r g m i n} _ {y _ {i}} \mathbb {E} _ {D _ {i} ^ {\text {t r a i n}} \sim \rho_ {i}} \left[ l _ {i} \left(y _ {i}, D _ {i} ^ {\text {t r a i n}}\right) + \frac {\lambda}{2} \| y _ {i} - x \| ^ {2} \right], \forall i \in [ M ] \text {a n d} x. \tag {9}
$$
where  $\mu$  is the distribution over all  $M$  tasks,  $\rho_{i}$  is the distribution of data from the task  $i$ ,  $D_{i}^{train}$  and  $D_{i}^{test}$  are the training and testing dataset of the task  $i$ ,  $x$  is the shared parameter of all tasks and  $y_{i}^{*}(x)$  is the best parameter for a regularized objective of task  $i$ ,  $l_{i}$  is a loss function that measures the average loss on the dataset of the  $i$ -th task, and  $\lambda$  is the regularization parameter to ensure the optimal solution obtained from the lower-level problem is not too far from the shared parameter obtained from the upper-level problem. Note that such a problem also occurs in personalized federated learning with each lower level being one user.
Note that the task distribution  $\mu$  is usually replaced by averaging over all  $M$  tasks. In such cases, existing works [Guo and Yang, 2021, Rajeswaran et al., 2019] only demonstrate a convergence rate that scales linearly with the number of tasks  $M$ . In contrast, the sample complexity of our proposed method does not depend on the number of tasks  $M$ , enabling substantially faster computation for a larger  $M$ . The seminal work, Model-agnostic Meta-learning (MAML) [Finn et al., 2017], is an approximation of Problem (9) via replacing  $y_{i}^{*}(x)$  with one-step gradient update, i.e.,  $\widehat{y}_i(x)\coloneqq x - \lambda^{-1}\nabla l_i(x,D_i^{train})$ .
We study the case where the loss function  $l_{i}(x,D), \forall i$  is a multi-class logistic loss using a linear classifier. The experiment is examined on tinyImageNet [Mnmoustafa, 2017] by pre-processing
Figure 2: Left: Performance of our proposed RT-MLMC algorithm and  $\mathrm{BSVRB}^{\mathrm{v2}}$  against upper-level iterations on meta-learning. Right: Performance of algorithms against the total computational time on meta-learning.
it using the pre-trained ResNet-18 network [He et al., 2016] to extract linear features. Since the network has learned a rich set of hierarchical features from the ImageNet dataset [Deng et al., 2009], it typically extracts useful features for other image datasets. Note that each task consists of labels of similar characteristics.
Figure 1 presents the average of logistic loss evaluated on the test dataset against the number of iterations, with each iteration representing one upper-level update. From the plot, we see that both DL-SGD and RT-MLMC methods tend to have better generalization performance when using a larger number of levels  $K$ . As shown in Table 2, RT-MLMC is about 9 times faster to compute the upper-level gradient estimator than DL-SGD when  $K$  is large.
Table 2: The computation time of DL-SGD/RT-MLMC gradient estimators on meta-learning.
K	DL-SGD	RT-MLMC
Mean	Variance	Mean	Variance
6	2.65e-02	6.34e-03	2.73e-02	1.46e-02
8	7.23e-02	7.77e-03	3.41e-02	1.85e-02
10	2.48e-01	2.75e-02	4.93e-02	4.06e-02
12	9.38e-01	3.71e-02	1.08e-01	5.44e-02
In contrast, the MAML baseline does not have superior performance since the one-step gradient update does not solve the lower-level problem to approximate global optimality. In Appendix D.1, we provide numerical results for a modified MAML algorithm with multiple gradient updates, which achieves better performance compared to MAML but is still worse than our proposed method.
After the initial submission of the paper, a con-current work Hu et al. [2023] proposed two types of algorithms (BSVRB $^{v1}$  and BSVRB $^{v2}$ ) that apply to the meta-learning formulation (9). Their proposed algorithm BSVRB $^{v1}$  is computationally expansive as it requires the exact computation of the inverse of Hessian matrix (which is of size  $5120 \times 5120$  in this example) with respect to  $\theta$  in each iteration of the upper-level update. In the following, we compare the performance of our algorithm with their proposed Hessian-free algorithm BSVRB $^{v2}$  in Figure 2.
In the left plot of Figure 2, we examine the performance of RT-MLMC method and  $\mathrm{BSVRB}^{\mathrm{v2}}$  by running the same number of total epochs. It shows that RT-MLMC method has much better performance in terms of test error. In the right plot of Figure 2, we examine the performance of RT-MLMC method and  $\mathrm{BSVRB}^{\mathrm{v2}}$  by running the same amount of computational time. Although the per-upper-level-iteration computational costs of  $\mathrm{BSVRB}^{\mathrm{v2}}$  is small, it takes a much longer time for  $\mathrm{BSVRB}^{\mathrm{v2}}$  to achieve a similar test error as RT-MLMC.
4.2 Wasserstein DRO with Side Information
The WDRO-SI [Yang et al., 2022] studies robust stochastic optimization with side information [Bertsimas and Kallus, 2020]. Let  $\xi$  denote the side information and  $\eta$  denote the randomness dependent on  $\xi$ . The WDRO-SI seeks to find a parameterized mapping  $f(x;\xi)$  from the side information  $\xi$  to a decision  $w$  that minimizes the expected loss w.r.t.  $\eta$  under the worst-case distributional shifts over  $(\xi,\eta)$ . Rigorously, with a penalty on the distributional robust constraint, WDRO-SI admits the form
$$
\min  _ {x} \max  _ {\mathbb {P}} \left\{\mathbb {E} _ {(\xi , \eta) \sim \mathbb {P}} [ l (f (x, \xi); \eta) ] - \lambda \mathrm {C} (\mathbb {P}, \mathbb {P} ^ {0}) \right\}, \tag {10}
$$
Figure 3: Test error on WDRO-SI against the number of upper-level updates for DL-SGD (left) and RT-MLMC (middle). Figure (Right) compares WDRO-SI with ERM and Wasserstein DRO that do not incorporate side information.  $m$  means the number of samples of  $Z$  generated from  $\mathbb{P}_{Z|X}$  for each realization of  $X$ .
where  $l(w; \eta)$  is the loss function dependent on the decision  $w$  and the random variable  $\eta$ ,  $\mathbb{P}^0$  is the nominal distribution of  $(\xi, \eta)$  that usually takes the form of an empirical distribution, and  $C(\cdot, \cdot)$  is a casual transport distance between distributions [Yang et al., 2022, Definition 1] – a variant of the Wasserstein distance that better captures the information from  $\xi$ . For distributionally robust feature-based newsvendor problems [Zhang et al., 2023], the covariate  $\xi$  can be temporal, spatial, or weather information,  $\eta$  is the random demand,  $f(x; \xi)$  denotes the ordering quantity for a given  $\xi$ , and  $l(f(x; \xi); \eta)$  characterizes the loss if the ordering quantity  $f(x; \xi)$  does not match the demand  $\eta$ .
Incorporating the cost function of the casual transport distance used in [Yang et al., 2022] and utilizing the dual form, the WDRO-SI problem in (10) can be reformulated as a special case of CSBO:
$$
\min  _ {x} \mathbb {E} _ {\xi \sim \mathbb {P} _ {\xi} ^ {0}} \mathbb {E} _ {\eta \sim \mathbb {P} _ {\eta | \xi} ^ {0}} \left[ l \left(f (x; y ^ {*} (x; \xi)), \eta\right) - \lambda \| y ^ {*} (x; \xi) - \xi \| ^ {2} \right] \quad (\text {u p p e r})
$$
$$
\text {w h e r e} y ^ {*} (x; \xi) := \operatorname {a r g m i n} _ {\widetilde {\xi}} \mathbb {E} _ {\eta \sim \mathbb {P} _ {\eta | \xi} ^ {0}} \left[ - l \left(f (x; \widetilde {\xi}), \eta\right) + \lambda \| \xi - \widetilde {\xi} \| ^ {2} \right], \forall \widetilde {\xi} \text {a n d} x. \quad (\text {l o w e r l e v e l}) \tag {11}
$$
The original work [Yang et al., 2022] only allows affine function  $f$  or non-parametric approximation, while our approach allows using neural network approximation such that  $f(x; \xi)$  is a neural network parameterized by  $x$ . Using Theorem 1, we obtain the first sample and gradient complexities for WDRO-SI. For the distributionally robust feature-based newsvendor problems, we compare the performance of DL-SGD and RT-MLMC. We compare with ERM and WDRO, which do not incorporate side information.
Fig. 3 (left) and (middle) present the results of test loss versus the number of upper-level iterations for DL-SGD and RT-MLMC, respectively. From the plot, using a larger number of epochs  $K$  for the lower-level problem generally admits lower testing loss values, i.e., better generalization performance.
Fig. 3 (right) highlights the importance of incorporating side information as the performance of WDRO-SI outperforms the other two baselines. In addition, more observations of  $\eta$  for a given side information  $\xi$  can enhance the performance. Table 3 reports the computational time for DL-SGD and RT-MLMC gradient estimators, and RT-MLMC is significantly faster since it properly balances the bias-variance-computation trade-off for the gradient simulation.
Table 3: Computation time of DL-SGD/RT-MLMC gradient estimators for WDRO-SI.
K	DL-SGD	RT-MLMC
Mean	Variance	Mean	Variance
2	1.27e-02	2.67e-03	5.04e-03	7.26e-04
4	5.25e-02	2.58e-03	1.25e-02	8.26e-03
6	1.68e-01	2.74e-03	2.02e-02	9.39e-03
8	4.63e-01	2.08e-03	3.41e-02	1.68e-02
We introduced the class of contextual stochastic bilevel optimization problems, which involve a contextual stochastic optimization problem at the lower level. In addition, we designed efficient gradient-based solution schemes and analyzed their sample and gradient complexities. Numerical results on two complementary applications showcase the expressiveness of the proposed problem class as well as the efficiency of the proposed algorithms. Future research should address generalized CSBO problems with constraints at the lower level, which will require alternative gradient estimators.
This research was supported by the Swiss National Science Foundation under the NCCR Automation, grant agreement 51NF40_180545, an NSF CAREER CCF-1650913, NSF DMS-2134037, CMMI-2015787, CMMI-2112533, DMS-1938106, DMS-1830210, and the Coca-Cola Foundation.