SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jincheng Cao
jinchengcao@utexas.edu
Nazanin Abolfazli
SIE Department
nazaninabolfazli@arizona.edu
Erfan Yazdandoost Hamedani
erfany@arizona.edu
In this paper, we study a class of stochastic bilevel optimization problems, also known as stochastic simple bilevel optimization, where we minimize a smooth stochastic objective function over the optimal solution set of another stochastic convex optimization problem. We introduce novel stochastic bilevel optimization methods that locally approximate the solution set of the lower-level problem via a stochastic cutting plane, and then run a conditional gradient update with variance reduction techniques to control the error induced by using stochastic gradients. For the case that the upper-level function is convex, our method requires  $\tilde{\mathcal{O}} (\max \{1 / \epsilon_f^2,1 / \epsilon_g^2\})$  stochastic oracle queries to obtain a solution that is  $\epsilon_{f}$ -optimal for the upper-level and  $\epsilon_{g}$ -optimal for the lower-level. This guarantee improves the previous best-known complexity of  $\mathcal{O}(\max \{1 / \epsilon_f^4,1 / \epsilon_g^4\})$ . Moreover, for the case that the upper-level function is non-convex, our method requires at most  $\tilde{\mathcal{O}} (\max \{1 / \epsilon_f^3,1 / \epsilon_g^3\})$  stochastic oracle queries to find an  $(\epsilon_{f},\epsilon_{g})$ -stationary point. In the finite-sum setting, we show that the number of stochastic oracle calls required by our method are  $\tilde{\mathcal{O}} (\sqrt{n} /\epsilon)$  and  $\tilde{\mathcal{O}} (\sqrt{n} /\epsilon^{2})$  for the convex and non-convex settings, respectively, where  $\epsilon = \min \{\epsilon_f,\epsilon_g\}$ .
An important class of bilevel optimization problems is simple bilevel optimization in which we aim to minimize an upper-level objective function over the solution set of a lower-level problem [1-4]. Recently this class of problems has attracted great attention in machine learning society due to their applications in continual learning [5], hyper-parameter optimization [6, 7], meta-learning [8, 9], and reinforcement learning [10, 11]. Motivated by large-scale learning problems, in this paper, we are particularly interested in the stochastic variant of the simple bilevel optimization where the upper and lower-level objective functions are the expectations of some random functions with unknown distributions and are accessible only through their samples. Hence, the computation of the objective function values or their gradients is not computationally tractable. Specifically, we focus on the stochastic simple bilevel problem defined as
$$
\min  _ {\mathbf {x} \in \mathbb {R} ^ {d}} f (\mathbf {x}) = \mathbb {E} [ \tilde {f} (\mathbf {x}, \theta) ] \quad \text {s . t .} \quad \mathbf {x} \in \underset {\mathbf {z} \in \mathcal {Z}} {\arg \min } g (\mathbf {z}) = \mathbb {E} [ \tilde {g} (\mathbf {z}, \xi) ], \tag {1}
$$
Table 1: Results on stochastic simple bilevel optimization. The abbreviations "SC", "C", and "NC" stand for "strongly convex", "convex", and "non-convex", respectively. Note that  $\epsilon = \min \{\epsilon_f, \epsilon_g\}$
References	Type	Upper level	Lower level	Convergence	Sample Complexity
Objective f	Objective g	Feasible set Z	Upper level	Lower level
aR-IP-SeG [16]	Stochastic	C, Lipschitz	C, Lipschitz	Closed	O(max{1/εf4, 1/εg4})	O(1/ε4)
Algorithm1	Stochastic	C, smooth	C, smooth	Compact	O(max{1/εf2, 1/εg2})	O(1/ε2)
Algorithm1	Stochastic	NC, smooth	C, smooth	Compact	O(max{1/εf3, 1/εg3})	O(1/ε3)
Algorithm2	Finite-sum	C, smooth	C, smooth	Compact	O(max{1/εf, 1/εg})	O(√n/ε)
Algorithm2	Finite-sum	NC, smooth	C, smooth	Compact	O(max{1/εf2, 1/εg2})	O(√n/ε2)
where  $\mathcal{Z}$  is compact and convex and  $\tilde{f},\tilde{g}:\mathbb{R}^d\to \mathbb{R}$  are continuously differentiable functions on an open set containing  $\mathcal{Z}$ , and  $\theta$  and  $\xi$  are some independent random variables drawn from some possibly unknown probability distributions. As a result, the functions  $f,g:\mathbb{R}^d\to \mathbb{R}$  are also continuously differentiable functions on  $\mathcal{Z}$ . We assume that  $g$  is convex but not necessarily strongly convex, and hence the solution set of the lower-level problem in (1) is in general not a singleton. We also study the finite sum version of the above problem where both functions can be written as the average of  $n$  component functions, i.e.,  $f(\mathbf{x}) = (1 / n)\sum_{i = 1}^{n}\tilde{f} (\mathbf{x},\theta_{i})$  and  $g(\mathbf{z}) = (1 / n)\sum_{i = 1}^{n}\tilde{g} (\mathbf{z},\xi_{i})$ .
The main challenge in solving problem (1), which is inherited from its deterministic variant, is the absence of access to the feasible set, i.e., the lower-level solution set. This issue eliminates the possibility of using any projection-based or projection-free methods. There have been some efforts to overcome this issue in the deterministic setting (where access to  $f$  and  $g$  and their gradients is possible), including [12-15], however, there is little done on the stochastic setting described above. In fact, the only work that addresses the stochastic problem in (1) is [16], where the authors present an iterative regularization-based stochastic extra gradient algorithm and show that it requires  $\mathcal{O}(1 / \epsilon_f^4)$  and  $\mathcal{O}(1 / \epsilon_g^4)$  queries to the stochastic gradient of the upper-level and lower-level function, respectively, to obtain a solution that is  $\epsilon_f$ -optimal for the upper-level and  $\epsilon_g$ -optimal for the lower-level. We improve these bounds and also extend our results to nonconvex settings.
Contributions. In this paper, we present novel projection-free stochastic bilevel optimization methods with tight non-asymptotic guarantees for both upper and lower-level problems. At each iteration, the algorithms use a small number of samples to build unbiased and low variance estimates and construct a cutting plane to locally approximate the solution set of the lower-level problem and then combine it with a Frank-Wolfe-type update on the upper-level objective. Our methods require careful construction of the cutting plane so that with high probability it contains the solution set of the lower-level problem, which is obtained by selecting proper function and gradient estimators to achieve the obtained optimal convergence guarantees. Next, we summarize our main theoretical results for the proposed Stochastic Bilevel Conditional Gradient methods for Infinite and Finite sample settings denoted by SBCGI and SBCGF, respectively.
- (Stochastic setting) We show that SBCGI (Algorithm 1), in the convex setting, finds a solution  $\hat{\mathbf{x}}$  that satisfies  $f(\hat{\mathbf{x}}) - f^{*} \leq \epsilon_{f}$  and  $g(\hat{\mathbf{x}}) - g^{*} \leq \epsilon_{g}$  with probability  $1 - \delta$  within  $\mathcal{O}(\log(d / \delta \epsilon) / \epsilon^{2})$  stochastic oracle queries, where  $\epsilon = \min\{\epsilon_{f}, \epsilon_{g}\}$ ,  $f^{*}$  is the optimal value of problem (1) and  $g^{*}$  is the optimal value of the lower-level problem. Moreover, in the non-convex setting, it finds  $\hat{\mathbf{x}}$  satisfying  $\mathcal{G}(\hat{\mathbf{x}}) \leq \epsilon_{f}$  and  $g(\hat{\mathbf{x}}) - g^{*} \leq \epsilon_{g}$  with probability  $1 - \delta$  within  $\mathcal{O}((\log(d / \delta \epsilon))^{3/2} / \epsilon^{3})$  stochastic oracle queries, where  $\mathcal{G}(\hat{\mathbf{x}})$  is the Frank-Wolfe (FW) gap.
- (Finite-sum setting) We show that SBCGF (Algorithm 2), in the convex setting, finds  $\hat{\mathbf{x}}$  that satisfies  $f(\hat{\mathbf{x}}) - f^{*} \leq \epsilon_{f}$  and  $g(\hat{\mathbf{x}}) - g^{*} \leq \epsilon_{g}$  with probability  $1 - \delta$  within  $\mathcal{O}(\sqrt{n} (\log (1 / \delta \epsilon))^{3 / 2} / \epsilon)$  stochastic oracle queries, where  $n$  is the number of samples of finite-sum problem. Moreover, in the nonconvex setting, it finds  $\hat{\mathbf{x}}$  that satisfies  $\mathcal{G}(\hat{\mathbf{x}}) \leq \epsilon_{f}$  and  $g(\hat{\mathbf{x}}) - g^{*} \leq \epsilon_{g}$  with probability  $1 - \delta$  within  $\mathcal{O}(\sqrt{n} \log (1 / \delta \epsilon) / \epsilon^{2})$  stochastic oracle queries.
General stochastic bilevel. In a general format of stochastic bilevel problems, the upper-level function  $f$  also depends on an extra variable  $y \in \mathbb{R}^p$  which also affects the lower-level objective,
$$
\min  _ {\mathbf {x} \in \mathbb {R} ^ {d}, \mathbf {y} \in \mathbb {R} ^ {p}} f (\mathbf {x}, \mathbf {y}) = \mathbb {E} [ \tilde {f} (\mathbf {x}, \mathbf {y}, \theta) ] \quad \text {s . t .} \mathbf {x} \in \underset {\mathbf {z} \in \mathcal {Z}} {\arg \min } g (\mathbf {z}, \mathbf {y}) = \mathbb {E} [ \tilde {g} (\mathbf {z}, \mathbf {y}, \xi) ]. \tag {2}
$$
There have been several works including [10, 17-21] on solving the general stochastic bilevel problem (2). However, they only focus on the setting where the lower-level problem is strongly convex, i.e.,  $g(\mathbf{z}, \mathbf{y})$  is strongly convex with respect to  $\mathbf{z}$  for any value of  $\mathbf{y}$ . In fact, (2) with a convex lower-level problem is known to be NP-hard [22]. Hence, the results of these works are not directly comparable with our work as we focus on a simpler setting, but our assumption on the lower-level objective function is weaker and it only requires the function to be convex.
Deterministic simple bilevel. There have been some recent results on non-asymptotic guarantees for the deterministic variant of problem (1). The BiG-SAM algorithm was presented in [14], and it was shown that its lower-level objective error converges to zero at a rate of  $\mathcal{O}(1 / t)$ , while the upper-level error asymptotically converges to zero. In [15], the authors achieved the first non-asymptotic rate for both upper- and lower-level problems by introducing an iterative regularization-based method which achieves an  $(\epsilon_f, \epsilon_g)$ -optimal solution after  $\mathcal{O}(\max \{1 / \epsilon_f^4, 1 / \epsilon_g^4\})$  iterations. In [12], the authors proposed a projection-free method for deterministic simple bilevel problems that has a complexity of  $\mathcal{O}(\max \{1 / \epsilon_f, 1 / \epsilon_g\})$  for convex upper-level and complexity of  $\mathcal{O}(\max \{1 / \epsilon_f^2, 1 / (\epsilon_f \epsilon_g)\})$  for non-convex upper-level. Moreover, in [23] the authors presented a switching gradient method to solve simple bilevel problems with convex smooth functions for both upper- and lower-level problems with complexity  $\mathcal{O}(1 / \epsilon)$ . However, all the above results are limited to the deterministic setting.
General bilevel without lower-level strong convexity. Recently, there are several recent works on general bilevel optimization problems without lower-level strong convexity including [24-27, 23]. However, they either have a weaker theoretical results like asymptotic convergence rate in [24] or have some additional assumptions. Specifically, in [25], the authors reformulated the problem as a constrained optimization problem and further assumes such problem to be a convex program. Moreover, in [26, 27], the authors in both papers assumed that the lower-level objective satisfies the PL inequality, while we assumed that the lower-level objective is convex. In [23], the authors used a looser convergence criterion that only guarantees convergence to a Goldstein stationary point. Since these works consider a more general class of problems, we argue that their theoretical results when applied to our setting are necessarily weaker.
Example 1: Over-parameterized regression. A general form of problem (1) is when the lower-level problem represents training loss and the upper-level represents test loss. The goal is to minimize the test loss by selecting one of the optimal solutions for the training loss [13]. An instance of that is the constrained regression problem, where we intend to find an optimal parameter vector  $\beta \in \mathbb{R}^d$  that minimizes the loss  $\ell_{\mathrm{tr}}(\beta)$  over the training dataset  $\mathcal{D}_{\mathrm{tr}}$ . To represent some prior knowledge, we usually constrain  $\beta$  to be in some subsets  $\mathcal{Z} \subseteq \mathbb{R}^d$ , e.g.,  $\mathcal{Z} = \{\beta | \| \beta \|_1 \leq \lambda\}$  for some  $\lambda > 0$  to induce sparsity. To handle multiple global minima, we adopt the over-parameterized approach, where the number of samples is less than the parameters. Although achieving one of these global minima is possible, not all optimal solutions perform equally on other datasets. Hence, we introduce an upper-level objective: the loss on a validation set  $\mathcal{D}_{\mathrm{val}}$ . This helps select a training loss optimizer that performs well on both training and validation sets. It leads to the following bilevel problem:
$$
\min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {d}} f (\boldsymbol {\beta}) \triangleq \ell_ {\text {v a l}} (\boldsymbol {\beta}) \quad \text {s . t .} \quad \boldsymbol {\beta} \in \underset {\mathbf {z} \in \mathcal {Z}} {\operatorname {a r g m i n}} g (\mathbf {z}) \triangleq \ell_ {\mathrm {t r}} (\mathbf {z}) \tag {3}
$$
In this case, both the upper- and lower-level losses are smooth and convex if  $\ell$  is smooth and convex.
Example 2: Dictionary learning. Problem (1) also appears in lifelong learning, where the learner takes a series of tasks sequentially and tries to accumulate knowledge from past tasks to improve performance in new tasks. Here we focus on continual dictionary learning. The aim of dictionary
learning is to obtain a compact representation of the input data. Let  $\mathbf{A} = \{\mathbf{a}_1,\dots ,\mathbf{a}_n\} \in \mathbb{R}^{m\times n}$  denote a dataset of  $n$  points. We seek a dictionary  $\mathbf{D} = [\mathbf{d}_1,\ldots ,\mathbf{d}_p]\in \mathbb{R}^{m\times p}$  such that all data points  $\mathbf{a}_i$  can be represented by a linear combination of basis vectors in  $\mathbf{D}$  which can be cast as [28-31]:
$$
\min  _ {\mathbf {D} \in \mathbb {R} ^ {m \times p}} \min  _ {\mathbf {X} \in \mathbb {R} ^ {p \times n}} \frac {1}{2 n} \sum_ {i \in \mathcal {N}} \| \mathbf {a} _ {i} - \mathbf {D} \mathbf {x} _ {i} \| _ {2} ^ {2} \quad \text {s . t .} \| \mathbf {d} _ {j} \| _ {2} \leq 1, j = 1, \dots , p; \| \mathbf {x} _ {i} \| _ {1} \leq \delta , i \in \mathcal {N}. \tag {4}
$$
Moreover, we denote  $\mathbf{X} = [\mathbf{x}_1,\dots ,\mathbf{x}_n]\in \mathbb{R}^{p\times n}$  as the coefficient matrix. In practice, data points usually arrive sequentially and the representation evolves gradually. Hence, the dictionary must be updated sequentially as well. Assume that we already have learned a dictionary  $\hat{\mathbf{D}}\in \mathbb{R}^{m\times p}$  and the corresponding coefficient matrix  $\hat{\mathbf{X}}\in \mathbb{R}^{p\times n}$  for the dataset  $\mathbf{A}$ . As a new dataset  $\mathbf{A}' = \{\mathbf{a}_1',\ldots ,\mathbf{a}_{n'}'\}$  arrives, we intend to enrich our dictionary by learning  $\tilde{\mathbf{D}}\in \mathbb{R}^{m\times q}(q > p)$  and the coefficient matrix  $\tilde{\mathbf{X}}\in \mathbb{R}^{q\times n'}$  for the new dataset while maintaining good performance of  $\tilde{\mathbf{D}}$  on the old dataset  $\mathbf{A}$  as well as the learned coefficient matrix  $\hat{\mathbf{X}}$ . This leads to the following stochastic bilevel problem:
$$
\min  _ {\tilde {\mathbf {D}} \in \mathbb {R} ^ {m \times q}} \min  _ {\tilde {\mathbf {X}} \in \mathbb {R} ^ {q \times n ^ {\prime}}} f (\tilde {\mathbf {D}}, \tilde {\mathbf {X}}) \quad \text {s . t .} \quad \| \tilde {\mathbf {x}} _ {k} \| _ {1} \leq \delta , k = 1, \dots , n ^ {\prime}; \tilde {\mathbf {D}} \in \operatorname * {a r g m i n} _ {\| \tilde {\mathbf {d}} _ {j} \| _ {2} \leq 1} g (\tilde {\mathbf {D}}), \tag {5}
$$
where  $f(\tilde{\mathbf{D}},\tilde{\mathbf{X}})\triangleq \frac{1}{2n^{\prime}}\sum_{k = 1}^{n^{\prime}}\| \mathbf{a}_{k}^{\prime} - \tilde{\mathbf{D}}\tilde{\mathbf{x}}_{k}\|_{2}^{2}$  represents the average reconstruction error on the new dataset  $\mathbf{A}'$ , and  $g(\tilde{\mathbf{D}})\triangleq \frac{1}{2n}\sum_{i = 1}^{n}\| \mathbf{a}_{i} - \tilde{\mathbf{D}}\hat{\mathbf{x}}_{i}\|_{2}^{2}$  represents the error on the old dataset  $\mathbf{A}$ . Note that we denote  $\hat{\mathbf{x}}_i$  as the prolonged vector in  $\mathbb{R}^q$  by appending zeros at the end. In problem (5), the upper-level objective is non-convex, while the lower-level loss is convex with multiple minima.
2.2 Assumptions and definitions
Next, we formally state the assumptions required in this work.
Assumption 2.1.  $\mathcal{Z}$  is convex and compact with diameter  $D$ , i.e.,  $\forall \mathbf{x}, \mathbf{y} \in \mathcal{Z}$ , we have  $\| \mathbf{x} - \mathbf{y} \| \leq D$ .
Assumption 2.2. The upper-level stochastic function  $\tilde{f}$  satisfies the following conditions:
(i)  $\nabla \tilde{f}$  is Lipschitz with constant  $L_{f}$ , i.e.,  $\forall \mathbf{x},\mathbf{y}\in \mathcal{Z},\forall \theta ,\| \nabla \tilde{f} (\mathbf{x},\theta) - \nabla \tilde{f} (\mathbf{y},\theta)\| \leq L_f\| \mathbf{x} - \mathbf{y}\|$ .
(ii) The stochastic gradients noise is sub-Gaussian,  $\mathbb{E}[\exp \{\| \nabla \tilde{f} (\mathbf{x},\theta) - \nabla f(\mathbf{x})\| ^2 /\sigma_f^2\} ]\leq \exp \{1\}$ .
Assumption 2.3. The lower-level stochastic function  $\tilde{g}$  satisfies the following conditions:
(i)  $g$  is convex and  $\nabla \tilde{g}$  is  $L_{g}$ -Lipschitz, i.e.,  $\forall \mathbf{x},\mathbf{y}\in \mathcal{Z},\forall \xi ,\| \nabla \tilde{g} (\mathbf{x},\xi) - \nabla \tilde{g} (\mathbf{y},\xi)\| \leq L_g\| \mathbf{x} - \mathbf{y}\|$ .
(ii) The stochastic gradients noise is sub-Gaussian,  $\mathbb{E}[\exp \{\| \nabla \tilde{g} (\mathbf{x},\xi) - \nabla g(\mathbf{x})\| ^2 /\sigma_g^2\} ]\leq \exp \{1\}$ .
(iii) The stochastic functions noise is sub-Gaussian,  $\mathbb{E}[\exp \{| \tilde{g} (\mathbf{x},\xi) - g(\mathbf{x})|^2 /\sigma_l^2\} ]\leq \exp \{1\}$ .
Remark 2.1. Assumptions 2.1 and 2.3(i) imply that  $\tilde{g}$  is Lipschitz continuous on an open set containing  $\mathcal{Z}$  with some constant  $L_{l}$ , i.e., for all  $\mathbf{x},\mathbf{y}\in \mathcal{Z}$ , we have  $|\tilde{g} (\mathbf{x}) - \tilde{g} (\mathbf{y})|\leq L_l\| \mathbf{x} - \mathbf{y}\|$ .
In the paper, we denote  $g^{*} \triangleq \min_{\mathbf{z} \in \mathcal{Z}} g(\mathbf{z})$  and  $\mathcal{X}_g^* \triangleq \arg \min_{\mathbf{z} \in \mathcal{Z}} g(\mathbf{z})$  as the optimal value and the optimal solution set of the lower-level problem, respectively. Note that by Assumption 2.3, the set  $\mathcal{X}_g^*$  is nonempty, convex, and compact, but typically not a singleton as  $g$  potentially has multiple minima on  $\mathcal{Z}$ . Furthermore, we denote  $f^{*}$  and  $\mathbf{x}^{*}$  as the optimal value and an optimal solution of problem (1), which are assured to exist since  $f$  is continuous and  $\mathcal{X}_g^*$  is compact.
Definition 2.1. When  $f$  is convex, a point  $\hat{\mathbf{x}} \in \mathcal{Z}$  is  $(\epsilon_f, \epsilon_g)$ -optimal if  $f(\hat{\mathbf{x}}) - f^* \leq \epsilon_f$  and  $g(\hat{\mathbf{x}}) - g^* \leq \epsilon_g$ . When  $f$  is non-convex,  $\hat{\mathbf{x}} \in \mathcal{Z}$  is  $(\epsilon_f, \epsilon_g)$ -optimal if  $\mathcal{G}(\hat{\mathbf{x}}) \leq \epsilon_f$  and  $g(\hat{\mathbf{x}}) - g^* \leq \epsilon_g$ , where  $\mathcal{G}(\hat{\mathbf{x}})$  is the FW gap [32, 33] defined as  $\mathcal{G}(\hat{\mathbf{x}}) \triangleq \max_{\mathbf{s} \in \mathcal{X}_g^*} \{ \langle \nabla f(\hat{\mathbf{x}}, \hat{\mathbf{x}} - \mathbf{s} \rangle \}$ .
Conditional gradient for simple bilevel optimization. A variant of the conditional gradient (CG) method for solving bilevel problems has been introduced in [12] which uses a cutting plane idea [34]
to approximate the solution set of the lower-level problem denoted by  $\mathcal{X}_g^*$ . More precisely, if one has access to  $\mathcal{X}_g^*$ , it is possible to run the FW update with stepsize  $\gamma_t$  as
$$
\mathbf {x} _ {t + 1} = \left(1 - \gamma_ {t}\right) \mathbf {x} _ {t} + \gamma_ {t} \mathbf {s} _ {t}, \quad \text {w h e r e} \quad \mathbf {s} _ {t} = \underset {\mathbf {s} \in \mathcal {X} _ {g} ^ {*}} {\arg \min } \langle \nabla f (\mathbf {x} _ {t}), s \rangle \tag {6}
$$
However, the set  $\mathcal{X}_g^*$  is not explicitly given and the above method is not implementable. In [12], the authors suggested the use of the following set:  $\mathcal{X}_t = \{\mathbf{s} \in \mathcal{Z} : \langle \nabla g(\mathbf{x}_t), \mathbf{s} - \mathbf{x}_t \rangle \leq g(\mathbf{x}_0) - g(\mathbf{x}_t)\}$  instead of the set  $\mathcal{X}_g^*$  in the FW update given in (6). Note that  $\mathbf{x}_0$  is selected in a way that  $g(\mathbf{x}_0) - g^*$  is smaller than  $\epsilon_g / 2$ , and such a point can be efficiently computed. A crucial property of the above set is that it always contains the solution set of the lower-level problem denoted by  $\mathcal{X}_g^*$ . This can be easily verified by the fact that for any  $\mathbf{x}_g^*$  in  $\mathcal{X}_g^*$  we have  $\mathbf{x}_g^* \in \mathcal{Z}$  and
$$
\langle \nabla g (\mathbf {x} _ {t}), \mathbf {x} _ {g} ^ {*} - \mathbf {x} _ {t} \rangle \leq g \left(\mathbf {x} _ {g} ^ {*}\right) - g \left(\mathbf {x} _ {t}\right) \leq g \left(\mathbf {x} _ {0}\right) - g \left(\mathbf {x} _ {t}\right), \tag {7}
$$
where the second inequality holds as  $g(\mathbf{x}_0) \geq g(\mathbf{x}_g^*)$ . As shown in [12], this condition is sufficient to show that if one follows the update in (6) with  $\mathcal{X}_t$  instead of  $\mathcal{X}_g^*$ , the iterates will converge to the optimal solution. However, this framework is not applicable to the stochastic setting as we cannot access the functions or their gradients. Next, we present our main idea to address this delicate issue.
Random set for the subproblem. A natural idea to address stochasticity is to replace all gradients and functions with their stochastic estimators for both the subproblem in (6), i.e.,  $\nabla f(\mathbf{x}_t)$ , as well as the construction of the cutting plane  $\mathcal{X}_t$ , i.e.,  $g(\mathbf{x}_t)$ , and  $\nabla g(\mathbf{x}_t)$ . However, this simple idea fails since the set  $\mathcal{X}_t$  may no longer contain the solution set  $\mathcal{X}_g^*$ . More precisely, if  $\hat{g}_t$  and  $\widehat{\nabla g}_t$  are unbiased estimators of  $g(\mathbf{x}_t)$  and  $\nabla g(\mathbf{x}_t)$ , respectively, for the following approximation set
$$
\mathcal {X} _ {t} ^ {\prime} = \left\{\mathbf {s} \in \mathcal {Z}: \langle \widehat {\nabla g} _ {t}, \mathbf {s} - \mathbf {x} _ {t} \rangle \leq g \left(\mathbf {x} _ {0}\right) - \hat {g} _ {t} \right\} \tag {8}
$$
we can not argue that it contains  $\mathcal{X}_g^*$ , as the second inequality in (7) does not hold, i.e.,  $\langle \widehat{\nabla g}(\mathbf{x}_t), \mathbf{x}_g^* - \mathbf{x}_t \rangle \leq \hat{g}(\mathbf{x}_g^*) - \hat{g}(\mathbf{x}_t) \not\leq g(\mathbf{x}_0) - \hat{g}(\mathbf{x}_t)$ . In the appendix, we numerically illustrate this point.
To address this issue, we tune the cutting plane by only moving it but not rotating it, i.e., adding another term to tolerate the noise from stochastic estimates. We introduce the stochastic cutting plane
$$
\hat {\mathcal {X}} _ {t} = \left\{\mathbf {s} \in \mathcal {Z}: \langle \widehat {\nabla g} _ {t}, \mathbf {s} - \mathbf {x} _ {t} \rangle \leq g \left(\mathbf {x} _ {0}\right) - \hat {g} _ {t} + K _ {t} \right\}, \tag {9}
$$