SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[42] Ohad Shamir, Nati Srebro, and Tong Zhang. "Communication-Efficient Distributed Optimization using an Approximate Newton-type Method". In: Proceedings of the 31st International Conference on Machine Learning. Ed. by Eric P. Xing and Tony Jebara. Vol. 32. Proceedings of Machine Learning Research 2. Beijing, China: PMLR, 22-24 Jun 2014, pp. 1000-1008.
[43] D. F. Shanno. "Conditioning of Quasi-Newton Methods for Function Minimization". In: Mathematics of Computation 24.111 (1970), pp. 647-656.
[44] Vatsal Sharan, Aaron Sidford, and Gregory Valiant. "Memory-Sample Tradeoffs for Linear Regression with Small Error". In: Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing. STOC 2019. Association for Computing Machinery, 2019, pp. 890-901.
[45] Virginia Smith et al. "CoCoA: A General Framework for Communication-Efficient Distributed Optimization". In: J. Mach. Learn. Res. 18.1 (Jan. 2017), pp. 8590-8638.
[46] Adrien Taylor and Yoel Drori. "An optimal gradient method for smooth strongly convex minimization". In: Mathematical Programming 199.1 (May 2023), pp. 557-594.
[47] Choon Hui Teo et al. "Bundle Methods for Regularized Risk Minimization". In: Journal of Machine Learning Research 11.10 (2010), pp. 311-365.
[48] Pravin M Vaidya. “A new algorithm for minimizing convex functions over convex sets”. In: Mathematical programming 73.3 (1996), pp. 291–341.
[49] Stephen A Vavasis. "Black-box complexity of local minimization". In: SIAM Journal on Optimization 3.1 (1993), pp. 60-80.
[50] Jialei Wang, Weiran Wang, and Nathan Srebro. "Memory and Communication Efficient Distributed Stochastic Optimization with Minibatch Prox". In: Proceedings of the 2017 Conference on Learning Theory. Ed. by Satyen Kale and Ohad Shamir. Vol. 65. Proceedings of Machine Learning Research. PMLR, July 2017, pp. 1882-1919.
[51] Jianqiao Wangni et al. "Gradient Sparsification for Communication-Efficient Distributed Optimization". In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. NIPS'18. Montreal, Canada: Curran Associates Inc., 2018, pp. 1306-1316.
[52] Blake Woodworth and Nathan Srebro. "Open problem: The oracle complexity of convex optimization with limited memory". In: Conference on Learning Theory. PMLR. 2019, pp. 3202-3210.
[53] Yuchen Zhang, John C. Duchi, and Martin J. Wainwright. "Communication-efficient algorithms for statistical optimization". In: 2012 IEEE 51st IEEE Conference on Decision and Control (CDC). 2012, pp. 6792-6792.
A Proof of the query complexity and memory usage of Algorithm 4
First, we give simple properties on the discretization functions. One can easily check that for any  $\pmb{x} \in C$ ,
$$
\left\| \boldsymbol {x} - \operatorname {D i s c r e t i z e} _ {d} (\boldsymbol {x}, \xi) \right\| \leq \xi \quad \text {a n d} \quad \left\| \operatorname {D i s c r e t i z e} _ {d} (\boldsymbol {x}, \xi) \right\| \leq \| \boldsymbol {x} \|. \tag {1}
$$
Further, one can easily check that to represent any output of  $\mathrm{Discretize}_d(\cdot ,\xi)$ , one needs at most  $d\ln \frac{2\sqrt{d}}{\xi} = \mathcal{O}(d\ln \frac{d}{\xi})$  bits.
We next prove Lemma 4.1.
Proof of Lemma 4.1. We first consider the case when the algorithm terminates because of a query  $\pmb{g} = O(\mathcal{P}_t)$  such that  $\| \pmb{g} \| \leq \delta / (2\sqrt{d})$ . Then, for any  $\pmb{x} \in \mathcal{C}_d$ , one directly has
$$
\boldsymbol {g} ^ {\top} \boldsymbol {x} - b \leq \boldsymbol {g} ^ {\top} (\boldsymbol {x} - \boldsymbol {\omega}) \leq 2 \sqrt {d} \| \boldsymbol {g} \| \leq \delta .
$$
where  $\omega$  is the volumetric center of the resulting polyhedron. In the second inequality we used the fact that  $\omega \in \mathcal{C}_d$ , otherwise the algorithm would not have terminated at that step.
We next turn to the other cases and start by showing that the output polyhedron does not contain a ball of radius  $\delta$ . This is immediate if the algorithm terminated because the polyhedron was empty. We then suppose this was not the case, and follow the same proof as given in [2]. Algorithm 1 and the one provided in [2] coincide when removing a constraint of the polyhedron. Hence, it suffices to consider the case when we add a constraint. We use the notation  $\tilde{\pmb{A}}^{\top} = [\pmb{A}^{\top}, \pmb{a}_{m+1}^{\top}]$ ,  $\tilde{\pmb{b}}^{\top} = [\pmb{b}^{\top}, b_{m+1}]$  for the updated matrix  $\pmb{A}$  and vector  $\pmb{b}$  after adding the constraint. We also denote  $\omega = \text{VolumetricCenter}(\pmb{A}, \pmb{b})$  (resp.  $\tilde{\omega} = \text{VolumetricCenter}(\tilde{\pmb{A}}, \tilde{\pmb{b}})$ ) the volumetric center of the polyhedron before (resp. after) adding the constraint. Next, we consider the vector  $(\pmb{b}')^{\top} = [\pmb{b}^{\top}, \pmb{a}_{m+1}^{\top} \omega]$ , which would have been obtained if the cut was performed at  $\omega$  exactly. We then denote  $\omega' = \text{VolumetricCenter}(\tilde{\pmb{A}}, \pmb{b}')$ . Then proof of [2] shows that
$$
V _ {\tilde {A}, b ^ {\prime}} \left(\omega^ {\prime}\right) \geq V _ {A, b} (\omega) + 0. 0 3 4 0.
$$
We now observe that by construction, we have  $\tilde{\pmb{b}}_{m + 1}\geq \pmb{a}_{m + 1}^{\top}\pmb {\omega}$ , so that the polyhedron associated to  $(\tilde{A},\tilde{b})$  is more constrained than the one associated to  $(\tilde{A},\boldsymbol {b}^{\prime})$ . As a result, we have  $V_{\tilde{A},\tilde{b}}(\boldsymbol {x})\geq V_{\tilde{A},\boldsymbol{b}^{\prime}}(\boldsymbol {x})$ , for any  $\boldsymbol {x}\in \mathbb{R}^{d}$  such that  $\tilde{A}\boldsymbol {x}\geq \tilde{\boldsymbol{b}}$ . Therefore,
$$
V _ {\tilde {A}, \tilde {b}} (\tilde {\omega}) \geq V _ {\tilde {A}, b ^ {\prime}} (\tilde {\omega}) \geq V _ {\tilde {A}, b ^ {\prime}} \left(\omega^ {\prime}\right) \geq V _ {A, b} (\omega) + 0. 0 3 4 0.
$$
This ends the modifications in the proof of [2]. With the notations of this paper, we still have  $\Delta V^{+} = 0.340$  and  $\Delta V^{-} = 0.326$ , so that  $\Delta V = 0.0014$ . Then, because  $c = \frac{1}{\Delta V}$ , the same proof shows that the procedure is successful for precision  $\delta$ : the final polyhedron  $(A, b)$  returned by Algorithm 1 does not contain a ball of radius  $> \delta$ . As a result, whether the algorithm performed all  $T_{max}$  iterations or not,  $\{\pmb{x}: A\pmb{x} \geq \pmb{b}\}$  does not contain a ball of radius  $> \delta'$ , where  $\pmb{A}$  and  $\pmb{b}$  define the constraints stored in the output  $\mathcal{P}$ . Now letting  $m$  be the objective value of the right optimization problem, there exists  $\pmb{x} \in \mathcal{C}_d$  such that for all  $t \leq T$ ,  $\pmb{g}_t^\top (\pmb{x} - \pmb{c}_t) \geq m$ . Therefore, for any  $\pmb{x}' \in B_d(\pmb{x}, m)$  one has
$$
\forall i \in [ m ], \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} ^ {\prime} - b _ {i} \geq m + \boldsymbol {a} _ {t} ^ {\top} (\boldsymbol {x} ^ {\prime} - \boldsymbol {x}) \geq m - \| \boldsymbol {x} ^ {\prime} - \boldsymbol {x} \| \geq 0.
$$
In the last inequality we used  $\| \pmb{a}_t\| \leq 1$ . This implies that the polyhedron contains  $B_{d}(\pmb{x},m)$ . Hence,  $m \leq \delta$ .
This ends the proof of the right inequality. The left equality is a direct application of strong duality for linear programming.
We now prove that Algorithm 4 has the desired oracle-complexity and memory usage.
We first describe the recursive calls of Algorithm 3 in more detail. To do so, consider running the procedure ApproxOracle  $(i,j,\mathcal{P}^{(1)},\ldots ,\mathcal{P}^{(i)})$  where  $i < p$ , which corresponds to running Algorithm 2 for specific oracles. We say that this is a level- $i$  run. Then, the algorithm performs at most  $2T(\delta ,k_{i + 1})$  calls to ApproxOracle  $(i + 1,i + 1,\mathcal{P}^{(1)},\dots ,\mathcal{P}^{(i)},\cdot)$ , where the factor 2 comes from the fact that
Vaidya's method Algorithm 1 is effectively run twice in Algorithm 2. The solution to  $(\mathcal{P}_{aux}(\mathcal{P}))$  has as many components as constraints in the last polyhedron, which is at most  $\frac{k_{i+1}}{\sigma_{min}} + 1$  by Lemma 4.2. Hence, the number of calls to ApproxOracle  $(i + 1, j, \mathcal{P}^{(1)}, \ldots, \mathcal{P}^{(i)}, \cdot)$  is at most  $\frac{k_{i+1}}{\sigma_{min}} + 1$ . In total, that is  $\mathcal{O}(k_{i+1} \ln \frac{1}{\delta})$  calls to the level  $i + 1$  of the recursion.
We next aim to understand the output of running ApproxOracle(1, 1,  $\mathcal{P}^{(1)})$ . We denote by  $\lambda (\mathcal{P}^{(1)})$  the solution  $\mathcal{P}_{aux}(\mathcal{P}^{\star})$  computed at 1.2 of the first call to Algorithm 2, where  $\mathcal{P}^{\star}$  is the output polyhedron of the first call to Algorithm 1. Denote by  $S(\mathcal{P}^{(1)})$  the set of indices of coordinates from  $\lambda (\mathcal{P}^{(1)})$  for which the procedure performed a call to ApproxOracle(2, 1,  $\mathcal{P}^{(1)},\cdot$ ). In other words,  $S(\mathcal{P}^{(1)})$  contains the indices of all coordinates of  $\lambda (\mathcal{P}^{(1)})$ , except those for which the corresponding query lay outside of the unit cube, or the initial constraints of the cube. For any index  $l\in S(\mathcal{P}^{(1)})$ , let  $\mathcal{P}_l^{(2)}$  denote the state of the current polyhedron  $(\mathcal{P}_t$  in 1.7 of Algorithm 2) when that call was performed. Up to discretization issues, the output of the complete procedure is
$$
\sum_ {l \in \mathcal {S} (\mathcal {P} ^ {(1)})} \lambda_ {l} (\mathcal {P} ^ {(1)}) \text {A p p r o x O r a c l e} (2, 1, \mathcal {P} ^ {(1)}, \mathcal {P} _ {l} ^ {(2)}).
$$
We continue in the recursion, defining  $\pmb{\lambda}(\mathcal{P}^{(1)},\mathcal{P}_l^{(2)})$  and  $\mathcal{S}(\mathcal{P}^{(1)},\mathcal{P}_l^{(2)})$  for all  $l\in S(\mathcal{P}^{(1)})$ , until we define all vectors of the form  $\pmb {\lambda}(\mathcal{P}^{(1)},\mathcal{P}_{l_2}^{(2)},\dots,\mathcal{P}_{l_r}^{(r)})$  and sets of the form  $\mathcal{S}(\mathcal{P}^{(1)},\mathcal{P}_{l_2}^{(2)},\dots,\mathcal{P}_{l_r}^{(r)})$  for  $i + 1\leq r\leq p - 1$ . To simplify the notation and emphasize that all these polyhedra depend on the recursive computation path, we adopt the notation
$$
\lambda^ {l _ {2}, \dots , l _ {r + 1}} := \lambda_ {l _ {r + 1}} \left(\mathcal {P} ^ {(1)}, \mathcal {P} _ {l _ {2}} ^ {(2)}, \dots , \mathcal {P} _ {l _ {r}} ^ {(r)}\right)
$$
$$
\mathcal {S} ^ {l _ {2}, \dots , l _ {r}} := \mathcal {S} (\mathcal {P} ^ {(1)}, \mathcal {P} _ {l _ {2}} ^ {(2)}, \dots , \mathcal {P} _ {l _ {r}} ^ {(r)})
$$
We recall that these polyhedron are kept in memory to query their volumetric center. For ease of notation, we write  $\boldsymbol{x}_1 = \text{VolumetricCenter}(\mathcal{P}^{(1)})$ , and we write  $\boldsymbol{c}^{l_2,\dots,l_r} = \text{VolumetricCenter}(\mathcal{P}_{l_r}^{(r)})$  for  $2 \leq r \leq p$ , where  $l_2, \ldots, l_{r-1}$  were the indices from the computation path leading up to  $\mathcal{P}_{l_r}^{(r)}$ . Last, we write  $O_S = (O_{S,1}, \ldots, O_{S,p})$ , where  $O_{S,i}: \mathcal{C}_d \to \mathbb{R}^{k_i}$  is the “ $\boldsymbol{x}_i$ ” component of  $O_S$ , for all  $i \in [p]$ .
With all these notations, we will show that the output of ApproxOracle  $(i,j,\mathcal{P}^{(1)},\mathcal{P}_{l_2}^{(2)},\ldots ,\mathcal{P}_{l_i}^{(i)})$  is approximately equal to the vector
$$
\begin{array}{l} G (i, j, \boldsymbol {x} _ {1}, \boldsymbol {c} ^ {l _ {2}}, \dots , \boldsymbol {c} ^ {l _ {2}, \dots , l _ {i}}) \\ := \sum_{\substack{l_{i + 1}\in \mathcal{S}, l_{i + 2}\in \mathcal{S}^{l_{i + 1}},\\ \ldots ,  l_{p}\in \mathcal{S}^{l_{i + 1},\ldots ,l_{p - 1}}}}\lambda^{l_{i + 1}}\lambda^{l_{i + 1},l_{i + 2}}\dots \lambda^{l_{i + 1},\ldots ,l_{p}}\cdot O_{S,j}(\boldsymbol{x}_{1},\boldsymbol{c}^{l_{2}},\ldots ,\boldsymbol{c}^{l_{2},\ldots ,l_{p}}), \\ \end{array}
$$
with the convention that for  $i = p$
$$
G (p, j, \boldsymbol {x} _ {1}, \boldsymbol {c} ^ {l _ {2}}, \dots , \boldsymbol {c} ^ {l _ {2}, \dots , l _ {p}}) := O _ {S, j} (\boldsymbol {x} _ {1}, \boldsymbol {c} ^ {l _ {2}}, \dots , \boldsymbol {c} ^ {l _ {2}, \dots , l _ {p}}).
$$
The corresponding computation tree is represented in Fig. 3. For convenience, we omitted the term  $j = 1$ .
We start the analysis with a simple result showing that if the oracle  $O_S$  returns separation vectors of norm bounded by one, then the responses from ApproxOracle also lie in the unit ball.
Lemma A.1. Fix  $\delta, \xi \in (0,1)$ ,  $1 \leq j \leq i \leq p$  and an oracle  $O_S = (O_{S,1},\dots,O_{S,p}): \mathcal{C}_d \to \mathbb{R}^d$ . Suppose that  $O_S$  takes values in the unit ball. For any  $s \in [i]$  let  $\mathcal{P}_{l_s}^{(s)} \in \mathcal{I}_{k_s}$  represent a bounded polyhedrons with VolumetricCenter  $(\mathcal{P}_{l_s}^{(s)}) \in \mathcal{C}_{k_s}$ . Then, one has
$$
\| \text {A p p r o x O r a c l e} _ {\delta , \xi , O _ {S}} (i, j, \mathcal {P} _ {l _ {1}} ^ {(1)}, \dots , \mathcal {P} _ {l _ {i}} ^ {(i)}) \| \leq 1.
$$
Proof. We prove this by simple induction on  $i$ . For convenience, we define the point  $\boldsymbol{x}_k = \text{VolumetricCenter}(\mathcal{P}_{l_k}^{(k)})$ . If  $i = p$ , we have
$$
\begin{array}{l} \left\| \text {A p p r o x O r a c l e} _ {\delta , \xi , O _ {S}} (i, j, \mathcal {P} _ {l _ {1}} ^ {(1)}, \dots , \mathcal {P} _ {l _ {i}} ^ {(i)}) \right\| = \left\| \text {D i s c r e t i z e} _ {k _ {j}} \left(O _ {S, j} \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {p}\right), \xi\right) \right\| \\ \leq \left\| O _ {S, j} \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {p}\right) \right\| \leq 1, \\ \end{array}
$$
Figure 3: Computation tree representing the recursive calls to ApproxOracle starting from the calls to ApproxOracle(1, 1, ·) from Algorithm 4
where in the first inequality we used Eq (1) and in the second inequality we used the fact that  $O_S(\pmb{x}_1,\dots ,\pmb{x}_p)$  has norm at most one. Now suppose that the result holds for  $i + 1\leq p$ . Then by construction, the output  $\mathrm{ApproxOracle}_{\delta ,\xi ,O_S}(i,j,\mathcal{P}_{l_1}^{(1)},\ldots ,\mathcal{P}_{l_i}^{(i)})$  is the result of iterative discretizations. Using Eq (1) and the previously defined notations, we obtain
$$
\begin{array}{l} \| \mathsf {A p p r o x O r a c l e} _ {\delta , \xi , O _ {S}} (i, j, \mathcal {P} _ {l _ {1}} ^ {(1)}, \ldots , \mathcal {P} _ {l _ {i}} ^ {(i)}) \| \\ \leq \left\| \sum_ {l _ {i + 1} \in S ^ {l _ {1}}, \dots , l _ {i}} \lambda^ {l _ {2}, \dots , l _ {i}} \mathsf {A p p r o x O r a c l e} _ {\delta , \xi , O _ {S}} (i + 1, j, \mathcal {P} _ {l _ {1}} ^ {(1)}, \dots , \mathcal {P} _ {l _ {i}} ^ {(i)}, \mathcal {P} _ {l _ {i + 1}} ^ {(i + 1)}) \right\| \leq 1. \\ \end{array}
$$
In the last inequality, we used the induction hypothesis together with the fact that  $\sum_{l_{i + 1}}\lambda^{l_2,\dots,l_{i + 1}}\leq 1$  using Eq (1). This ends the induction and the proof.
We are now ready to compare the output of Algorithm 3 to  $G(i,j,\pmb{x}_1,\pmb{c}^{l_2},\dots,\pmb{c}^{l_2,\dots,l_i})$ .
Lemma A.2. Fix  $\delta, \xi \in (0,1)$ ,  $1 \leq j \leq i \leq p$  and an oracle  $O_S = (O_{S,1},\dots,O_{S,p}): \mathcal{C}_d \to \mathbb{R}^d$ . Suppose that  $O_S$  takes values in the unit ball. For any  $s \in [i]$  let  $\mathcal{P}_{l_s}^{(s)} \in \mathcal{I}_{k_s}$  represent a bounded polyhedron with VolumetricCenter  $(\mathcal{P}_{l_s}^{(s)}) \in \mathcal{C}_{k_s}$ . Denote  $\pmb{x}_r = \pmb{c}(\mathcal{P}_{l_r}^{(r)})$  for  $r \in [i]$ . Then,
$$
\| \mathsf {A p p r o x O r a c l e} _ {\delta , \xi , O _ {S}} (i, j, \mathcal {P} _ {l _ {1}} ^ {(1)}, \ldots , \mathcal {P} _ {l _ {i}} ^ {(i)}) - G (i, j, \boldsymbol {x} _ {1}, \ldots , \boldsymbol {x} _ {i}) \| \leq \frac {4}{\sigma_ {m i n}} d \xi .
$$
Proof. We prove by simple induction on  $i$  that
$$
\begin{array}{l} \| \mathsf {A p p r o x O r a c l e} _ {\delta , \xi , O _ {S}} (i, j, \mathcal {P} _ {l _ {1}} ^ {(1)}, \ldots , \mathcal {P} _ {l _ {i}} ^ {(i)}) - G (i, j, \boldsymbol {x} _ {1}, \ldots , \boldsymbol {x} _ {i}) \| \\ \leq \left(1 + \frac {2}{\sigma_ {m i n}} (k _ {i + 1} + \dots + k _ {p}) + 2 (p - i)\right) \xi . \\ \end{array}
$$
First, for  $i = p$ , the result is immediate since the discretization is with precision  $\xi$  (1.4 of Algorithm 3). Now suppose that this is the case for  $i \leq p$  and any valid values of other parameters. For conciseness, we write  $G = (\mathcal{P}_{l_1}^{(1)},\dots,\mathcal{P}_{l_{i - 1}}^{(i - 1)})$ . Next, recall that by Lemma 4.2,  $|S^{l_2,\dots ,l_{i - 1}}| \leq \frac{k_i}{\sigma_{min}} +1$ . Hence,
the discretizations due to 1.8 of Algorithm 2 can affect the estimate for at most that number of rounds. Then, we have
$$
\begin{array}{l} \left\| \mathsf {A p p r o x O r a c l e} _ {\delta , \xi , O _ {S}} (i - 1, j, \boldsymbol {G}) - \sum_ {l _ {i} \in \mathcal {S} ^ {l _ {2}, \dots , l _ {i - 1}}} \tilde {\lambda} ^ {l _ {2}, \dots , l _ {i}} \mathsf {A p p r o x O r a c l e} _ {\delta , \xi , O _ {S}} (i, j, \boldsymbol {G}, \mathcal {P} _ {l _ {i}} ^ {(i)}) \right\| \\ \leq \left(\frac {k _ {i}}{\sigma_ {m i n}} + 1\right) \xi , \\ \end{array}
$$
where  $\tilde{\lambda}^{l_2,\dots,l_i}$  are the discretized coefficients that are used during the computation 1.8 of Algorithm 2. Now using Lemma A.1, we have
$$
\begin{array}{l} \left\| \sum_ {l _ {i} \in \mathcal {S} ^ {l _ {2}, \dots , l _ {i - 1}}} (\tilde {\lambda} ^ {l _ {2}, \dots , l _ {i}} - \lambda^ {l _ {2}, \dots , l _ {i}}) \mathrm {A p p r o x O r a c l e} _ {\delta , \xi , O _ {S}} (i, j, \boldsymbol {G}, \mathcal {P} _ {l _ {i}} ^ {(i)}) \right\| \\ \leq \| \tilde {\boldsymbol {\lambda}} ^ {l _ {i + 1}, \dots , l _ {i - 1}} - \boldsymbol {\lambda} ^ {l _ {i + 1}, \dots , l _ {i - 1}} \| _ {1} \leq \left(\frac {k _ {i}}{\sigma_ {m i n}} + 1\right) \xi . \\ \end{array}
$$
In the last inequality we used the fact that  $\lambda$  has at most  $\frac{k_i}{\sigma_{min}} + 1$  non-zero coefficients. As a result, using the induction for each term of the sum, and the fact that  $\sum_{l_i} \lambda^{l_2, \dots, l_i} \leq 1$ , we obtain
$$
\begin{array}{l} \left\| \text {A p p r o x O r a c l e} _ {\delta , \xi , \mathcal {O} _ {f}} (i - 1, j, \boldsymbol {G}) - G (i - 1, j, \boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {i - 1}) \right\| \\ \leq \left(1 + \frac {2}{\sigma_ {\text {m i n}}} \left(k _ {i + 1} + \dots + k _ {p}\right) + 2 (p - i)\right) \xi + \left(\frac {2 k _ {i}}{\sigma_ {\text {m i n}}} + 2\right) \xi , \\ \end{array}
$$
which completes the induction. Noting that  $k_{i + 1} + \ldots + k_p \leq k_1 + \ldots + k_p \leq d$  and  $p - i \leq d - 1$  ends the proof.
Next, we show that the outputs of Algorithm 3 provide approximate separation hyperplanes for the first  $i$  coordinates  $(\pmb{x}_1,\dots ,\pmb{x}_i)$ .
Lemma A.3. Fix  $\delta, \xi \in (0,1)$ ,  $1 \leq j \leq i \leq p$  and an oracle  $O_S = (O_{S,1},\dots,O_{S,p}): \mathcal{C}_d \to \mathbb{R}^d$  for accuracy  $\epsilon > 0$ . Suppose that  $O_S$  takes values in the unit ball  $B_d(0,1)$ . For any  $s \in [i]$  let  $\mathcal{P}_{l_s}^{(s)} \in \mathcal{I}_{k_s}$  represent a bounded polyhedron with VolumetricCenter  $(\mathcal{P}_{l_s}^{(s)}) \in \mathcal{C}_{k_s}$ . Denote  $\boldsymbol{x}_r = \boldsymbol{c}(\mathcal{P}_{l_r}^{(r)})$  for  $r \in [i]$ . Suppose that when running ApproxOracle $_{\delta,\xi,O_S}(i,i,\mathcal{P}_{l_1}^{(1)},\dots,\mathcal{P}_{l_i}^{(i)})$ , no successful vector was queried. Then, any vector  $\boldsymbol{x}^\star = (\boldsymbol{x}_1^\star, \dots, \boldsymbol{x}_p^\star) \in \mathcal{C}_d$  such that  $B_d(\boldsymbol{x}^\star, \epsilon)$  is contained in the successful set satisfies
$$
\sum_ {r \in [ i ]} \operatorname {A p p r o x O r a c l e} _ {\delta , \xi , O _ {S}} (i, r, \mathcal {P} _ {l _ {1}} ^ {(1)}, \dots , \mathcal {P} _ {l _ {i}} ^ {(i)}) ^ {\top} (\boldsymbol {x} _ {r} ^ {\star} - \boldsymbol {x} _ {r}) \geq \epsilon - \frac {8 d ^ {5 / 2}}{\sigma_ {m i n}} \xi - d \delta .
$$
Proof. For  $i \leq r \leq p$  and  $j \leq r$ , we use the notation
$$
\boldsymbol {g} _ {j} ^ {l _ {i + 1}, \dots , l _ {r}} = \operatorname {A p p r o x O r a c l e} _ {\delta , \xi , O _ {S}} (r, j, \mathcal {P} _ {l _ {1}} ^ {(1)}, \dots , \mathcal {P} _ {l _ {r}} ^ {(r)}).
$$
Using Lemma A.2, we always have for  $j \in [r]$ ,
$$
\left\| \boldsymbol {g} _ {j} ^ {l _ {i + 1}, \dots , l _ {r}} - G (r, j, \boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {i}, \boldsymbol {c} ^ {l _ {i + 1}}, \dots , \boldsymbol {c} ^ {l _ {i + 1}, \dots , l _ {r}}) \right\| \leq \frac {4 d}{\sigma_ {\min }} \xi . \tag {2}
$$
Also, observe that by Lemma A.1 the recursive outputs of ApproxOracle always have norm bounded by one.
Next, let  $\mathcal{T}^{l_{i+1},\dots,l_{r-1}}$  be the set of indices corresponding to coordinates of  $\lambda^{l_{i+1},\dots,l_{r-1}}$  for which the procedure ApproxOracle did not call for a level- $r$  computation. These correspond to 1. constraints from the initial cube  $\mathcal{P}_0$ , or 2. cases when the volumetric center was out of the unit cube (1.6-7 of Algorithm 1) and as a result, the index of the added constraint was  $-1$  instead of the current iteration index  $t$ . Similarly as above, for any  $t \in \mathcal{T}^{l_{i+1},\dots,l_{r-1}}$ , we denote by  $\pmb{g}_r^{l_{i+1},\dots,l_{r-1},t}$  the corresponding
vector  $\mathbf{a}_t$ . We recall that by construction, this vector is of the form  $\pm \mathbf{e}_j$  for some  $j \in [k_r]$ . Then, from Lemma 4.1, since the responses of the oracle always have norm bounded by one, for all  $\mathbf{y}_r \in \mathcal{C}_{k_r}$ ,
$$
\sum_ {l _ {r} \in \mathcal {S} ^ {l _ {i + 1}, \dots , l _ {r - 1}} \cup \mathcal {T} ^ {l _ {i + 1}, \dots , l _ {r - 1}}} \lambda^ {l _ {i + 1}, \dots , l _ {r}} \left(\boldsymbol {g} _ {r} ^ {l _ {i + 1}, \dots , l _ {r}}\right) ^ {\top} \left(\boldsymbol {y} _ {r} - \boldsymbol {c} ^ {l _ {i + 1}, \dots , l _ {r}}\right) \leq \delta . \tag {3}
$$
For conciseness, we use the shorthand  $(\mathcal{S} \cup \mathcal{T})^{l_{i+1},\dots,l_{r-1}} := \mathcal{S}^{l_{i+1},\dots,l_{r-1}} \cup \mathcal{T}^{l_{i+1},\dots,l_{r-1}}$ , which contains all indices from coordinates of  $\lambda^{l_{i+1},\dots,l_{r-1}}$ . In particular,
$$
\sum_ {l _ {r} \in (\mathcal {S} \cup \mathcal {T}) ^ {l _ {i + 1}, \dots , l _ {r - 1}}} \lambda^ {l _ {i + 1}, \dots , l _ {r}} = 1. \tag {4}
$$
We now proceed to estimate the precision of the vectors  $G(i,j,\pmb{x}_1,\dots,\pmb{x}_i)$  as approximate separation hyperplanes for coordinates  $(\pmb{x}_1,\dots,\pmb{x}_i)$ . Let  $\pmb{x}^{\star}\in \mathcal{C}_d$  such that  $B_{d}(\pmb{x}^{\star},\epsilon)$  is within the successful set. Then, for any choice of  $l_{i + 1}\in S,\ldots ,l_p\in S^{l_{i + 1},\ldots ,l_{p - 1}}$ , since we did not query a successful vector, we have for all  $z\in B_d(\pmb{x}^\star ,\epsilon)$
$$
O _ {S} (\pmb {x} _ {1}, \dots , \pmb {x} _ {i}, \pmb {c} ^ {l _ {i + 1}}, \dots , \pmb {c} ^ {l _ {i + 1}, \dots , l _ {p}}) ^ {\top} (\pmb {z} - (\pmb {x} _ {1}, \dots , \pmb {x} _ {i}, \pmb {c} ^ {l _ {i + 1}}, \dots , \pmb {c} ^ {l _ {i + 1}, \dots , l _ {p}})) \geq 0.
$$
As a result, because the responses from  $O_S$  have unit norm,
$$
O _ {S} \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {i}, \boldsymbol {c} ^ {l _ {i + 1}}, \dots , \boldsymbol {c} ^ {l _ {i + 1}, \dots , l _ {p}}\right) ^ {\top} \left(\boldsymbol {x} ^ {\star} - \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {i}, \boldsymbol {c} ^ {l _ {i + 1}}, \dots , \boldsymbol {c} ^ {l _ {i + 1}, \dots , l _ {p}}\right)\right) \geq \epsilon . \tag {5}
$$