SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: Trade-offs between available memory and first-order oracle-complexity for the feasibility problem over the unit ball. MC=Memory-constrained. GD=Gradient Descent. The left picture corresponds to the regime  $\epsilon \gg d^{-\Omega(d)}$  and  $\epsilon \leq 1/poly(d)$  and the right picture represents the regime  $\epsilon \leq d^{-\mathcal{O}(d)}$ . For both figures, the dashed pink "L" (resp. green inverted "L") region corresponds to historical lower (resp. upper) bounds for randomized algorithms. The solid pink (resp. red) lower bound tradeoff is due to [31] (resp. [5]) for randomized algorithms (resp. deterministic algorithms). The purple region is a lower bound tradeoff for the feasibility problem for accuracy  $\epsilon$  and deterministic algorithms [5]. All these lower-bound trade-offs are represented with their  $\ln \frac{1}{\epsilon}$  dependence (Theorem 3.3). We use memory-constrained Vaidya's method to gain a factor  $\ln \frac{1}{\epsilon}$  in memory compared to memory-constrained center-of-mass [52], which gives the light green region, and a class of algorithms represented in dark green, that allows trading query-complexity for an extra  $\ln \frac{1}{\epsilon} / \ln d$  factor saved in memory (Theorem 3.2). The dark green dashed region in the left figure emphasizes that the area covered by our class of algorithms depends highly on the regime for the accuracy  $\epsilon$ : the resulting improvement in memory is more significant as  $\epsilon$  is smaller. In the regime when  $\epsilon \leq d^{-\mathcal{O}(d)}$  (right figure), our class of algorithms improves over the oracle-complexity of gradient descent while keeping the optimal memory  $\mathcal{O}(d \ln \frac{1}{\epsilon})$ .
using  $\mathcal{O}(d^2\ln d\vee d\ln \frac{1}{\epsilon})$  memory with  $\mathcal{O}\big(\frac{1}{\epsilon^2}\wedge (C\ln \frac{1}{\epsilon})^d\big)$  oracle-complexity.
Importantly, for  $\epsilon \leq \frac{1}{d^{\Omega(d)}}$ , taking  $p = d$  yields an algorithm that uses the optimal memory  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  and has an improved query complexity over gradient descent. In this regime of small (virtually constant) dimension, for the same memory usage, gradient descent has a query complexity that is polynomial in  $\epsilon$ ,  $\mathcal{O}\left(\frac{1}{\epsilon^2}\right)$ , while our algorithm has poly-logarithmic dependence in  $\epsilon$ ,  $\mathcal{O}_d\left(\ln^d\frac{1}{\epsilon}\right)$ , where  $\mathcal{O}_d$  hides an exponential constant in  $d$ . It remains open whether this  $\ln^d\frac{1}{\epsilon}$  dependence in the oracle-complexity is necessary. To the best of our knowledge, this is the first example of an algorithm that improves over gradient descent while keeping its optimal memory usage in any regime where  $\epsilon \leq \frac{1}{\sqrt{d}}$ . While this improvement holds only in the exponential regime  $\epsilon \leq \frac{1}{d^{\mathcal{O}(d)}}$ , Theorem 3.2 still provides a non-trivial trade-off whenever  $\ln \frac{1}{\epsilon} \gg \ln d$ , and improves over the known memory-constrained center-of-mass in the standard regime  $\epsilon \leq \frac{1}{\sqrt{d}}$  [52]. Fig. 1 depicts the trade-offs in the two regimes mentioned earlier.
Last, we note that the lower-bound trade-offs presented in [31, 5] do not show a dependence in the accuracy  $\epsilon$ . Especially in the regime when  $\ln \frac{1}{\epsilon} \gg \ln d$ , this yields sub-optimal lower bounds (in fact even in the regime  $\epsilon = 1 / \mathrm{poly}(d)$ ), our more careful analysis improves the lower bound on the memory by a  $\ln d$  factor. We show with simple arguments that one can extend their results to include a  $\ln \frac{1}{\epsilon}$  factor for both memory and query complexity. Fig. 1 presented these improved lower bounds.
Theorem 3.3. For  $\epsilon \leq 1/poly(d)$  and any  $\delta \in [0,1]$  (the notation  $\tilde{\Omega}$  hides  $\ln^{\mathcal{O}(1)}d$  factors),
1. any (randomized) algorithm guaranteed to minimize 1-Lipschitz convex functions over the unit ball with accuracy  $\epsilon$  uses  $d^{5/4-\delta} \ln \frac{1}{\epsilon}$  bits of memory or makes  $\tilde{\Omega}(d^{1+4\delta/3} \ln \frac{1}{\epsilon})$  queries,
2. any deterministic algorithm guaranteed to minimize 1-Lipschitz convex functions over the unit ball with accuracy  $\epsilon$  uses  $d^{2 - \delta}\ln \frac{1}{\epsilon}$  bits of memory or makes  $\tilde{\Omega} (d^{1 + \delta /3}\ln \frac{1}{\epsilon})$  queries,
3. any deterministic algorithm guaranteed to solve the feasibility problem over the unit ball with accuracy  $\epsilon$  uses  $d^{2 - \delta}\ln \frac{1}{\epsilon}$  bits of memory or makes  $\hat{\Omega} (d^{1 + \delta}\ln \frac{1}{\epsilon})$  queries.
The proof is given in Appendix C and the arguments therein could readily be used to exhibit the  $\ln \frac{1}{\epsilon}$  dependence of potential future works improving over these lower bounds trade-offs.
Sketch of proof. At a high level, [31, 5] use a barrier term  $\| A\pmb{x}\|_{\infty}$  where  $\pmb{A}$  has  $\Theta(d)$  rows: if an algorithm does not have enough memory,  $\pmb{A}$  cannot be fully stored which in turn incurs a sub-optimal oracle-complexity. To achieve a  $\ln \frac{1}{\epsilon}$  improvement in memory (Appendix C.1), we modify the sampling of rows of  $\pmb{A}$ , from uniform on vertices of the hypercube to uniform in an  $\epsilon$ -net. The proof can then be adapted accordingly. Last, one can improve the oracle-complexity by a  $\ln \frac{1}{\epsilon} / \ln d$  factor (Appendix C.2) using a standard rescaling argument [34].
4 Memory-constrained feasibility problem without computation
In this section, we present a class of algorithms that are memory-constrained according to Definition 2.1 and achieve the desired memory and oracle-complexity bounds. We emphasize that the memory constraint is only applied between calls to the oracle and as a result, the algorithm is allowed infinite computation memory and computation power between calls to the oracle.
We start by defining discretization functions that will be used in our algorithms. For  $\xi > 0$  and  $x \in [-1,1]$ , we pose  $\mathrm{Discretize}_1(x,\xi) = \mathrm{sign}(x) \cdot \xi \lfloor |x| / \xi \rfloor$ . Next, we define the discretization  $\mathrm{Discretize}_d$  for general dimensions  $d \geq 1$ . For any  $\pmb{x} \in C$  and  $\xi > 0$ ,
$$
\operatorname {D i s c r e t i z e} _ {d} (\boldsymbol {x}, \xi) = \left(\operatorname {D i s c r e t i z e} _ {1} \left(x _ {1}, \xi / \sqrt {d}\right), \dots , \operatorname {D i s c r e t i z e} _ {1} \left(x _ {d}, \xi / \sqrt {d}\right)\right).
$$
4.1 Memory-constrained Vaidya's method
Our algorithm recursively uses Vaidya's cutting-plane method [48] and subsequent works expanding on this method. We briefly describe the method. Given a polyhedron  $\mathcal{P} = \{\pmb{x} : \pmb{A}\pmb{x} \geq \pmb{b}\}$ , we define  $s_i(\pmb{x}) = \pmb{a}_i^\top \pmb{x} - b_i$  and  $\pmb{S}_x = \text{diag}(s_i(x), i \in [d])$ . We will also use the shorthand  $\pmb{A}_x = \pmb{S}_x^{-1}\pmb{A}$ . The volumetric barrier is defined as
$$
V _ {\pmb {A}, \pmb {b}} (\pmb {x}) = \frac {1}{2} \ln \det (\pmb {A} _ {x} ^ {\top} \pmb {A} _ {x}).
$$
At each step, Vaidya's method queries the volumetric center of the polyhedron, which is the point minimizing the volumetric barrier. For convenience, we denote by VolumetricCenter this function, i.e., for any  $\mathbf{A} \in \mathbb{R}^{m \times d}$  and  $\mathbf{b} \in \mathbb{R}^d$  defining a non-empty polyhedron  $\mathcal{P} = \{\mathbf{x} : \mathbf{A}\mathbf{x} \geq \mathbf{b}\}$ ,
$$
\operatorname {V o l u m e r i c C e n t e r} (\boldsymbol {A}, \boldsymbol {b}) = \arg \min  _ {\boldsymbol {x}: \boldsymbol {A} \boldsymbol {x} > \boldsymbol {b}} V _ {\boldsymbol {A}, \boldsymbol {b}} (\boldsymbol {x}).
$$
When the polyhedron is unbounded, we can for instance take VolometricCenter  $(A, b) = 0$ . Vaidya's method makes use of leverage scores for each constraint  $i$  of the polyhedron, defined as  $\sigma_{i} = (A_{x}H^{-1}A_{x}^{\top})_{i,i}$ , where  $H = A_{x}^{\top}A_{x}$ . We are now ready to define the update procedure for the polyhedron considered by Vaidya's volumetric method. We denote by  $\mathcal{P}_t$  the polyhedron stored in memory after making  $t$  queries. The method keeps in memory the constraints defining the current polyhedron and the iteration index  $k$  when these constraints were added, which will be necessary for our next procedures. Hence, the polyhedron will be stored in the form  $\mathcal{P}_t = \{(k_i, a_i, b_i), i \in [m]\}$ , and the associated constraints are given via  $\{x : Ax \geq b\}$  where  $A^{\top} = [a_1, \ldots, a_m]$  and  $b^{\top} = [b_1, \ldots, b_m]$ . By abuse of notation, we will write VolometricCenter  $(\mathcal{P})$  for the volumetric center of the polyhedron VolometricCenter  $(A, b)$  where  $A$  and  $b$  define the constraints stored in  $\mathcal{P}$ .
Initially, the polyhedron is simply  $\mathcal{C}_d$ , these constraints are given  $-1$  index for convenience, and they will not play a role in the next steps. At each iteration, if the constraint  $i \in [m]$  with minimum leverage score  $\sigma_i$  falls below a given threshold  $\sigma_{\min}$ , it is removed from the polyhedron. Otherwise, we query the volumetric center of the current polyhedron and add the separation hyperplane as a constraint to the polyhedron. We bound the number of iterations of the procedure by
$$
T (\delta , d) = \left\lceil c \cdot d \left(1. 4 \ln \frac {1}{\delta} + 2 \ln d + 2 \ln (1 + 1 / \sigma_ {m i n})\right)\right\rceil ,
$$
where  $\sigma_{min}$  and  $c$  are parameters that will be fixed shortly. Instead of making a call directly to the oracle  $O_S$ , we instead suppose that one has access to an oracle  $O:\mathcal{I}_d\to \mathbb{R}^d$  where  $\mathcal{I}_d = (\mathbb{Z}\times \mathbb{R}^{d + 1})^\star$  has exactly the shape of the memory storing the information from the polyhedron. This form of oracle is used in our recursive calls to Vaidya's method. For example, such an oracle can simply be  $O:\mathcal{P}\in \mathcal{I}_d\mapsto O_S(\mathrm{VolumetricCenter}(\mathcal{P}))$ . Last, in our recursive method, we will not assume that oracle responses are normalized. As a result, we specify that if the norm of the response is too small, we can stop the algorithm. We assume however that the oracle already returns discretized vectors, which will be ensured in the following procedures. The cutting-plane algorithm is formally described in Algorithm 1. With an appropriate choice of parameters, this procedure finds an approximate solution of feasibility problems. We base the constants from [2].
Input:  $O:\mathcal{I}_d\to \mathbb{R}^d$ $\delta ,\xi \in (0,1)$    
1 Let  $T_{max} = T(\delta ,d)$  and initialize  $\mathcal{P}_0\coloneqq \{(-1,\pmb {e}_i, - 1),(-1, - \pmb {e}_i, - 1),i\in [d]\}$    
2 for  $t = 0,\dots ,T_{max}$  do   
3 if  $\{\pmb {x}:A\pmb {x}\geq \pmb {b}\} = \emptyset$  then return  $\mathcal{P}_t$  .   
4 if  $\min_{i\in [m]}\sigma_i <   \sigma_{min}$  then   
5  $\begin{array}{rlr}{\mathcal{P}_{t + 1} = \mathcal{P}_t\setminus \{(k_j,a_j,b_j)\}} & \text{where} j\in \arg \min_{i\in [m]}\sigma_i \end{array}$    
6 else if  $\omega \coloneqq$  VolumetricCenter(  $\mathcal{P}_t$  )  $\notin \mathcal{C}_d$  then   
7  $\begin{array}{rl}{\mathcal{P}_{t + 1} = \mathcal{P}_t\cup \{(-1, - sign(\omega_j)e_j, - 1)\}} & \text{where} j\in [d]\text{has} |\omega_j| > 1 \end{array}$    
8 else   
9  $\begin{array}{r}\pmb {g} = O(\mathcal{P}_t)\text{and} b = \xi \left\lceil \frac{\pmb{g}^\top\pmb{\omega}}{\xi}\right\rceil ,\text{where}\pmb {\omega} = \text{VolumetricCenter} (\mathcal{P}_t)\\ \mathcal{P}_{t + 1} = \mathcal{P}_t\cup \{(t,\pmb {g},b)\} \\ \text{if}\| \pmb {g}\| \leq \delta \text{then return}\mathcal{P}_{t + 1}; \end{array}$    
10   
11
Algorithm 1: Memory-constrained Vaidya's volumetric method
Lemma 4.1. Fix  $\sigma_{min} = 0.04$  and  $c = \frac{1}{0.0014} \approx 715$ . Let  $\delta, \xi \in (0,1)$  and  $O: \mathcal{I}_d \to \mathbb{R}^d$ . Write  $\mathcal{P} = \{(k_i, a_i, b_i), i \in [m]\}$  as the output of Algorithm 1 run with  $O$ ,  $\delta$  and  $\xi$ . Then,
$$
\min_{\substack{\lambda_{i}\geq 0, i\in [m],\\ \sum_{i\in [m]}\lambda_{i} = 1}}\max_{\boldsymbol {y}\in \mathcal{C}_{d}}\sum_{i = 1}^{m}\lambda_{i}(\boldsymbol{a}_{i}^{\top}\boldsymbol {y} - b_{i}) = \max_{\boldsymbol {x}\in \mathcal{C}_{d}}\min_{i\in [m]}(\boldsymbol{a}_{i}^{\top}\boldsymbol {x} - b_{i})\leq \delta .
$$
From now, we use the parameters  $\sigma_{min} = 0.04$  and  $c = 1 / 0.0014$  as in Lemma 4.1. Since the memory of both Vaidya's method and center-of-mass consists primarily of the constraints, we recall an important feature of Vaidya's method that the number of constraints at any time is  $\mathcal{O}(d)$ .
Lemma 4.2 ([48, 1, 2]). At any time while running Algorithm 1, the number of constraints of the current polyhedron is at most  $\frac{d}{\sigma_{min}} + 1$ .
4.2 A recursive algorithm
We write  $\mathcal{C}_{m + n} = \mathcal{C}_m\times \mathcal{C}_n$  and aim to apply Vaidya's method to the first  $m$  coordinates. To do so, we need to approximate a separation oracle on these  $m$  coordinates only, which corresponds to giving separation hyperplanes with small values for the last  $n$  coordinates. This can be achieved using the following auxiliary linear program. For  $\mathcal{P}\in \mathcal{I}_n$ , we define
$$
\min_{\substack{\lambda_{i}\geq 0, i\in [m],\\ \sum_{i\in [m]}\lambda_{i} = 1}}\max_{\boldsymbol {y}\in \mathcal{C}_{n}}\sum_{i = 1}^{m}\lambda_{i}(\boldsymbol{a}_{i}^{\top}\boldsymbol {y} - b_{i}), \quad m = |\mathcal{P}| \qquad \qquad (\mathcal{P}_{aux}(\mathcal{P}))
$$
where as before,  $A$  and  $b$  define the constraints stored in  $\mathcal{P}$ . The procedure to obtain an approximate separation oracle on the first  $n$  coordinates  $C_n$  is given in Algorithm 2 and using Lemma 4.1 we can show that this procedure provides approximate separation vectors for the first  $n$  coordinates.
Input:  $\delta, \xi, O_x: \mathcal{I}_n \to \mathbb{R}^m$  and  $O_y: \mathcal{I}_n \to \mathbb{R}^n$
1 Run Algorithm 1 with  $\delta, \xi$  and  $O_y$  to obtain polyhedron  $\mathcal{P}^{\star}$
2 Solve  $\mathcal{P}_{aux}(\mathcal{P}^{\star})$  to get a solution  $\lambda^{\star}$
3 Store  $\pmb{k}^{\star} = (k_{i}, i \in [m])$  where  $m = |\mathcal{P}^{\star}|$ , and  $\lambda^{\star} \gets \text{Discretize}(\lambda^{\star}, \xi)$
4 Initialize  $\mathcal{P}_0\coloneqq \left\{(-1,\pmb {e}_i, - 1),(-1 - \pmb {e}_i, - 1),i\in [d]\right\}$  and  $\pmb {u} = \pmb {0}\in \mathbb{R}^{m}$
5 for  $t = 0,1,\ldots ,\max_{i}k_{i}$  do
6 if  $t = k_i^\star$  for some  $i\in [m]$  then 7  $\begin{array}{rl} & {\pmb {g}_x = O_x(\mathcal{P}_t)}\\ & {\pmb {u}\gets \mathrm{Discretize}_m(\pmb {u} + \lambda_i^\star \pmb {g}_x,\xi)}\\ & {\mathrm{Update~}\mathcal{P}_t\mathrm{~to~get~}\mathcal{P}_{t + 1}\mathrm{~as~in~Algorithm~}1} \end{array}$  8 9
10 end
11 return  $\pmb{u}$
Algorithm 2: ApproxSeparationVector $_{\delta, \xi}(O_x, O_y)$
The next step involves using this approximation recursively. We write  $d = \sum_{i=1}^{p} k_i$ , and interpret  $\mathcal{C}_d$  as  $\mathcal{C}_{k_1} \times \dots \times \mathcal{C}_{k_p}$ . In particular, for  $\boldsymbol{x} \in \mathcal{C}_d$ , we write  $\boldsymbol{x} = (\boldsymbol{x}_1, \ldots, \boldsymbol{x}_p)$  where  $\boldsymbol{x}_i \in \mathcal{C}_{k_i}$  for  $i \in [p]$ . Applying Algorithm 2 recursively, we can obtain an approximate separation oracle for the first  $i$  coordinates  $\mathcal{C}_{k_1} \times \dots \times \mathcal{C}_{k_i}$ . However, storing such separation vectors would be too memory-expensive, e.g., for  $i = p$ , that would correspond to storing the separation hyperplanes from the oracle  $O_S$  directly. Instead, given  $j \in [i]$ , Algorithm 3 recursively computes the  $x_j$  component of an approximate separation oracle for the first  $i$  variables  $(\boldsymbol{x}_1, \ldots, \boldsymbol{x}_i)$ , via the procedure ApproxOracle(i, j).
Input:  $\delta, \xi, 1 \leq j \leq i \leq p$ ,  $\mathcal{P}^{(r)} \in \mathcal{I}_{k_r}$  for  $r \in [i]$ ,  $O_S : \mathcal{C}_d \to \mathbb{R}^d$
1 if  $i = p$  then
$\pmb{x}_r = \mathrm{VolumetricCenter}(\pmb{A}_r, \pmb{b}_r)$  where  $(\pmb{A}_r, \pmb{b}_r)$  defines the constraints stored in  $\mathcal{P}^{(r)}$  for  $r \in [p]$ $(\pmb{g}_1, \dots, \pmb{g}_p) = O_S(\pmb{x}_1, \dots, \pmb{x}_p)$ $\pmb{\mathrm{return Discretize}}_{k_j}(\pmb{g}_j, \xi)$
5 end
6 Define  $O_{x}:\mathcal{I}_{k_{i + 1}}\to \mathbb{R}^{k_{j}}$  as ApproxOracle  $\delta ,\xi ,\mathcal{O}_f(i + 1,j,\mathcal{P}^{(1)},\dots ,\mathcal{P}^{(i)},\cdot)$
7 Define  $O_y:\mathcal{I}_{k_{i + 1}}\to \mathbb{R}^{k_{i + 1}}$  as ApproxOracle  $\delta ,\xi ,\mathcal{O}_f(i + 1,i + 1,\mathcal{P}^{(1)},\ldots ,\mathcal{P}^{(i)},\cdot)$
8 return ApproxSeparationVector $_{\delta, \xi}(O_x, O_y)$
Algorithm 3: ApproxOracle $_{\delta, \xi, O_S}(i, j, \mathcal{P}^{(1)}, \ldots, \mathcal{P}^{(i)})$
We can then use ApproxOracle $_{\delta, \xi, O_S}(1, 1, \cdot)$  to solve the original problem with the memory-constrained Vaidya's method. In Appendix A, we show that taking  $\delta = \frac{\epsilon}{4d}$  and  $\xi = \frac{\sigma_{min} \epsilon}{32d^{5/2}}$  achieves the desired oracle-complexity and memory usage. The final algorithm is given in Algorithm 4.
Input:  $\delta, \xi$ , and  $\mathcal{O}_S: \mathcal{C}_d \to \mathbb{R}^d$  a separation oracle
Check : Throughout the algorithm, if  ${O}_{S}$  returned Success to a query  $x$  ,return  $x$
1 Run Algorithm 1 with parameters  $\delta$  and  $\xi$  and oracle ApproxOracle $_{\delta, \xi, O_S}(1, 1, \cdot)$
Algorithm 4: Memory-constrained algorithm for convex optimization
A geometric illustration of the recursive step. In Figure 2, we give a 2-dimensional feasibility problem with target  $\pmb{p}^{*} = (p_{1}^{*}, p_{2}^{*})$  and two blocks (i.e.  $p = 2$ ) as an illustration of our recursive approach (Algorithm 2) to construct an approximate separating hyperplane for a "reduced" problem.
Suppose at a step of the Algorithm 4, the current value of the  $x_{1}$  coordinate is  $c$ . We aim to find an approximate separating hyperplane between  $x_{1} = p_{1}^{*}$  and  $x_{1} = c$ . Algorithm 2 first runs Algorithm 1 (i.e. the memory-constrained Vaidya) to find two separating hyperplanes (the two blue hyperplanes). Lemma 4.1 then guarantees the existence of a convex combination of the 2 blue hyperplanes – the
Figure 2: Intuition for the recursive procedure in Algorithm 4. Using the separation hyperplanes (blue) found by Algorithm 1, i.e., the memory-constrained Vaidya, it constructs an approximate separation hyperplane (red) between  $x_{1} = c$  and the target  $x_{1} = p_{1}^{*}$ .
red hyperplane- which is approximately parallel to the  $x_{2}$  -axis and thus can serve as an approximate separating hyperplane between  $x_{1} = p_{1}^{*}$  and  $x_{1} = c$
Sketch of proof. At the high level, the algorithm recursively runs Vaidya's method Algorithm 1 for each level of computation  $i \in [p]$ . Since each run of Algorithm 4 requires  $\mathcal{O}\left(\frac{d}{p} \ln \frac{1}{\epsilon}\right)$  queries, the total number of calls to the oracle, which is exponential in the number of levels, is  $\mathcal{O}(\mathcal{O}\left(\frac{d}{p} \ln \frac{1}{\epsilon}\right)^p)$ . As for the memory usage, the algorithm mainly needs to keep in memory the constraints defining the polyhedrons at each level  $i \in [p]$ . From Lemma 4.2, each polyhedron only requires  $\mathcal{O}\left(\frac{d}{p}\right)$  constraints that each require  $\mathcal{O}\left(\frac{d}{p} \ln \frac{1}{\epsilon}\right)$  bits of memory. Hence, the total memory needed is  $\mathcal{O}\left(\frac{d^2}{p} \ln \frac{1}{\epsilon}\right)$ . The main difficulty lies in showing that the algorithm is successful. To do so, we need to show that the precision in the successive approximated separation oracles Algorithm 2 is sufficient. To avoid an exponential dependence of the approximation error in  $p$ —which would be prohibitive for the memory usage of our method—each run of Vaidya's method Algorithm 1 is run for more iterations than the precision of the separation vectors would classically allow. To give intuition, if the separation oracle came from a convex optimization subgradient oracle for a function  $f$ , the iterates at a level  $i$  do not converge to the true "minimizer" of  $\min_{\boldsymbol{x}_i} f^{(i)}(\boldsymbol{x}_1, \ldots, \boldsymbol{x}_i)$ , where  $f^{(i)}(\cdot) = \min_{\boldsymbol{x}_{i+1}, \ldots, \boldsymbol{x}_p} f(\cdot, \boldsymbol{x}_{i+1}, \ldots, \boldsymbol{x}_p)$ , but instead converge to a close enough point while still providing meaningful approximate subgradients at the higher level  $i-1$  (in Algorithm 2).
To the best of our knowledge, this work is the first to provide some positive trade-off between oracle-complexity and memory-usage for convex optimization or the feasibility problem, as opposed to lower-bound impossibility results [31, 5]. Our trade-offs are more significant in a high accuracy regime: when  $\ln \frac{1}{\epsilon} \approx d^{c}$ , for  $c > 0$  our trade-offs are polynomial, while the improvements when  $\ln \frac{1}{\epsilon} = \mathrm{poly}(\ln d)$  are only in  $\ln d$  factors. A natural open direction [52] is whether there exist algorithms with polynomial trade-offs in that case. We also show that in the exponential regime  $\ln \frac{1}{\epsilon} \geq \Omega(d \ln d)$ , gradient descent is not Pareto-optimal. Instead, one can keep the optimal memory and decrease the dependence in  $\epsilon$  of the oracle-complexity from  $\frac{1}{\epsilon^2}$  to  $(\ln \frac{1}{\epsilon})^d$ . The question of whether the exponential dependence in  $d$  is necessary is left open. Last, our algorithms rely on the consistency of the oracle, which allows re-computations. While this is a classical assumption, gradient descent and classical cutting-plane methods do not need it; removing this assumption could be an interesting research direction (potentially, this could also yield stronger lower bounds).
This work was partly funded ONR grant N00014-18-1-2122, AFOSR grant FA9550-19-1-0263, and AFOSR grant FA9550-23-1-0182.