SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Moise Blanchard
Operations Research Center MIT
moiseb@mit.edu
Junhui Zhang
junhuiz@mit.edu
Patrick Jaillet
jailllet@mit.edu
We propose a family of recursive cutting-plane algorithms to solve feasibility problems with constrained memory, which can also be used for first-order convex optimization. Precisely, in order to find a point within a ball of radius  $\epsilon$  with a separation oracle in dimension  $d$  —or to minimize 1-Lipschitz convex functions to accuracy  $\epsilon$  over the unit ball—our algorithms use  $\mathcal{O}\left(\frac{d^2}{p} \ln \frac{1}{\epsilon}\right)$  bits of memory, and make  $\mathcal{O}\left((C\frac{d}{p} \ln \frac{1}{\epsilon})^p\right)$  oracle calls, for some universal constant  $C \geq 1$ . The family is parametrized by  $p \in [d]$  and provides an oracle-complexity/memory trade-off in the sub-polynomial regime  $\ln \frac{1}{\epsilon} \gg \ln d$ . While several works gave lower-bound trade-offs (impossibility results) [31, 5]—we explicit here their dependence with  $\ln \frac{1}{\epsilon}$ , showing that these also hold in any sub-polynomial regime—to the best of our knowledge this is the first class of algorithms that provides a positive trade-off between gradient descent and cutting-plane methods in any regime with  $\epsilon \leq 1/\sqrt{d}$ . The algorithms divide the  $d$  variables into  $p$  blocks and optimize over blocks sequentially, with approximate separation vectors constructed using a variant of Vaidya's method. In the regime  $\epsilon \leq d^{-\Omega(d)}$ , our algorithm with  $p = d$  achieves the information-theoretic optimal memory usage and improves the oracle-complexity of gradient descent.
Optimization algorithms are ubiquitous in machine learning, from solving simple regressions to training neural networks. Their essential roles have motivated numerous studies on their efficiencies, which are usually analyzed through the lens of oracle-complexity: given an oracle (such as function value, or subgradient oracle), how many calls to the oracle are needed for an algorithm to output an approximate optimal solution? [34]. However, ever-growing problem sizes have shown an inadequacy in considering only the oracle-complexity, and have motivated the study of the trade-off between oracle-complexity and other resources such as memory [52, 31, 5] and communication[25, 40, 42, 45, 33, 53, 51, 50].
In this work, we study the oracle-complexity/memory trade-off for first-order non-smooth convex optimization, and the closely related feasibility problem, with a focus on developing memory efficient
(deterministic) algorithms. Since [52] formally posed as open problems, the question of characterizing this trade-off, there have been exciting results showing what is impossible: for convex optimization in  $\mathbb{R}^d$ , [31] shows that any randomized algorithm with  $d^{1.25 - \delta}$  bits of memory needs at least  $\tilde{\Omega}(d^{1 + 4\delta / 3})$  queries, and this has later been improved for deterministic algorithms to  $d^{1 - \delta}$  bits of memory or  $\tilde{\Omega}(d^{1 + \delta / 3})$  queries by [5]; in addition [5] shows that for the feasibility problem with a separation oracle, any algorithm which uses  $d^{2 - \delta}$  bits of memory needs at least  $\tilde{\Omega}(d^{1 + \delta})$  queries.
Despite these recent results on the lower bounds, all known first-order convex optimization algorithms that output an  $\epsilon$ -suboptimal point fall into two categories: those that have quadratic memory in the dimension  $d$  but can potentially achieve the optimal  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  query complexity, as represented by the center-of-mass method, and those that have  $\mathcal{O}\left(\frac{1}{\epsilon^2}\right)$  query complexity but only need the optimal  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  bits of memory, as represented by the classical gradient descent [52]. In addition, the above-mentioned memory bounds apply only between queries, and in particular the center-of-mass method [52] is allowed to use infinite memory during computations.
We propose a family of memory-constrained algorithms for the stronger feasibility problem in which one aims to find a point within a set  $Q$  containing a ball of radius  $\epsilon$ , with access to a separation oracle. In particular, this can be used for convex optimization since the subgradient information provides a separation vector. Our algorithms use  $\mathcal{O}\left(\frac{d^2}{p} \ln \frac{1}{\epsilon}\right)$  bits of memory (including during computations) and  $\mathcal{O}\left((C \frac{d}{p} \ln \frac{1}{\epsilon})^p\right)$  queries for some universal constant  $C \geq 1$ , and a parameter  $p \in [d]$  that can be chosen by the user. Intuitively, in the context of convex optimization, the algorithms are based on the idea that for any function  $f(x, y)$  convex in the pair  $(x, y)$ , the partial minimum  $\min_y f(x, y)$  as a function of  $x$  is still convex and, using a variant of Vaidya's method proposed in [27], our algorithm can approximate subgradients for that function  $\min_y f(x, y)$ , thereby turning an optimization problem with variables  $(x, y)$  to one with just  $x$ . This idea, applied recursively with the variables divided into  $p$  blocks, gives our family of algorithms and the above-mentioned memory and query complexity. The main algorithmic contribution is in how we design the recursive dimension reduction procedure: a technical step of the design and analysis is to ensure that the necessary precision for recursive computations can be achieved using low memory. Last, our algorithms account for memory usage throughout computations, as opposed to simply between calls to the gradient oracle, which was the traditional approach in the literature.
When  $p = 1$ , our algorithm is a memory-constrained version of Vaidya's method [48, 27], and improves over the center-of-mass [52] method by a factor of  $\ln \frac{1}{\epsilon}$  in terms of memory while having optimal oracle-complexity. The improvements provided by our algorithms are more significant in regimes when  $\epsilon$  is very small in the dimension  $d$ : increasing the parameter  $p$  can further reduce the memory usage of Vaidya's method  $(p = 1)$  by a factor  $\ln \frac{1}{\epsilon} / \ln d$ , while still improving over the oracle-complexity of gradient descent. In particular, in a regime  $\ln \frac{1}{\epsilon} = \mathrm{poly}(\ln d)$ , these memory improvements are only in terms of  $\ln d$  factors. However, in sub-polynomial regimes with potentially  $\ln \frac{1}{\epsilon} = d^{c}$  for some constant  $c > 0$ , these provide polynomial improvements to the memory of standard cutting-plane methods.
As a summary, this paper makes the following contributions.
- Our class of algorithms provides a trade-off between memory-usage and oracle-complexity whenever  $\ln \frac{1}{\epsilon} \gg \ln d$ . Further, taking  $p = 1$  improves the memory-usage from center-of-mass [52] by a factor  $\ln \frac{1}{\epsilon}$ , while preserving the optimal oracle-complexity.
- For  $\ln \frac{1}{\epsilon} \geq \Omega(d \ln d)$ , our algorithm with  $p = d$  is the first known algorithm that outperforms gradient descent in terms of the oracle-complexity, but still maintains the optimal  $\mathcal{O}(d \ln \frac{1}{\epsilon})$  memory usage.
- We show how to obtain a  $\ln \frac{1}{\epsilon}$  dependence in the known lower-bound trade-offs [31, 5], confirming that the oracle-complexity/memory trade-off is necessary for any regime  $\epsilon \lesssim \frac{1}{\sqrt{d}}$ .
In this section, we precise the formal setup for our results. We follow the framework introduced in [52], to define the memory constraint on algorithms with access to an oracle  $\mathcal{O}: S \to \mathcal{R}$  which
takes as input a query  $q \in S$  and outputs a response  $\mathcal{O}(q) \in \mathcal{R}$ . Here, the algorithm is constrained to update an internal  $M$ -bit memory between queries to the oracle.
Definition 2.1 ( $M$ -bit memory-constrained algorithm [52, 31, 5]). Let  $\mathcal{O}: S \to \mathcal{R}$  be an oracle. An  $M$ -bit memory-constrained algorithm is specified by a query function  $\psi_{query}: \{0,1\}^M \to S$  and an update function  $\psi_{update}: \{0,1\}^M \times S \times \mathcal{R} \to \{0,1\}^M$ . The algorithm starts with the memory state  $\mathrm{Memory}_0 = 0^M$  and iteratively makes queries to the oracle. At iteration  $t$ , it makes the query  $q_t = \psi_{query}(\mathrm{Memory}_{t-1})$  to the oracle, receives the response  $r_t = \mathcal{O}(q_t)$  then updates its memory  $\mathrm{Memory}_t = \psi_{update}(\mathrm{Memory}_{t-1}, q_t, r_t)$ .
The algorithm can stop at any iteration and the last query is its final output. Importantly, this model does not enforce constraints on the memory usage during the computation of  $\psi_{update}$  and  $\psi_{query}$ . This is ensured in the stronger notion of a memory-constrained algorithm with computations. These are precisely algorithms that have constrained memory including for computations, with the only specificity that they need a decoder function  $\phi$  to make queries to the oracle from their bit memory, and a discretization function  $\psi$  to write a discretized response into the algorithm's memory.
Definition 2.2 ( $M$ -bit memory-constrained algorithm with computations). Let  $\mathcal{O}: S \to \mathcal{R}$  be an oracle. We suppose that we are given a decoding function  $\phi: \{0,1\}^{\star} \to S$  and a discretization function  $\psi: \mathcal{R} \times \mathbb{N} \to \{0,1\}^{\star}$  such that  $\psi(r,n) \in \{0,1\}^n$  for all  $r \in \mathcal{R}$ . An  $M$ -bit memory-constrained algorithm with computations is only allowed to use an  $M$ -bit memory in  $\{0,1\}^M$  even during computations. The algorithm has three special memory placements  $Q, N, R$ . Say the contents of  $Q$  and  $N$  are  $q$  and  $n$  respectively. To make a query,  $R$  must contain at least  $n$  bits. The algorithm submits  $q$  to the encoder which then submits the query  $\phi(q)$  to the oracle. If  $r = \mathcal{O}(\phi(q))$  is the oracle response, the discretization function then writes  $\psi(r,n)$  in the placement  $R$ .
Feasibility problem. In this problem, the goal is to find a point  $\pmb{x} \in Q$ , where  $Q \subset \mathcal{C}_d \coloneqq [-1, 1]^d$  is a convex set. We choose the cube  $[-1, 1]^d$  as prior bound for convenience in our later algorithms, but the choice of norm for this prior ball can be arbitrary and does not affect our results. The algorithm has access to a separation oracle  $O_S: \mathcal{C}_d \to \{\text{Success}\} \cup \mathbb{R}^d$ , such that for a query  $\pmb{x} \in \mathbb{R}^d$  either returns Success if  $\pmb{x} \in Q$ , or a separating hyperplane  $\pmb{g} \in \mathbb{R}^d$ , i.e., such that  $g^\top \pmb{x} < g^\top \pmb{x}'$  for any  $\pmb{x}' \in Q$ . We suppose that the separating hyperplanes are normalized,  $\| \pmb{g} \|_2 = 1$ . An algorithm solves the feasibility problem with accuracy  $\epsilon$  if the algorithm is successful for any feasibility problem such that  $Q$  contains an  $\epsilon$ -ball  $B_d(\pmb{x}^\star, \epsilon)$  for  $\pmb{x}^\star \in \mathcal{C}_d$ .
As an important remark, this formulation asks that the separation oracle is consistent over time: when queried at the exact same point  $\pmb{x}$ , the oracle always returns the same separation vector. In this context, we can use the natural decoding function  $\phi$  which takes as input  $d$  sequences of bits and outputs the vector with coordinates given by the sequences interpreted in base 2. Similarly, the natural discretization function  $\psi$  takes as input the separation hyperplane  $\pmb{g}$  and outputs a discretized version up to the desired accuracy. From now, we can omit these implementation details and consider that the algorithm can query the oracle for discretized queries  $\pmb{x}$ , up to specified rounding errors.
Remark 2.1. An algorithm for the feasibility problem with accuracy  $\epsilon/(2\sqrt{d})$  can be used for first-order convex optimization. Suppose one aims to minimize a 1-Lipschitz convex function  $f$  over the unit ball, and output an  $\epsilon$ -suboptimal solution, i.e., find a point  $\pmb{x}$  such that  $f(\pmb{x}) \leq \min_{\pmb{y} \in B_d(0,1)} f(\pmb{y}) + \epsilon$ . A separation oracle for  $Q = \{\pmb{x}: f(\pmb{x}) \leq \min_{\pmb{y} \in B_d(0,1)} f(\pmb{y}) + \epsilon\}$  is given at a query  $\pmb{x}$  by the subgradient information from the first-order oracle:  $-\frac{\partial f(\pmb{x})}{\|\partial f(\pmb{x})\|}$ . Its computation can also be carried out memory-efficiently up to rounding errors since if  $\|\partial f(\pmb{x})\| \leq \epsilon/(2\sqrt{d})$ , the algorithm can return  $\pmb{x}$  and already has the guarantee that  $\pmb{x}$  is an  $\epsilon$ -suboptimal solution ( $\mathcal{C}_d$  has diameter  $2\sqrt{d}$ ). Notice that because  $f$  is 1-Lipschitz,  $Q$  contains a ball of radius  $\epsilon/(2\sqrt{d})$  (the factor  $1/(2\sqrt{d})$  is due to potential boundary issues). Hence, it suffices to run the algorithm for the feasibility problem while keeping in memory the queried point with best function value.
2.1 Known trade-offs between oracle-complexity and memory
Known lower-bound trade-offs. All known lower bounds apply to the more general class of memory-constrained algorithms without computational constraints given in Definition 2.1. [34] first showed that  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  queries are needed for solving convex optimization to ensure that one finds an  $\epsilon$ -suboptimal solution. Further,  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  bits of memory are needed even just to output a solution in
the unit ball with  $\epsilon$  accuracy [52]. These historical lower bounds apply in particular to the feasibility problem and are represented in the pictures of Fig. 1 as the dashed pink region.
More recently, [31] showed that achieving both optimal oracle-complexity and optimal memory is impossible for convex optimization. They show that a possibly randomized algorithm with  $d^{1.25 - \delta}$  bits of memory makes at least  $\tilde{\Omega}(d^{1 + 4\delta / 3})$  queries. This result was extended for deterministic algorithms in [5] which shows that a deterministic algorithm with  $d^{1 - \delta}$  bits of memory makes  $\tilde{\Omega}(d^{1 + \delta / 3})$  queries. For the feasibility problem, they give an improved trade-off: any deterministic algorithm with  $d^{2 - \delta}$  bits of memory makes  $\tilde{\Omega}(d^{1 + \delta})$  queries. These trade-offs are represented in the left picture of Fig. 1 as the pink, red, and purple solid region, respectively. Using a clever and more careful analysis,[11] showed that similar lower bounds can be carried out for deterministic algorithms as well.
Known upper-bound trade-offs. Prior to this work, to the best of our knowledge only two algorithms were known in the oracle-complexity/memory landscape. First, cutting-plane algorithms achieve the optimal oracle-complexity  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  but use quadratic memory. The memory-constrained (MC) center-of-mass method analyzed in [52] uses in particular  $\mathcal{O}(d^2\ln^2\frac{1}{\epsilon})$  memory. Instead, if one uses Vaidya's method which only needs to store  $\mathcal{O}(d)$  cuts instead of  $\mathcal{O}(d\ln \frac{1}{\epsilon})$ , we show that one can achieve  $\mathcal{O}(d^2\ln \frac{1}{\epsilon})$  memory. These algorithms only use the separation oracle and hence apply to both convex optimization and the feasibility problem. On the other hand, the memory-constrained gradient descent for convex optimization [52] uses the optimal  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  memory but makes  $\mathcal{O}(\frac{1}{\epsilon^2})$  iterations. While the analysis in [52] is only carried for convex optimization, we can give a modified proof showing that gradient descent can also be used for the feasibility problem.
2.2 Other related works
Vaidya's method [48, 38, 1, 2] and the variant [27] that we use in our algorithms, belong to the family of cutting-plane methods. Perhaps the simplest example of an algorithm in this family is the center-of-mass method, which achieves the optimal  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  oracle-complexity but is computationally intractable, and the only known random walk-based implementation [4] has computational complexity  $\mathcal{O}(d^7\ln \frac{1}{\epsilon})$ . Another example is the ellipsoid method, which has suboptimal  $\mathcal{O}(d^2\ln \frac{1}{\epsilon})$  query complexity, but has an improved computational complexity  $\mathcal{O}(d^4\ln \frac{1}{\epsilon})$ . [8] pointed out that Vaidya's method achieves the best of both worlds by sharing the  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  optimal query complexity of the center-of-mass, and achieving a computational complexity of  $\mathcal{O}(d^{1 + \omega}\ln \frac{1}{\epsilon})^1$ . In a major breakthrough, this computational complexity was improved to  $\mathcal{O}(d^3\ln^3\frac{1}{\epsilon})$  in [27], then to  $\mathcal{O}(d^3\ln \frac{1}{\epsilon})$  in [22]. We refer to [8, 27, 22] for more detailed comparisons of these algorithms.
Another popular convex optimization algorithm that requires quadratic memory is the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm [43, 7, 20, 21], which stores an approximated inverse Hessian matrix as gradient preconditioner. Several works aimed to reduce the memory usage of BFGS; in particular, the limited memory BFGS (L-BFGS) stores a few vectors instead of the entire approximated inverse Hessian matrix [37, 30]. However, it is still an open question whether even the original BFGS converges for non-smooth convex objectives [29].
Lying at the other extreme of the oracle-complexity/memory trade-off is gradient descent, which achieves the optimal memory usage but requires significantly more queries than center-of-mass or Vaidya's method in the regime  $\epsilon \lesssim \frac{1}{\sqrt{d}}$ . There is a rich literature of works aiming to speed up gradient descent, such as the optimized gradient method [17, 16], Nesterov's Acceleration [35], the triple momentum method [41], geometric descent [9], quadratic averaging [18], the information-theoretic exact method [46], or Big-Step-Little-Step method [23]. Interested readers can find a comprehensive survey on acceleration methods in [12]. However, these acceleration methods usually require additional smoothness or strong convexity assumptions (or both) on the objective function, due to the known  $\Omega\left(\frac{1}{\epsilon^2}\right)$  query lower bound in the large-scale regime  $\epsilon \gtrsim \frac{1}{\sqrt{d}}$  for any first order method where the query points lie in the span of the subgradients of previous query points [36].
Besides accelerating gradient descent, researchers have investigated more efficient ways to leverage subgradients obtained in previous iterations. Of interest are bundle methods [3, 24, 28], that have
${}^{1}\omega < 2.373$  is the exponent of matrix multiplication
found a wide range of applications [47, 26]. In their simplest form, they minimize the sum of the maximum of linear lower bounds constructed using past oracle queries, and a regularization term penalizing the distance from the current iteration variable. Although the theoretical convergence rate of the bundle method is the same as that of gradient descent, in practice, bundle methods can benefit from previous information and substantially outperform gradient descent [3].
Our works are focused on high-accuracy regimes, when the accuracy  $\epsilon$  is sub-polynomial. We note that for their lower-bound result on randomized algorithms, [11] also required sub-polynomial accuracies, which raises the question whether this is a general phenomenon for the study of memory-constrained algorithms in convex optimization. This also relates our work to the study of low-dimensional problems—or even constant dimension—which has been investigated in the literature [49, 10].
Last, the increasing size of optimization problems has also motivated the development of communication-efficient optimization algorithms in distributed settings such as [25, 40, 42, 45, 33, 53, 51, 50]. Moreover, recent works have explored the trade-off between sample complexity and memory/communication complexity for learning problems under the streaming model, with notable contributions including [6, 13, 14, 39, 44, 32].
We first check that the memory-constrained gradient descent method solves feasibility problems. This was known for convex optimization [52] and the same algorithm with a modified proof gives the following result. For completeness, the proof is given in Appendix D.
Proposition 3.1. The memory-constrained gradient descent algorithm solves the feasibility problem with accuracy  $\epsilon \leq \frac{1}{\sqrt{d}}$  using  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  bits of memory and  $\mathcal{O}(\frac{1}{\epsilon^2})$  separation oracle calls.
Our main contribution is a class of algorithms based on Vaidya's cutting-plane method that provide a query-complexity / memory tradeoff for optimization in  $\mathbb{R}^d$ . More precisely, we show the following, where  $\omega < 2.373$  is the exponent of matrix multiplication, such that multiplying two  $n \times n$  matrices runs in  $\mathcal{O}(n^\omega)$  time.
Theorem 3.2. For any  $1 \leq p \leq d$ , there is a deterministic first-order algorithm that solves the feasibility problem in dimension  $d$  for accuracy  $\epsilon \leq \frac{1}{\sqrt{d}}$ , using  $\mathcal{O}\left(\frac{d^2}{p} \ln \frac{1}{\epsilon}\right)$  bits of memory (including during computations), with  $\mathcal{O}\left((C \frac{d}{p} \ln \frac{1}{\epsilon})^p\right)$  calls to the separation oracle, and computational complexity  $\mathcal{O}\left((C \left(\frac{d}{p}\right)^{1 + \omega} \ln \frac{1}{\epsilon}\right)^p$ , where  $C \geq 1$  is a universal constant.
For simplicity, in Section 4, we describe algorithms that achieve this trade-off without computation concerns (Definition 2.1), which already provide the main elements of our method. The proof of oracle-complexity and memory usage is given in Appendix A. In Appendix B, we consider computational constraints and give corresponding algorithms using the cutting-plane method of [27].
To better understand the implications of Theorem 3.2, it is useful to compare the provided class of algorithms to the two algorithms known in the oracle-complexity/memory tradeoff landscape: the memory-constrained center-of-mass method and the memory-constrained gradient descent [52].
For  $p = 1$ , our resulting procedure, which is essentially a memory-constrained Vaidya's algorithm, has optimal oracle-complexity  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  and uses  $\mathcal{O}(d^2\ln \frac{1}{\epsilon})$  bits of memory. This improves by a  $\ln \frac{1}{\epsilon}$  factor the memory usage of the center-of-mass-based algorithm provided in [52], which used  $\mathcal{O}(d^2\ln^2\frac{1}{\epsilon})$  memory and had the same optimal oracle-complexity.
Next, we recall that the memory-constrained gradient descent method used the optimal number  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  bits of memory (including for computations), and a sub-optimal  $\mathcal{O}(\frac{1}{\epsilon^2})$  oracle-complexity. While the memory of our algorithms decreases with  $p$ , their oracle-complexity is exponential in  $p$ . This significantly restricts the values of  $p$  for which the oracle-complexity is improved over that of gradient descent. The range of application of Theorem 3.2 is given in the next result, where  $\vee$  and  $\wedge$  represent maximum and minimum respectively.
Corollary 3.1. The algorithms given in Theorem 3.2 effectively provide a tradeoff for  $p \leq O\left(\frac{\ln \frac{1}{\epsilon}}{\ln d} \wedge d\right)$ . Precisely, this provides a tradeoff between
using  $\mathcal{O}(d^2\ln \frac{1}{\epsilon})$  memory with optimal  $\mathcal{O}(d\ln \frac{1}{\epsilon})$  oracle-complexity, and