Title: Memory-Constrained Algorithms for Convex Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Convex optimization

feasibility problem

first-order methods

memory constraints

cutting planes

oracle complexity
Abstract: We propose a family of recursive cutting-plane algorithms to solve feasibility problems with constrained memory, which can also be used for first-order convex optimization. Precisely, in order to find a point within a ball of radius $\epsilon$ with a separation oracle in dimension $d$---or to minimize $1$-Lipschitz convex functions to accuracy $\epsilon$ over the unit ball---our algorithms use $\mathcal O(\frac{d^2}{p}\ln \frac{1}{\epsilon})$ bits of memory, and make $\mathcal O((C\frac{d}{p}\ln \frac{1}{\epsilon})^p)$ oracle calls. The family is parametrized by $p\in[d]$ and provides an oracle-complexity/memory trade-off in the sub-polynomial regime $\ln\frac{1}{\epsilon}\gg\ln d$. While several works gave lower-bound trade-offs (impossibility results)---we explicit here their dependence with $\ln\frac{1}{\epsilon}$, showing that these also hold in any sub-polynomial regime---to the best of our knowledge this is the first class of algorithms that provides a positive trade-off between gradient descent and cutting-plane methods in any regime with $\epsilon\leq 1/\sqrt d$. The algorithms divide the $d$ variables into $p$ blocks and optimize over blocks sequentially, with approximate separation vectors constructed using a variant of Vaidya's method. In the regime $\epsilon \leq d^{-\Omega(d)}$, our algorithm with $p=d$ achieves the information-theoretic optimal memory usage and improves the oracle-complexity of gradient descent.

Memory-Constrained Algorithms for Convex Optimization

Abstract

We propose a family of recursive cutting-plane algorithms to solve feasibility problems with constrained memory, which can also be used for first-order convex optimization. Precisely, in order to find a point within a ball of radius ϵ with a separation oracle in dimension d -or to minimize 1 -Lipschitz convex functions to accuracy ϵ over the unit ball-our algorithms use O ( d 2 p ln 1 ϵ ) bits of memory, and make O (( C d p ln 1 ϵ ) p ) oracle calls, for some universal constant C ≥ 1 . The family is parametrized by p ∈ [ d ] and provides an oracle-complexity/memory trade-off in the sub-polynomial regime ln 1 ϵ ≫ ln d . While several works gave lower-bound trade-offs (impossibility results) [31, 5]-we explicit here their dependence with ln 1 ϵ , showing that these also hold in any sub-polynomial regime-to the best of our knowledge this is the first class of algorithms that provides a positive trade-off between gradient descent and cutting-plane methods in any regime with ϵ ≤ 1 / √ d . The algorithms divide the d variables into p blocks and optimize over blocks sequentially, with approximate separation vectors constructed using a variant of Vaidya's method. In the regime ϵ ≤ d -Ω( d ) , our algorithm with p = d achieves the information-theoretic optimal memory usage and improves the oracle-complexity of gradient descent.

1 Introduction

Optimization algorithms are ubiquitous in machine learning, from solving simple regressions to training neural networks. Their essential roles have motivated numerous studies on their efficiencies, which are usually analyzed through the lens of oracle-complexity: given an oracle (such as function value, or subgradient oracle), how many calls to the oracle are needed for an algorithm to output an approximate optimal solution? [34]. However, ever-growing problem sizes have shown an inadequacy in considering only the oracle-complexity, and have motivated the study of the trade-off between oracle-complexity and other resources such as memory [52, 31, 5] and communication[25, 40, 42, 45, 33, 53, 51, 50].

In this work, we study the oracle-complexity/memory trade-off for first-order non-smooth convex optimization, and the closely related feasibility problem, with a focus on developing memory efficient

(deterministic) algorithms. Since [52] formally posed as open problem the question of characterizing this trade-off, there have been exciting results showing what is impossible: for convex optimization in R d , [31] shows that any randomized algorithm with d 1 . 25 -δ bits of memory needs at least ˜ Ω( d 1+4 δ/ 3 ) queries, and this has later been improved for deterministic algorithms to d 1 -δ bits of memory or ˜ Ω( d 1+ δ/ 3 ) queries by [5]; in addition [5] shows that for the feasibility problem with a separation oracle, any algorithm which uses d 2 -δ bits of memory needs at least ˜ Ω( d 1+ δ ) queries.

Despite these recent results on the lower bounds, all known first-order convex optimization algorithms that output an ϵ -suboptimal point fall into two categories: those that have quadratic memory in the dimension d but can potentially achieve the optimal O ( d ln 1 ϵ ) query complexity, as represented by the center-of-mass method, and those that have O ( 1 ϵ 2 ) query complexity but only need the optimal O ( d ln 1 ϵ ) bits of memory, as represented by the classical gradient descent [52]. In addition, the above-mentioned memory bounds apply only between queries, and in particular the center-of-mass method [52] is allowed to use infinite memory during computations.

We propose a family of memory-constrained algorithms for the stronger feasibility problem in which one aims to find a point within a set Q containing a ball of radius ϵ , with access to a separation oracle. In particular, this can be used for convex optimization since the subgradient information provides a separation vector. Our algorithms use O ( d 2 p ln 1 ϵ ) bits of memory (including during computations) and O (( C d p ln 1 ϵ ) p ) queries for some universal constant C ≥ 1 , and a parameter p ∈ [ d ] that can be chosen by the user. Intuitively, in the context of convex optimization, the algorithms are based on the idea that for any function f ( x , y ) convex in the pair ( x , y ) , the partial minimum min y f ( x , y ) as a function of x is still convex and, using a variant of Vaidya's method proposed in [27], our algorithm can approximate subgradients for that function min y f ( x , y ) , thereby turning an optimization problem with variables ( x , y ) to one with just x . This idea, applied recursively with the variables divided into p blocks, gives our family of algorithms and the above-mentioned memory and query complexity. The main algorithmic contribution is in how we design the recursive dimension reduction procedure: a technical step of the design and analysis is to ensure that the necessary precision for recursive computations can be achieved using low memory. Last, our algorithms account for memory usage throughout computations, as opposed to simply between calls to the gradient oracle, which was the traditional approach in the literature.

When p = 1 , our algorithm is a memory-constrained version of Vaidya's method [48, 27], and improves over the center-of-mass [52] method by a factor of ln 1 ϵ in terms of memory while having optimal oracle-complexity. The improvements provided by our algorithms are more significant in regimes when ϵ is very small in the dimension d : increasing the parameter p can further reduce the memory usage of Vaidya's method ( p = 1 ) by a factor ln 1 ϵ / ln d , while still improving over the oracle-complexity of gradient descent. In particular, in a regime ln 1 ϵ = poly (ln d ) , these memory improvements are only in terms of ln d factors. However, in sub-polynomial regimes with potentially ln 1 ϵ = d c for some constant c > 0 , these provide polynomial improvements to the memory of standard cutting-plane methods.

As a summary, this paper makes the following contributions.

Our class of algorithms provides a trade-off between memory-usage and oracle-complexity whenever ln 1 ϵ ≫ ln d . Further, taking p = 1 improves the memory-usage from center-ofmass [52] by a factor ln 1 ϵ , while preserving the optimal oracle-complexity.

For ln 1 ϵ ≥ Ω( d ln d ) , our algorithm with p = d is the first known algorithm that outperforms gradient descent in terms of the oracle-complexity, but still maintains the optimal O ( d ln 1 ϵ ) memory usage.

We show how to obtain a ln 1 ϵ dependence in the known lower-bound trade-offs [31, 5], confirming that the oracle-complexity/memory trade-off is necessary for any regime ϵ ≲ 1 √ d .

2 Setup and Preliminaries

In this section, we precise the formal setup for our results. We follow the framework introduced in [52], to define the memory constraint on algorithms with access to an oracle O : S → R which

takes as input a query q ∈ S and outputs a response O ( q ) ∈ R . Here, the algorithm is constrained to update an internal M -bit memory between queries to the oracle.

Definition 2.1 ( M -bit memory-constrained algorithm [52, 31, 5]) . Let O : S → R be an oracle. An M -bit memory-constrained algorithm is specified by a query function ψ query : { 0 , 1 } M →S and an update function ψ update : { 0 , 1 } M ×S × R → { 0 , 1 } M . The algorithm starts with the memory state Memory 0 = 0 M and iteratively makes queries to the oracle. At iteration t , it makes the query q t = ψ query ( Memory t -1 ) to the oracle, receives the response r t = O ( q t ) then updates its memory Memory t = ψ update ( Memory t -1 , q t , r t ) .

The algorithm can stop at any iteration and the last query is its final output. Importantly, this model does not enforce constraints on the memory usage during the computation of ψ update and ψ query . This is ensured in the stronger notion of a memory-constrained algorithm with computations. These are precisely algorithms that have constrained memory including for computations, with the only specificity that they need a decoder function ϕ to make queries to the oracle from their bit memory, and a discretization function ψ to write a discretized response into the algorithm's memory.

Definition 2.2 ( M -bit memory-constrained algorithm with computations) . Let O : S → R be an oracle. We suppose that we are given a decoding function ϕ : { 0 , 1 } ⋆ → S and a discretization function ψ : R × N → { 0 , 1 } ⋆ such that ψ ( r, n ) ∈ { 0 , 1 } n for all r ∈ R . An M -bit memoryconstrained algorithm with computations is only allowed to use an M -bit memory in { 0 , 1 } M even during computations. The algorithm has three special memory placements Q,N,R . Say the contents of Q and N are q and n respectively. To make a query, R must contain at least n bits. The algorithm submits q to the encoder which then submits the query ϕ ( q ) to the oracle. If r = O ( ϕ ( q )) is the oracle response, the discretization function then writes ψ ( r, n ) in the placement R .

Feasibility problem. In this problem, the goal is to find a point x ∈ Q , where Q ⊂ C d := [ -1 , 1] d is a convex set. We choose the cube [ -1 , 1] d as prior bound for convenience in our later algorithms, but the choice of norm for this prior ball can be arbitrary and does not affect our results. The algorithm has access to a separation oracle O S : C d →{ Success } ∪ R d , such that for a query x ∈ R d either returns Success if x ∈ Q , or a separating hyperplane g ∈ R d , i.e., such that g ⊤ x < g ⊤ x ′ for any x ′ ∈ Q . We suppose that the separating hyperplanes are normalized, ∥ g ∥ 2 = 1 . An algorithm solves the feasibility problem with accuracy ϵ if the algorithm is successful for any feasibility problem such that Q contains an ϵ -ball B d ( x ⋆ , ϵ ) for x ⋆ ∈ C d .

As an important remark, this formulation asks that the separation oracle is consistent over time: when queried at the exact same point x , the oracle always returns the same separation vector. In this context, we can use the natural decoding function ϕ which takes as input d sequences of bits and outputs the vector with coordinates given by the sequences interpreted in base 2. Similarly, the natural discretization function ψ takes as input the separation hyperplane g and outputs a discretized version up to the desired accuracy. From now, we can omit these implementation details and consider that the algorithm can query the oracle for discretized queries x , up to specified rounding errors.

Remark 2.1. An algorithm for the feasibility problem with accuracy ϵ/ (2 d ) can be used for firstorder convex optimization. Suppose one aims to minimize a 1 -Lipschitz convex function f over the unit ball, and output an ϵ -suboptimal solution, i.e., find a point x such that f ( x ) ≤ min y ∈ B d (0 , 1) f ( y )+ ϵ . A separation oracle for Q = { x : f ( x ) ≤ min y ∈ B d (0 , 1) f ( y ) + ϵ } is given at a query x by the subgradient information from the first-order oracle: -∂f ( x ) ∥ ∂f ( x ) ∥ . Its computation can also be carried out memory-efficiently up to rounding errors since if ∥ ∂f ( x ) ∥ ≤ ϵ/ (2 √ d ) , the algorithm can return x and already has the guarantee that x is an ϵ -suboptimal solution ( C d has diameter 2 √ d ). Notice that because f is 1 -Lipschitz, Q contains a ball of radius ϵ/ (2 √ d ) (the factor 1 / (2 √ d ) is due to potential boundary issues). Hence, it suffices to run the algorithm for the feasibility problem while keeping in memory the queried point with best function value.

2.1 Known trade-offs between oracle-complexity and memory

Known lower-bound trade-offs. All known lower bounds apply to the more general class of memory-constrained algorithms without computational constraints given in Definition 2.1. [34] first showed that O ( d ln 1 ϵ ) queries are needed for solving convex optimization to ensure that one finds an ϵ -suboptimal solution. Further, O ( d ln 1 ϵ ) bits of memory are needed even just to output a solution in

√

the unit ball with ϵ accuracy [52]. These historical lower bounds apply in particular to the feasibility problem and are represented in the pictures of Fig. 1 as the dashed pink region.

More recently, [31] showed that achieving both optimal oracle-complexity and optimal memory is impossible for convex optimization. They show that a possibly randomized algorithm with d 1 . 25 -δ bits of memory makes at least ˜ Ω( d 1+4 δ/ 3 ) queries. This result was extended for deterministic algorithms in [5] which shows that a deterministic algorithm with d 1 -δ bits of memory makes ˜ Ω( d 1+ δ/ 3 ) queries. For the feasibility problem, they give an improved trade-off: any deterministic algorithm with d 2 -δ bits of memory makes ˜ Ω( d 1+ δ ) queries. These trade-offs are represented in the left picture of Fig. 1 as the pink, red, and purple solid region, respectively. Using a clever and more careful analysis,[11] showed that similar lower bounds can be carried out for deterministic algorithms as well.

Known upper-bound trade-offs. Prior to this work, to the best of our knowledge only two algorithms were known in the oracle-complexity/memory landscape. First, cutting-plane algorithms achieve the optimal oracle-complexity O ( d ln 1 ϵ ) but use quadratic memory. The memory-constrained (MC) center-of-mass method analyzed in [52] uses in particular O ( d 2 ln 2 1 ϵ ) memory. Instead, if one uses Vaidya's method which only needs to store O ( d ) cuts instead of O ( d ln 1 ϵ ) , we show that one can achieve O ( d 2 ln 1 ϵ ) memory. These algorithms only use the separation oracle and hence apply to both convex optimization and the feasibility problem. On the other hand, the memory-constrained gradient descent for convex optimization [52] uses the optimal O ( d ln 1 ϵ ) memory but makes O ( 1 ϵ 2 ) iterations. While the analysis in [52] is only carried for convex optimization, we can give a modified proof showing that gradient descent can also be used for the feasibility problem.

2.2 Other related works

Vaidya's method [48, 38, 1, 2] and the variant [27] that we use in our algorithms, belong to the family of cutting-plane methods. Perhaps the simplest example of an algorithm in this family is the center-of-mass method, which achieves the optimal O ( d ln 1 ϵ ) oracle-complexity but is computationally intractable, and the only known random walk-based implementation [4] has computational complexity O ( d 7 ln 1 ϵ ) . Another example is the ellipsoid method, which has suboptimal O ( d 2 ln 1 ϵ ) query complexity, but has an improved computational complexity O ( d 4 ln 1 ϵ ) . [8] pointed out that Vaidya's method achieves the best of both worlds by sharing the O ( d ln 1 ϵ ) optimal query complexity of the center-of-mass, and achieving a computational complexity of O ( d 1+ ω ln 1 ϵ ) 1 . In a major breakthrough, this computational complexity was improved to O ( d 3 ln 3 1 ϵ ) in [27], then to O ( d 3 ln 1 ϵ ) in [22]. We refer to [8, 27, 22] for more detailed comparisons of these algorithms.

Another popular convex optimization algorithm that requires quadratic memory is the BroydenFletcher- Goldfarb- Shanno (BFGS) algorithm [43, 7, 20, 21], which stores an approximated inverse Hessian matrix as gradient preconditioner. Several works aimed to reduce the memory usage of BFGS; in particular, the limited memory BFGS (L-BFGS) stores a few vectors instead of the entire approximated inverse Hessian matrix [37, 30]. However, it is still an open question whether even the original BFGS converges for non-smooth convex objectives [29].

Lying at the other extreme of the oracle-complexity/memory trade-off is gradient descent, which achieves the optimal memory usage but requires significantly more queries than center-of-mass or Vaidya's method in the regime ϵ ≲ 1 √ d . There is a rich literature of works aiming to speed up gradient descent, such as the optimized gradient method [17, 16], Nesterov's Acceleration [35], the triple momentum method [41], geometric descent [9], quadratic averaging [18], the informationtheoretic exact method [46], or Big-Step-Little-Step method [23]. Interested readers can find a comprehensive survey on acceleration methods in [12]. However, these acceleration methods usually require additional smoothness or strong convexity assumptions (or both) on the objective function, due to the known Ω( 1 ϵ 2 ) query lower bound in the large-scale regime ϵ ≳ 1 √ d for any first order method where the query points lie in the span of the subgradients of previous query points [36].

Besides accelerating gradient descent, researchers have investigated more efficient ways to leverage subgradients obtained in previous iterations. Of interest are bundle methods [3, 24, 28], that have

found a wide range of applications [47, 26]. In their simplest form, they minimize the sum of the maximum of linear lower bounds constructed using past oracle queries, and a regularization term penalizing the distance from the current iteration variable. Although the theoretical convergence rate of the bundle method is the same as that of gradient descent, in practice, bundle methods can benefit from previous information and substantially outperform gradient descent [3].

Our works are focused on high-accuracy regimes, when the accuracy ϵ is sub-polynomial. We note that for their lower-bound result on randomized algorithms, [11] also required sub-polynomial accuracies, which raises the question whether this is a general phenomenon for the study of memory-constrained algorithms in convex optimization. This also relates our work to the study of low-dimensional problems-or even constant dimension-which has been investigated in the literature [49, 10].

Last, the increasing size of optimization problems has also motivated the development of communication-efficient optimization algorithms in distributed settings such as [25, 40, 42, 45, 33, 53, 51, 50]. Moreover, recent works have explored the trade-off between sample complexity and memory/communication complexity for learning problems under the streaming model, with notable contributions including [6, 13, 14, 39, 44, 32].

3 Main results

We first check that the memory-constrained gradient descent method solves feasibility problems. This was known for convex optimization [52] and the same algorithm with a modified proof gives the following result. For completeness, the proof is given in Appendix D.

Proposition 3.1. The memory-constrained gradient descent algorithm solves the feasibility problem with accuracy ϵ ≤ 1 √ d using O ( d ln 1 ϵ ) bits of memory and O ( 1 ϵ 2 ) separation oracle calls.

Our main contribution is a class of algorithms based on Vaidya's cutting-plane method that provide a query-complexity / memory tradeoff for optimization in R d . More precisely, we show the following, where ω < 2 . 373 is the exponent of matrix multiplication, such that multiplying two n × n matrices runs in O ( n ω ) time.

Theorem 3.2. For any 1 ≤ p ≤ d , there is a deterministic first-order algorithm that solves the feasibility problem in dimension d for accuracy ϵ ≤ 1 √ , using O ( d 2 ln 1 ) bits of memory (includ- ing during computations), with p ϵ calls to the separation oracle, and computational complexity O (( C ( d ) 1+ ω ln 1 ) p ) , where C ≥ 1 is a universal constant.

d p ϵ O (( C d ln 1 ) p ) p ϵ

For simplicity, in Section 4, we describe algorithms that achieve this trade-off without computation concerns (Definition 2.1), which already provide the main elements of our method. The proof of oracle-complexity and memory usage is given in Appendix A. In Appendix B, we consider computational constraints and give corresponding algorithms using the cutting-plane method of [27].

To better understand the implications of Theorem 3.2, it is useful to compare the provided class of algorithms to the two algorithms known in the oracle-complexity/memory tradeoff landscape: the memory-constrained center-of-mass method and the memory-constrained gradient descent [52].

For p = 1 , our resulting procedure, which is essentially a memory-constrained Vaidya's algorithm, has optimal oracle-complexity O ( d ln 1 ϵ ) and uses O ( d 2 ln 1 ϵ ) bits of memory. This improves by a ln 1 ϵ factor the memory usage of the center-of-mass-based algorithm provided in [52], which used O ( d 2 ln 2 1 ϵ ) memory and had the same optimal oracle-complexity.

Next, we recall that the memory-constrained gradient descent method used the optimal number O ( d ln 1 ϵ ) bits of memory (including for computations), and a sub-optimal O ( 1 ϵ 2 ) oracle-complexity. While the memory of our algorithms decreases with p , their oracle-complexity is exponential in p . This significantly restricts the values of p for which the oracle-complexity is improved over that of gradient descent. The range of application of Theorem 3.2 is given in the next result, where ∨ and ∧ represent maximum and minimum respectively.

Corollary 3.1. The algorithms given in Theorem 3.2 effectively provide a tradeoff for p ≤ O ( ln 1 ϵ ln d ∧ d ) . Precisely, this provides a tradeoff between

using O ( d 2 ln 1 ϵ ) memory with optimal O ( d ln 1 ϵ ) oracle-complexity, and

Oracle

complexity

d

ln

1

ϵ

d

5

/

4

ln

d

ln

1

ϵ

d

2

ln

d

d

ln

1

ϵ

d

4

/

3

ln

4

/

3

d

ln

1

ϵ

d

2

ln

2

d

ln

1

ϵ

1

/ϵ

2

MC-GD [52]

MC-Center

of Mass [52]

MC-Vaidya

d

2

ln

2 1

ϵ

d

2

ln

1

ϵ

d

2

ln

2

d

ln

1

ϵ

Memory (bits)

Regime

ln

d

≪

ln

1

ϵ

≪

d

ln

d

d

ln

1

ϵ

d

ln

1

ϵ

1

/ϵ

2

d

2

ln

2 1

ϵ

d

2

ln

1

ϵ

(

C

ln

1

ϵ

)

d

Regime

ln

1

ϵ

= Ω(

d

ln

d

)

· using O ( d 2 ln d ∨ d ln 1 ϵ ) memory with O ( 1 ϵ 2 ∧ ( C ln 1 ϵ ) d ) oracle-complexity.

Importantly, for ϵ ≤ 1 d Ω( d ) , taking p = d yields an algorithm that uses the optimal memory O ( d ln 1 ϵ ) and has an improved query complexity over gradient descent. In this regime of small (virtually constant) dimension, for the same memory usage, gradient descent has a query complexity that is polynomial in ϵ , O ( 1 ϵ 2 ) , while our algorithm has poly-logarithmic dependence in ϵ , O d (ln d 1 ϵ ) , where O d hides an exponential constant in d . It remains open whether this ln d 1 ϵ dependence in the oraclecomplexity is necessary. To the best of our knowledge, this is the first example of an algorithm that improves over gradient descent while keeping its optimal memory usage in any regime where ϵ ≤ 1 √ d . While this improvement holds only in the exponential regime ϵ ≤ 1 d O ( d ) , Theorem 3.2 still provides a non-trivial trade-off whenever ln 1 ϵ ≫ ln d , and improves over the known memory-constrained center-of-mass in the standard regime ϵ ≤ 1 √ d [52]. Fig. 1 depicts the trade-offs in the two regimes mentioned earlier.

Last, we note that the lower-bound trade-offs presented in [31, 5] do not show a dependence in the accuracy ϵ . Especially in the regime when ln 1 ϵ ≫ ln d , this yields sub-optimal lower bounds (in fact even in the regime ϵ = 1 / poly ( d ) , our more careful analysis improves the lower bound on the memory by a ln d factor). We show with simple arguments that one can extend their results to include a ln 1 ϵ factor for both memory and query complexity. Fig. 1 presented these improved lower bounds.

Theorem 3.3. For ϵ ≤ 1 / poly ( d ) and any δ ∈ [0 , 1] (the notation ˜ Ω hides ln O (1) d factors),

any (randomized) algorithm guaranteed to minimize 1 -Lipschitz convex functions over the unit ball with accuracy ϵ uses d 5 / 4 -δ ln 1 ϵ bits of memory or makes ˜ Ω( d 1+4 δ/ 3 ln 1 ϵ ) queries,

any deterministic algorithm guaranteed to minimize 1 -Lipschitz convex functions over the unit ball with accuracy ϵ uses d 2 -δ ln 1 ϵ bits of memory or makes ˜ Ω( d 1+ δ/ 3 ln 1 ϵ ) queries,

any deterministic algorithm guaranteed to solve the feasibility problem over the unit ball with accuracy ϵ uses d 2 -δ ln 1 ϵ bits of memory or makes ˜ Ω( d 1+ δ ln 1 ϵ ) queries.

The proof is given in Appendix C and the arguments therein could readily be used to exhibit the ln 1 ϵ dependence of potential future works improving over these lower bounds trade-offs.

Sketch of proof. At a high level, [31, 5] use a barrier term ∥ Ax ∥ ∞ where A has Θ( d ) rows: if an algorithm does not have enough memory, A cannot be fully stored which in turn incurs a sub-optimal oracle-complexity. To achieve a ln 1 ϵ improvement in memory (Appendix C.1), we modify the sampling of rows of A , from uniform on vertices of the hypercube to uniform in an ϵ -net. The proof can then be adapted accordingly. Last, one can improve the oracle-complexity by a ln 1 ϵ / ln d factor (Appendix C.2) using a standard rescaling argument [34].

4 Memory-constrained feasibility problem without computation

In this section, we present a class of algorithms that are memory-constrained according to Definition 2.1 and achieve the desired memory and oracle-complexity bounds. We emphasize that the memory constraint is only applied between calls to the oracle and as a result, the algorithm is allowed infinite computation memory and computation power between calls to the oracle.

We start by defining discretization functions that will be used in our algorithms. For ξ > 0 and x ∈ [ -1 , 1] , we pose Discretize 1 ( x, ξ ) = sign ( x ) · ξ ⌊| x | /ξ ⌋ . Next, we define the discretization Discretize d for general dimensions d ≥ 1 . For any x ∈ C and ξ > 0 ,

4.1 Memory-constrained Vaidya's method

Our algorithm recursively uses Vaidya's cutting-plane method [48] and subsequent works expanding on this method. We briefly describe the method. Given a polyhedron P = { x : Ax ≥ b } , we define s i ( x ) = a ⊤ i x -b i and S x = diag ( s i ( x ) , i ∈ [ d ]) . We will also use the shorthand A x = S -1 x A . The volumetric barrier is defined as

At each step, Vaidya's method queries the volumetric center of the polyhedron, which is the point minimizing the volumetric barrier. For convenience, we denote by VolumetricCenter this function, i.e., for any A ∈ R m × d and b ∈ R d defining a non-empty polyhedron P = { x : Ax ≥ b } ,

When the polyhedron is unbounded, we can for instance take VolumetricCenter ( A , b ) = 0 . Vaidya's method makes use of leverage scores for each constraint i of the polyhedron, defined as σ i = ( A x H -1 A ⊤ x ) i,i , where H = A ⊤ x A x . We are now ready to define the update procedure for the polyhedron considered by Vaidya's volumetric method. We denote by P t the polyhedron stored in memory after making t queries. The method keeps in memory the constraints defining the current polyhedron and the iteration index k when these constraints were added, which will be necessary for our next procedures. Hence, the polyhedron will be stored in the form P t = { ( k i , a i , b i ) , i ∈ [ m ] } , and the associated constraints are given via { x : Ax ≥ b } where A ⊤ = [ a 1 , . . . , a m ] and b ⊤ = [ b 1 , . . . , b m ] . By abuse of notation, we will write VolumetricCenter ( P ) for the volumetric center of the polyhedron VolumetricCenter ( A , b ) where A and b define the constraints stored in P .

Initially, the polyhedron is simply C d , these constraints are given -1 index for convenience, and they will not play a role in the next steps. At each iteration, if the constraint i ∈ [ m ] with minimum leverage score σ i falls below a given threshold σ min , it is removed from the polyhedron. Otherwise, we query the volumetric center of the current polyhedron and add the separation hyperplane as a constraint to the polyhedron. We bound the number of iterations of the procedure by

where σ min and c are parameters that will be fixed shortly. Instead of making a call directly to the oracle O S , we instead suppose that one has access to an oracle O : I d → R d where I d = ( Z × R d +1 ) ⋆ has exactly the shape of the memory storing the information from the polyhedron. This form of oracle is used in our recursive calls to Vaidya's method. For example, such an oracle can simply be O : P ∈ I d ↦→ O S ( VolumetricCenter ( P )) . Last, in our recursive method, we will not assume that oracle responses are normalized. As a result, we specify that if the norm of the response is too small, we can stop the algorithm. We assume however that the oracle already returns discretized vectors, which will be ensured in the following procedures. The cutting-plane algorithm is formally described in Algorithm 1. With an appropriate choice of parameters, this procedure finds an approximate solution of feasibility problems. We base the constants from [2].

Lemma 4.1. Fix σ min = 0 . 04 and c = 1 0 . 0014 ≈ 715 . Let δ, ξ ∈ (0 , 1) and O : I d → R d . Write P = { ( k i , a i , b i ) , i ∈ [ m ] } as the output of Algorithm 1 run with O , δ and ξ . Then,

From now, we use the parameters σ min = 0 . 04 and c = 1 / 0 . 0014 as in Lemma 4.1. Since the memory of both Vaidya's method and center-of-mass consists primarily of the constraints, we recall an important feature of Vaidya's method that the number of constraints at any time is O ( d ) .

Lemma 4.2 ([48, 1, 2]) . At any time while running Algorithm 1, the number of constraints of the current polyhedron is at most d σ min +1 .

4.2 A recursive algorithm

We write C m + n = C m ×C n and aim to apply Vaidya's method to the first m coordinates. To do so, we need to approximate a separation oracle on these m coordinates only, which corresponds to giving separation hyperplanes with small values for the last n coordinates. This can be achieved using the following auxiliary linear program. For P ∈ I n , we define

where as before, A and b define the constraints stored in P . The procedure to obtain an approximate separation oracle on the first n coordinates C n is given in Algorithm 2 and using Lemma 4.1 we can show that this procedure provides approximate separation vectors for the first n coordinates.

The next step involves using this approximation recursively. We write d = ∑ p i =1 k i , and interpret C d as C k 1 ×··· × C k p . In particular, for x ∈ C d , we write x = ( x 1 , . . . , x p ) where x i ∈ C k i for i ∈ [ p ] . Applying Algorithm 2 recursively, we can obtain an approximate separation oracle for the first i coordinates C k 1 ×···×C k i . However, storing such separation vectors would be too memory-expensive, e.g., for i = p , that would correspond to storing the separation hyperplanes from the oracle O S directly. Instead, given j ∈ [ i ] , Algorithm 3 recursively computes the x j component of an approximate separation oracle for the first i variables ( x 1 , . . . , x i ) , via the procedure ApproxOracle ( i, j ) .

We can then use ApproxOracle δ,ξ,O S (1 , 1 , · ) to solve the original problem with the memoryconstrained Vaidya's method. In Appendix A, we show that taking δ = ϵ 4 d and ξ = σ min ϵ 32 d 5 / 2 achieves the desired oracle-complexity and memory usage. The final algorithm is given in Algorithm 4.

A geometric illustration of the recursive step. In Figure 2, we give a 2-dimensional feasibility problem with target p ∗ = ( p ∗ 1 , p ∗ 2 ) and two blocks (i.e. p = 2 ) as an illustration of our recursive approach (Algorithm 2) to construct an approximate separating hyperplane for a 'reduced' problem.

Suppose at a step of the Algorithm 4, the current value of the x 1 coordinate is c . We aim to find an approximate separating hyperplane between x 1 = p ∗ 1 and x 1 = c . Algorithm 2 first runs Algorithm 1 (i.e. the memory-constrained Vaidya) to find two separating hyperplanes (the two blue hyperplanes). Lemma 4.1 then guarantees the existence of a convex combination of the 2 blue hyperplanes - the

x

1

x

2

x

1

=

c

x

1

=

p

∗

1

p

∗

red hyperplane- which is approximately parallel to the x 2 -axis and thus can serve as an approximate separating hyperplane between x 1 = p ∗ 1 and x 1 = c .

Sketch of proof. At the high level, the algorithm recursively runs Vaidya's method Algorithm 1 for each level of computation i ∈ [ p ] . Since each run of Algorithm 4 requires O ( d p ln 1 ϵ ) queries, the total number of calls to the oracle, which is exponential in the number of levels, is O ( O ( d p ln 1 ϵ ) p ) . As for the memory usage, the algorithm mainly needs to keep in memory the constraints defining the polyhedrons at each level i ∈ [ p ] . From Lemma 4.2, each polyhedron only requires O ( d p ) constraints that each require O ( d p ln 1 ϵ ) bits of memory. Hence, the total memory needed is O ( d 2 p ln 1 ϵ ) . The main difficulty lies in showing that the algorithm is successful. To do so, we need to show that the precision in the successive approximated separation oracles Algorithm 2 is sufficient. To avoid an exponential dependence of the approximation error in p -which would be prohibitive for the memory usage of our method-each run of Vaidya's method Algorithm 1 is run for more iterations than the precision of the separation vectors would classically allow. To give intuition, if the separation oracle came from a convex optimization subgradient oracle for a function f , the iterates at a level i do not converge to the true 'minimizer' of min x i f ( i ) ( x 1 , . . . , x i ) , where f ( i ) ( · ) = min x i +1 ,..., x p f ( · , x i +1 , . . . , x p ) , but instead converge to a close enough point while still providing meaningful approximate subgradients at the higher level i -1 (in Algorithm 2).

5 Discussion and Conclusion

To the best of our knowledge, this work is the first to provide some positive trade-off between oracle-complexity and memory-usage for convex optimization or the feasibility problem, as opposed to lower-bound impossibility results [31, 5]. Our trade-offs are more significant in a high accuracy regime: when ln 1 ϵ ≈ d c , for c > 0 our trade-offs are polynomial, while the improvements when ln 1 ϵ = poly (ln d ) are only in ln d factors. A natural open direction [52] is whether there exist algorithms with polynomial trade-offs in that case. We also show that in the exponential regime ln 1 ϵ ≥ Ω( d ln d ) , gradient descent is not Pareto-optimal. Instead, one can keep the optimal memory and decrease the dependence in ϵ of the oracle-complexity from 1 ϵ 2 to (ln 1 ϵ ) d . The question of whether the exponential dependence in d is necessary is left open. Last, our algorithms rely on the consistency of the oracle, which allows re-computations. While this is a classical assumption, gradient descent and classical cutting-plane methods do not need it; removing this assumption could be an interesting research direction (potentially, this could also yield stronger lower bounds).

Acknowledgments and Disclosure of Funding

This work was partly funded ONR grant N00014-18-1-2122, AFOSR grant FA9550-19-1-0263, and AFOSR grant FA9550-23-1-0182.

References

Kurt M Anstreicher. 'On Vaidya's volumetric cutting plane method for convex programming'. In: Mathematics of Operations Research 22.1 (1997), pp. 63-89.

Kurt M Anstreicher. 'Towards a practical volumetric cutting plane method for convex programming'. In: SIAM Journal on Optimization 9.1 (1998), pp. 190-206.

Aharon Ben-Tal and Arkadi Nemirovski. 'Non-euclidean restricted memory level method for large-scale convex optimization'. In: Mathematical Programming 102.3 (Jan. 2005), pp. 407456.

Dimitris Bertsimas and Santosh Vempala. 'Solving convex programs by random walks'. In: Journal of the ACM (JACM) 51.4 (2004), pp. 540-556.

Moise Blanchard, Junhui Zhang, and Patrick Jaillet. 'Quadratic Memory is Necessary for Optimal Query Complexity in Convex Optimization: Center-of-Mass is Pareto-Optimal'. In: arXiv preprint arXiv:2302.04963 (2023).

Mark Braverman et al. 'Communication Lower Bounds for Statistical Estimation Problems via a Distributed Data Processing Inequality'. In: Proceedings of the Forty-Eighth Annual ACM Symposium on Theory of Computing . STOC '16. Cambridge, MA, USA: Association for Computing Machinery, 2016, pp. 1011-1020.

C. G. Broyden. 'The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations'. In: IMA Journal of Applied Mathematics 6.1 (Mar. 1970), pp. 76-90.

Sébastien Bubeck. 'Convex Optimization: Algorithms and Complexity'. In: Foundations and Trends® in Machine Learning 8.3-4 (2015), pp. 231-357.

Sébastien Bubeck, Yin Tat Lee, and Mohit Singh. A geometric alternative to Nesterov's accelerated gradient descent . 2015.

Sébastien Bubeck and Dan Mikulincer. 'How to trap a gradient flow'. In: Conference on Learning Theory . PMLR. 2020, pp. 940-960.

Xi Chen and Binghui Peng. 'Memory-Query Tradeoffs for Randomized Convex Optimization'. In: arXiv preprint arXiv:2306.12534 (2023).

Alexandre d'Aspremont, Damien Scieur, and Adrien Taylor. 'Acceleration Methods'. In: Foundations and Trends® in Optimization 5.1-2 (2021), pp. 1-245.

Yuval Dagan, Gil Kur, and Ohad Shamir. 'Space lower bounds for linear prediction in the streaming model'. In: Proceedings of the Thirty-Second Conference on Learning Theory . Ed. by Alina Beygelzimer and Daniel Hsu. Vol. 99. Proceedings of Machine Learning Research. PMLR, 25-28 Jun 2019, pp. 929-954.

Yuval Dagan and Ohad Shamir. 'Detecting Correlations with Little Memory and Communication'. In: Proceedings of the 31st Conference On Learning Theory . Ed. by Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet. Vol. 75. Proceedings of Machine Learning Research. PMLR, June 2018, pp. 1145-1198.

James Demmel, Ioana Dumitriu, and Olga Holtz. 'Fast linear algebra is stable'. In: Numerische Mathematik 108.1 (2007), pp. 59-91.

Kim Donghwan and Jeffrey Fessler. 'Optimized first-order methods for smooth convex minimization'. In: Mathematical Programming 159 (June 2014).

Yoel Drori and Marc Teboulle. 'Performance of first-order methods for smooth convex minimization: a novel approach'. In: Mathematical Programming 145.1 (June 2014), pp. 451482.

Dmitriy Drusvyatskiy, Maryam Fazel, and Scott Roy. 'An Optimal First Order Method Based on Optimal Quadratic Averaging'. In: SIAM Journal on Optimization 28.1 (2018), pp. 251271.

Uriel Feige and Gideon Schechtman. 'On the optimality of the random hyperplane rounding technique for MAX CUT'. In: Random Structures & Algorithms 20.3 (2002), pp. 403-440.

R. Fletcher. 'A new approach to variable metric algorithms'. In: The Computer Journal 13.3 (Jan. 1970), pp. 317-322.

Donald Goldfarb. 'A Family of Variable-Metric Methods Derived by Variational Means'. In: Mathematics of Computation 24.109 (1970), pp. 23-26.

Haotian Jiang et al. 'An Improved Cutting Plane Method for Convex Optimization, ConvexConcave Games, and Its Applications'. In: Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing . STOC 2020. Chicago, IL, USA: Association for Computing Machinery, 2020, pp. 944-953.

Jonathan Kelner et al. 'Big-Step-Little-Step: Efficient Gradient Methods for Objectives with Multiple Scales'. In: Proceedings of Thirty Fifth Conference on Learning Theory . Ed. by Po-Ling Loh and Maxim Raginsky. Vol. 178. Proceedings of Machine Learning Research. PMLR, Feb. 2022, pp. 2431-2540.

Guanghui Lan. 'Bundle-level type methods uniformly optimal for smooth and nonsmooth convex optimization'. In: Mathematical Programming 149.1 (Feb. 2015), pp. 1-45.

Guanghui Lan, Soomin Lee, and Yi Zhou. 'Communication-efficient algorithms for decentralized and stochastic optimization'. In: Mathematical Programming 180.1 (Mar. 2020), pp. 237-284.

Quoc Le, Alex Smola, and S.V.N. Vishwanathan. 'Bundle Methods for Machine Learning'. In: Advances in Neural Information Processing Systems . Ed. by J. Platt et al. V ol. 20. Curran Associates, Inc., 2007.

Yin Tat Lee, Aaron Sidford, and Sam Chiu-wai Wong. 'A faster cutting plane method and its implications for combinatorial and convex optimization'. In: 2015 IEEE 56th Annual Symposium on Foundations of Computer Science . IEEE. 2015, pp. 1049-1065.

Claude Lemaréchal, Arkadi Nemirovski, and Yurii Nesterov. 'New variants of bundle methods'. In: Mathematical Programming 69.1 (July 1995), pp. 111-147.

Adrian S. Lewis and Michael L. Overton. 'Nonsmooth optimization via quasi-Newton methods'. In: Mathematical Programming 141.1 (Oct. 2013), pp. 135-163.

Dong C. Liu and Jorge Nocedal. 'On the limited memory BFGS method for large scale optimization'. In: Mathematical Programming 45.1 (Aug. 1989), pp. 503-528.

Annie Marsden et al. 'Efficient convex optimization requires superlinear memory'. In: Conference on Learning Theory . PMLR. 2022, pp. 2390-2430.

Dana Moshkovitz and Michal Moshkovitz. 'Mixing Implies Lower Bounds for Space Bounded Learning'. In: Proceedings of the 2017 Conference on Learning Theory . PMLR, 2017, pp. 1516-1566.

João F. C. Mota et al. 'D-ADMM: A Communication-Efficient Distributed Algorithm for Separable Optimization'. In: IEEE Transactions on Signal Processing 61.10 (2013), pp. 27182723.

Arkadi Nemirovski and David Borisovich Yudin. Problem Complexity and Method Efficiency in Optimization . A Wiley-Interscience publication. Wiley, 1983.

Yurii Nesterov. 'A method of solving a convex programming problem with convergence rate O (1 /k 2 ) '. In: Dokl. Akad. Nauk SSSR 269 (3 1983), pp. 543-547.

Yurii Nesterov. Introductory lectures on convex optimization: A basic course . Vol. 87. Springer Science & Business Media, 2003.

Jorge Nocedal. 'Updating Quasi-Newton Matrices with Limited Storage'. In: Mathematics of Computation 35.151 (1980), pp. 773-782.

Srinivasan Ramaswamy and John E Mitchell. A long step cutting plane algorithm that uses the volumetric barrier . Tech. rep. Citeseer, 1995.

Ran Raz. 'A Time-Space Lower Bound for a Large Class of Learning Problems'. In: 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) . 2017, pp. 732742.

Sashank J. Reddi et al. 'AIDE: Fast and Communication Efficient Distributed Optimization'. In: ArXiv abs/1608.06879 (2016).

B. Van Scoy, R. A. Freeman, and K. M. Lynch. 'The Fastest Known Globally Convergent First-Order Method for Minimizing Strongly Convex Functions'. In: IEEE Control Systems Letters PP.99 (2017), pp. 1-1.

Ohad Shamir, Nati Srebro, and Tong Zhang. 'Communication-Efficient Distributed Optimization using an Approximate Newton-type Method'. In: Proceedings of the 31st International Conference on Machine Learning . Ed. by Eric P. Xing and Tony Jebara. Vol. 32. Proceedings of Machine Learning Research 2. Bejing, China: PMLR, 22-24 Jun 2014, pp. 1000-1008.

D. F. Shanno. 'Conditioning of Quasi-Newton Methods for Function Minimization'. In: Mathematics of Computation 24.111 (1970), pp. 647-656.

Vatsal Sharan, Aaron Sidford, and Gregory Valiant. 'Memory-Sample Tradeoffs for Linear Regression with Small Error'. In: Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing . STOC 2019. Association for Computing Machinery, 2019, pp. 890-901.

Virginia Smith et al. 'CoCoA: A General Framework for Communication-Efficient Distributed Optimization'. In: J. Mach. Learn. Res. 18.1 (Jan. 2017), pp. 8590-8638.

Adrien Taylor and Yoel Drori. 'An optimal gradient method for smooth strongly convex minimization'. In: Mathematical Programming 199.1 (May 2023), pp. 557-594.

Choon Hui Teo et al. 'Bundle Methods for Regularized Risk Minimization'. In: Journal of Machine Learning Research 11.10 (2010), pp. 311-365.

Pravin M Vaidya. 'A new algorithm for minimizing convex functions over convex sets'. In: Mathematical programming 73.3 (1996), pp. 291-341.

Stephen A Vavasis. 'Black-box complexity of local minimization'. In: SIAM Journal on Optimization 3.1 (1993), pp. 60-80.

Jialei Wang, Weiran Wang, and Nathan Srebro. 'Memory and Communication Efficient Distributed Stochastic Optimization with Minibatch Prox'. In: Proceedings of the 2017 Conference on Learning Theory . Ed. by Satyen Kale and Ohad Shamir. Vol. 65. Proceedings of Machine Learning Research. PMLR, July 2017, pp. 1882-1919.

Jianqiao Wangni et al. 'Gradient Sparsification for Communication-Efficient Distributed Optimization'. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems . NIPS'18. Montréal, Canada: Curran Associates Inc., 2018, pp. 13061316.

Blake Woodworth and Nathan Srebro. 'Open problem: The oracle complexity of convex optimization with limited memory'. In: Conference on Learning Theory . PMLR. 2019, pp. 32023210.

Yuchen Zhang, John C. Duchi, and Martin J. Wainwright. 'Communication-efficient algorithms for statistical optimization'. In: 2012 IEEE 51st IEEE Conference on Decision and Control (CDC) . 2012, pp. 6792-6792.

A Proof of the query complexity and memory usage of Algorithm 4

First, we give simple properties on the discretization functions. One can easily check that for any x ∈ C ,

Further, one can easily check that to represent any output of Discretize d ( · , ξ ) , one needs at most d ln 2 √ d ξ = O ( d ln d ξ ) bits.

Proof of Lemma 4.1. We first consider the case when the algorithm terminates because of a query g = O ( P t ) such that ∥ g ∥ ≤ δ/ (2 √ d ) . Then, for any x ∈ C d , one directly has

where ω is the volumetric center of the resulting polyhedron. In the second inequality we used the fact that ω ∈ C d , otherwise the algorithm would not have terminated at that step.

We next turn to the other cases and start by showing that the output polyhedron does not contain a ball of radius δ. This is immediate if the algorithm terminated because the polyhedron was empty. We then suppose this was not the case, and follow the same proof as given in [2]. Algorithm 1 and the one provided in [2] coincide when removing a constraint of the polyhedron. Hence, it suffices to consider the case when we add a constraint. We use the notation ˜ A ⊤ = [ A ⊤ , a ⊤ m +1 ] , ˜ b ⊤ = [ b ⊤ , b m +1 ] for the updated matrix A and vector b after adding the constraint. We also denote ω = VolumetricCenter ( A , b ) (resp. ˜ ω = VolumetricCenter ( ˜ A , ˜ b ) ) the volumetric center of the polyhedron before (resp. after) adding the constraint. Next, we consider the vector ( b ′ ) ⊤ = [ b ⊤ , a ⊤ m +1 ω ] , which would have been obtained if the cut was performed at ω exactly. We then denote ω ′ = VolumetricCenter ( ˜ A , b ′ ) . Then proof of [2] shows that

We now observe that by construction, we have ˜ b m +1 ≥ a ⊤ m +1 ω , so that the polyhedron associated to ( ˜ A , ˜ b ) is more constrained than the one associated to ( ˜ A , b ′ ) . As a result, we have V ˜ A , ˜ b ( x ) ≥ V ˜ A , b ′ ( x ) , for any x ∈ R d such that ˜ Ax ≥ ˜ b . Therefore,

This ends the modifications in the proof of [2]. With the notations of this paper, we still have ∆ V + = 0 . 340 and ∆ V -= 0 . 326 , so that ∆ V = 0 . 0014 . Then, because c = 1 ∆ V , the same proof shows that the procedure is successful for precision δ : the final polyhedron ( A , b ) returned by Algorithm 1 does not contains a ball of radius > δ . As a result, whether the algorithm performed all T max iterations or not, { x : Ax ≥ b } does not contain a ball of radius > δ ′ , where A and b define the constraints stored in the output P . Now letting m be the objective value of the right optimization problem, there exists x ∈ C d such that for all t ≤ T , g ⊤ t ( x -c t ) ≥ m . Therefore, for any x ′ ∈ B d ( x , m ) one has

In the last inequality we used ∥ a t ∥ ≤ 1 . This implies that the polyhedron contains B d ( x , m ) . Hence, m ≤ δ .

This ends the proof of the right inequality. The left equality is a direct application of strong duality for linear programming.

We now prove that Algorithm 4 has the desired oracle-complexity and memory usage.

We first describe the recursive calls of Algorithm 3 in more detail. To do so, consider running the procedure ApproxOracle ( i, j, P (1) , . . . , P ( i ) ) where i < p , which corresponds to running Algorithm 2 for specific oracles. We say that this is a leveli run. Then, the algorithm performs at most 2 T ( δ, k i +1 ) calls to ApproxOracle ( i +1 , i +1 , P (1) , . . . , P ( i ) , · ) , where the factor 2 comes from the fact that

Vaidya's method Algorithm 1 is effectively run twice in Algorithm 2. The solution to ( P aux ( P ) ) has as many components as constraints in the last polyhedron, which is at most k i +1 σ min +1 by Lemma 4.2. Hence, the number of calls to ApproxOracle ( i +1 , j, P (1) , . . . , P ( i ) , · ) is at most k i +1 σ min +1 . In total, that is O ( k i +1 ln 1 δ ) calls to the level i +1 of the recursion.

We next aim to understand the output of running ApproxOracle (1 , 1 , P (1) ) . We denote by λ ( P (1) ) the solution P aux ( P ⋆ ) computed at l.2 of the first call to Algorithm 2, where P ⋆ is the output polyhedron of the first call to Algorithm 1. Denote by S ( P (1) ) the set of indices of coordinates from λ ( P (1) ) for which the procedure performed a call to ApproxOracle (2 , 1 , P (1) , · ) . In other words, S ( P (1) ) contains the indices of all coordinates of λ ( P (1) ) , except those for which the corresponding query lay outside of the unit cube, or the initial constraints of the cube. For any index l ∈ S ( P (1) ) , let P (2) l denote the state of the current polyhedron ( P t in l.7 of Algorithm 2) when that call was performed. Up to discretization issues, the output of the complete procedure is

We continue in the recursion, defining λ ( P (1) , P (2) l ) and S ( P (1) , P (2) l ) for all l ∈ S ( P (1) ) , until we define all vectors of the form λ ( P (1) , P (2) l 2 , . . . , P ( r ) l r ) and sets of the form S ( P (1) , P (2) l 2 , . . . , P ( r ) l r ) for i +1 ≤ r ≤ p -1 . To simplify the notation and emphasize that all these polyhedra depend on the recursive computation path, we adopt the notation

We recall that these polyhedron are kept in memory to query their volumetric center. For ease of notation, we write x 1 = VolumetricCenter ( P (1) ) , and we write c l 2 ,...,l r = VolumetricCenter ( P ( r ) l r ) for 2 ≤ r ≤ p , where l 2 , . . . , l r -1 were the indices from the computation path leading up to P ( r ) l r . Last, we write O S = ( O S, 1 , . . . , O S,p ) , where O S,i : C d → R k i is the ' x i ' component of O S , for all i ∈ [ p ] .

With all these notations, we will show that the output of ApproxOracle ( i, j, P (1) , P (2) l 2 , . . . , P ( i ) l i ) is approximately equal to the vector

with the convention that for i = p ,

The corresponding computation tree is represented in Fig. 3. For convenience, we omitted the term j = 1 .

We start the analysis with a simple result showing that if the oracle O S returns separation vectors of norm bounded by one, then the responses from ApproxOracle also lie in the unit ball.

Lemma A.1. Fix δ, ξ ∈ (0 , 1) , 1 ≤ j ≤ i ≤ p and an oracle O S = ( O S, 1 , . . . , O S,p ) : C d → R d . Suppose that O S takes values in the unit ball. For any s ∈ [ i ] let P ( s ) l s ∈ I k s represent a bounded polyhedrons with VolumetricCenter ( P ( s ) l s ) ∈ C k s . Then, one has

Proof. We prove this by simple induction on i . For convenience, we define the point x k = VolumetricCenter ( P ( k ) l k ) . If i = p , we have

G

(1)

G

(2

,

c

1

)

λ

1

. . .

G

(2

,

c

l

2

)

G

(3

,

c

l

2

,

c

l

2

,

1

)

λ

l

2

,

1

. . .

G

(3

,

c

l

2

,

c

l

2

,l

3

)

G

(

p

-

1

,

c

l

2

, . . . ,

c

l

2

...,l

p

-

1

)

O

S,j

(

c

l

2

, . . . ,

c

l

2

,...,l

p

-

1

,

1

)

λ

l

2

,...,l

p

-

1

,

1

. . .

O

S,j

(

c

l

2

, . . . ,

c

l

2

,...,l

p

)

λ

l

2

,...,l

p

. . .

O

S,j

(

c

l

2

, . . . ,

c

l

2

,...,l

p

-

1

,m

p

)

λ

l

2

,...,l

p

-

1

,m

p

.

.

.

λ

l

2

,l

3

. . .

G

(3

,

c

l

2

,

c

l

2

,m

3

)

λ

l

2

,m

3

λ

l

2

. . .

G

(2

,

c

m

2

)

λ

m

2

where in the first inequality we used Eq (1) and in the second inequality we used the fact that O S ( x 1 , . . . , x p ) has norm at most one. Now suppose that the result holds for i +1 ≤ p . Then by construction, the output ApproxOracle δ,ξ,O S ( i, j, P (1) l 1 , . . . , P ( i ) l i ) is the result of iterative discretizations. Using Eq (1) and the previously defined notations, we obtain

In the last inequality, we used the induction hypothesis together with the fact that ∑ l i +1 λ l 2 ,...,l i +1 ≤ 1 using Eq (1). This ends the induction and the proof.

We are now ready to compare the output of Algorithm 3 to G ( i, j, x 1 , c l 2 , . . . , c l 2 ,...,l i ) .

Lemma A.2. Fix δ, ξ ∈ (0 , 1) , 1 ≤ j ≤ i ≤ p and an oracle O S = ( O S, 1 , . . . , O S,p ) : C d → R d . Suppose that O S takes values in the unit ball. For any s ∈ [ i ] let P ( s ) l s ∈ I k s represent a bounded polyhedron with VolumetricCenter ( P ( s ) l s ) ∈ C k s . Denote x r = c ( P ( r ) l r ) for r ∈ [ i ] . Then,

Proof. We prove by simple induction on i that

First, for i = p , the result is immediate since the discretization is with precision ξ (l.4 of Algorithm 3). Now suppose that this is the case for i ≤ p and any valid values of other parameters. For conciseness, we write G = ( P (1) l 1 , . . . , P ( i -1) l i -1 ) . Next, recall that by Lemma 4.2, |S l 2 ,...,l i -1 | ≤ k i σ min +1 . Hence,

the discretizations due to l.8 of Algorithm 2 can affect the estimate for at most that number of rounds. Then, we have

where ˜ λ l 2 ,...,l i are the discretized coefficients that are used during the computation l.8 of Algorithm 2. Now using Lemma A.1, we have

In the last inequality we used the fact that λ has at most k i σ min +1 non-zero coefficients. As a result, using the induction for each term of the sum, and the fact that ∑ l i λ l 2 ,...,l i ≤ 1 , we obtain

which completes the induction. Noting that k i +1 + . . . + k p ≤ k 1 + . . . + k p ≤ d and p -i ≤ d -1 ends the proof.

Next, we show that the outputs of Algorithm 3 provide approximate separation hyperplanes for the first i coordinates ( x 1 , . . . , x i ) .

Lemma A.3. Fix δ, ξ ∈ (0 , 1) , 1 ≤ j ≤ i ≤ p and an oracle O S = ( O S, 1 , . . . , O S,p ) : C d → R d for accuracy ϵ > 0 . Suppose that O S takes values in the unit ball B d (0 , 1) . For any s ∈ [ i ] let P ( s ) l s ∈ I k s represent a bounded polyhedron with VolumetricCenter ( P ( s ) l s ) ∈ C k s . Denote x r = c ( P ( r ) l r ) for r ∈ [ i ] . Suppose that when running ApproxOracle δ,ξ,O S ( i, i, P (1) l 1 , . . . , P ( i ) l i ) , no successful vector was queried. Then, any vector x ⋆ = ( x ⋆ 1 , . . . , x ⋆ p ) ∈ C d such that B d ( x ⋆ , ϵ ) is contained in the successful set satisfies

Proof. For i ≤ r ≤ p and j ≤ r , we use the notation

Using Lemma A.2, we always have for j ∈ [ r ] ,

Also, observe that by Lemma A.1 the recursive outputs of ApproxOracle always have norm bounded by one.

Next, let T l i +1 ,...,l r -1 be the set of indices corresponding to coordinates of λ l i +1 ,...,l r -1 for which the procedure ApproxOracle did not call for a levelr computation. These correspond to 1. constraints from the initial cube P 0 , or 2. cases when the volumetric center was out of the unit cube (l.6-7 of Algorithm 1) and as a result, the index of the added constraint was -1 instead of the current iteration index t . Similarly as above, for any t ∈ T l i +1 ,...,l r -1 , we denote by g l i +1 ,...,l r -1 ,t r the corresponding

vector a t . We recall that by construction, this vector is of the form ± e j for some j ∈ [ k r ] . Then, from Lemma 4.1, since the responses of the oracle always have norm bounded by one, for all y r ∈ C k r ,

For conciseness, we use the shorthand ( S ∪ T ) l i +1 ,...,l r -1 := S l i +1 ,...,l r -1 ∪ T l i +1 ,...,l r -1 , which contains all indices from coordinates of λ l i +1 ,...,l r -1 . In particular,

We now proceed to estimate the precision of the vectors G ( i, j, x 1 , . . . , x i ) as approximate separation hyperplanes for coordinates ( x 1 , . . . , x i ) . Let x ⋆ ∈ C d such that B d ( x ⋆ , ϵ ) is within the successful set. Then, for any choice of l i +1 ∈ S , . . . , l p ∈ S l i +1 ,...,l p -1 , since we did not query a successful vector, we have for all z ∈ B d ( x ⋆ , ϵ ) ,

As a result, because the responses from O S have unit norm,

Now write x ⋆ = ( x ⋆ 1 , . . . , x ⋆ p ) . In addition to the previous equation, for l i +1 ∈ S , . . . , l r -1 ∈ S l i +1 ,...,l r -2 and any l r ∈ T l i +1 ,...,l r -1 , one has ( g l i +1 ,...,l r r ) ⊤ x ⋆ r +1 ≥ ϵ , because x ⋆ is within the cube C d and at least at distance ϵ from the constraints of the cube. Similarly as when l r ∈ S l i +1 ,...,l r -1 , for any l r ∈ T l i +1 ,...,l r -1 we denote by c l i +1 ,...,l r the volumetric center of the polyhedron P ( r ) l r along the corresponding computation path, if l r corresponded to an added constraints when c l i +1 ,...,l r / ∈ C k r . Otherwise, if l r corresponded to the constraint a = ± e j of the initial cube, we pose c l i +1 ,...,l r = -a . Now by construction, in both cases one has ( g l i +1 ,...,l r r ) ⊤ c l i +1 ,...,l r ≤ -1 (l.7 of Algorithm 1). Thus,

Recalling Eq (4), we then sum all equations of the form Eq (5) and Eq (6) along the computation path, to obtain

Now using the convention

for any l i +1 ∈ S , . . . , l r -1 ∈ S l i +1 ,...,l r -2 , we can write

We next relate the terms G to the output of ApproxOracle . For simplicity, let us write G = ( P (1) l 1 , . . . , P ( i ) l i ) , which by abuse of notation was assimilated to ( x 1 , . . . , x i ) . Recall that by construction and hypothesis, all points where the oracle was queried belong to C d , so that for instance

∥ x ⋆ r -c l i +1 ,...,l r ∥ ≤ 2 √ k r ≤ 2 √ d for any l r ∈ S l i +1 ,...,l r -1 . Using the above equations together with Eq (2) and Lemma A.2 gives

where in the second inequality, we used Eq (3). Using p ≤ d , this ends the proof of the lemma.

We are now ready to show that Algorithm 4 is a valid algorithm for convex optimization.

Theorem A.1. Let ϵ ∈ (0 , 1) and O S : C d → R d be a separation oracle such that the successful set contains a ball of radius ϵ . Pose δ = ϵ 4 d and ξ = σ min ϵ 32 d 5 / 2 . Next, let p ≥ 1 and k 1 , . . . , k p ≤ ⌈ d p ⌉ such that k 1 + . . . + k p = d . With these parameters, Algorithm 4 finds a successful vector with ( C d p ln d ϵ ) p queries and using memory O ( d 2 p ln d ϵ ) , for some universal constant C > 0 .

Proof. Suppose by contradiction that Algorithm 4 never queried a successful point. Then, with the chosen parameters, Lemma A.3 shows that, for any vector x ⋆ = ( x ⋆ 1 , . . . , x ⋆ p ) such that B d ( x ⋆ , ϵ ) is within the successful set, with the same notations, one has

Now denote by ( a t , b t ) the constraints that were added at any time during the run of Algorithm 1 when using the oracle ApproxOracle with i = j = 1 . The previous equation shows that for all such constraints,

where ω t is the volumetric center of the polyhedron at time t during Vaidya's method Algorithm 1. Now, since the algorithm terminated, by Lemma 4.1, we have that

This is absurd since δ + ξ < ϵ 2 . This ends the proof that Algorithm 4 finds a successful vector.

We now estimate its oracle-complexity and memory usage. First, recall that a run of ApproxOracle of level i makes O ( k i +1 ln 1 δ ) calls to level-( i +1) runs of ApproxOracle . As a result, the oraclecomplexity Q d ( ϵ ; k 1 , . . . , k p ) satisfies

for some universal constants C, C ′ ≥ 2 .

We now turn to the memory of the algorithm. For each level i ∈ [ p ] of runs for ApproxOracle , we keep memory placements for

the value j ( i ) of the corresponding call to ApproxOracle ( i, j ( i ) , · ) (for l.6-7 of Algorithm 3): O (ln d ) bits,

the iteration number t ( i ) during the run of Algorithm 1 or within Algorithm 2: O (ln( k i ln 1 δ )) bits

the polyhedron constraints contained in the state of P ( i ) : O ( k i × k i ln 1 ξ ) bits,

i

1

. . .

p

j

j

(1)

j

(

p

)

Iteration index

t

(1)

t

(

p

)

Polyhedron

P

(1)

=







k

1

,

a

1

, b

1

k

2

,

a

2

, b

2

. . .

k

m

,

a

m

, b

m





P

(

p

)

Computed

dual variables

(

k

⋆

(1)

,

λ

⋆

(1)

) =

(

k

⋆

1

, λ

⋆

1

k

⋆

2

, λ

⋆

2

. . .

)

(

k

⋆

(

p

)

,

λ

⋆

(

p

)

)

Working separation vector

u

(1)

u

(

p

)

potentially, already computed dual variables λ ⋆ and their corresponding vector of constraint indices k ⋆ (l.3 of Algorithm 2): O ( k i × ln 1 ξ ) bits,

the working vector u ( i ) (updated l.8 of Algorithm 2): O ( k i ln 1 ξ ) bits.

The memory structure is summarized in Table 1.

We can then check that this memory is sufficient to run Algorithm 4. An important point is that for any run of ApproxOracle ( i, j, · ) , in Algorithm 2, after running Vaidya's method Algorithm 1 and storing the dual variables λ ⋆ and corresponding indices k ⋆ within their placements ( k ⋆ ( i ) , λ ⋆ ( i ) ) (l.1-3 of Algorithm 2), the iteration index t ( i ) and polyhedron P ( i ) memory placements are reset and can be used again for the second run of Vaidya's method (l.4-10 of Algorithm 2). During this second run, the vector u is stored in its corresponding memory placement u ( i ) and updated along the algorithm. Once this run is finished, the output of ApproxOracle ( i, j, · ) is readily available in the placement u ( i ) . For i = p , the algorithm does not need to wait for the output of a level-( i +1) computation and can directly use the j ( p ) -th component of the returned separation vector from the oracle O S . As a result, the number of bits of memory used throughout the algorithm is at most

This ends the proof of the theorem.

We can already give the useful range for p for our algorithms, which will also apply to the case with computational-memory constraints Appendix B.

Proof of Corollary 3.1. Suppose ϵ ≥ 1 d d . Then, for some p max = Θ( C ln 1 ϵ 2 ln d ) ≤ d , the algorithm from Theorem 3.2 yields a O ( 1 ϵ 2 ) oracle-complexity. On the other hand, if ϵ ≤ 1 d d , we can take p max = d , which gives an oracle-complexity O (( C ln 1 ϵ ) d ) .

B Memory-constrained feasibility problem with computations

In the last section we gave the main ideas that allow reducing the storage memory. However, Algorithm 4 does not account for memory constraints in computations as per Definition 2.2. For instance, computing the volumetric center VolumetricCenter ( P ) already requires infinite memory for infinite precision. More importantly, even if one discretizes the queries, the necessary precision and computational power may be prohibitive with the classical Vaidya's method Algorithm 1. Even finding a feasible point in the polyhedron (let alone the volumetric center) using only the constraints is itself computationally intensive. There has been significant work to make Vaidya's method computationally tractable [48, 1, 2]. These works address the issue of computational tractability, but the memory issue

is still present. Indeed, the precision depends among other parameters on the condition number of the matrix H in order to compute the leverage scores σ i for i ∈ [ m ] , which may not be well-conditioned. Second, to avoid memory overflow, we also need to ensure that the points queried have bounded norm, which is again not a priori guaranteed in the original version Algorithm 1.

To solve these issues and also give a computationally-efficient algorithm, the cutting-plane subroutine Algorithm 1 needs to be modified. In particular, the volumetric barrier needs to include regularization terms. Fortunately, these have already been studied in [27]. In a major breakthrough, this paper gave a cutting-plane algorithm with O ( d 3 ln O (1) d ϵ ) runtime complexity, improving over the seminal work from Vaidya and subsequent works which had O ( d 1+ ω ln O (1) d ϵ ) runtime complexity, where O ( d ω ) is the computational complexity of matrix multiplication. To achieve this result, they introduce various regularizing terms together with the logarithmic barrier. While the main motivation of [27] was computational complexity, as a side effect, these regularization terms also ensure that computations can be carried with efficient memory. We then use their method as a subroutine.

For the sake of exposition and conciseness, we describe a simplified version of their method, that is also deterministic. This comes at the expense of a suboptimal running time O ( d 1+ ω ln O (1) 1 ϵ ) . We recall that our main concern is in memory usage rather than achieving the optimal runtime. The main technicality of this section is to show that their simplified method is numerically stable, and we emphasize that the original algorithm could also be shown to be numerically stable with similar techniques, leading to a time improvement from ˜ O ( d 1+ ω ) to ˜ O ( d 3 ) . The memory usage, however, would not be improved.

B.1 A memory-efficient Vaidya's method for computations, via [27]

Fix a polyhedron P = { x : Ax ≥ b } . Using the same notations as for Vaidya's method in Section 4.1, we define the new leverage scores ψ ( x ) i = ( A x ( A ⊤ x A x + λ I ) -1 A ⊤ x ) i,i and Ψ ( x ) = diag ( ψ ( x )) . Let µ ( x ) = min i ψ ( x ) i . Last, let Q ( x ) = A ⊤ x ( c e I + Ψ ( x )) A x + λ I , where c e > 0 is a constant parameter to be defined. In [27], they consider minimizing the volumetric-analytic hybrid barrier function

We can check [27] that

where 1 is the vector of ones. The following procedure gives a way to minimize this function efficiently given a good starting point.

We then present their simplified cutting-plane method.

In both Algorithm 5 and Algorithm 6, notice that the updates require to compute in particular the leverage scores ψ ( x ) , which can be computed in O ( d ω ) time using their formula. To achieve the O ( d 3 ln O (1) 1 ϵ ) computational complexity, an amortized computational cost O ( d 2 ) is needed. The algorithm from [27] achieves this through various careful techniques aiming to update estimates of these leverage scores. The above cutting-plane algorithm is exactly that of [27] when these estimates are always exact (i.e. recomputed at each iteration), which yields the d ω -2 overhead time complexity. In particular, the original proof of convergence and correctness of [27] directly applies to this simplified algorithm.

It remains to check whether one can implement this algorithm with efficient memory, corresponding to checking this method's numerical stability.

Lemma B.1. Suppose that each iterate of the centering Algorithm 5, ∥∇ p ( x ( k -1) ) ∥ Q ( x (0) ) -1 is computed up to precision (1 -1 64 ) r η (l.2), and x ( k ) is computed up to an error ζ ( k ) with ∥ ζ ( k ) ∥ Q ( x (0) ) ≤ 1 2 10 r (1 -1 64 ) r η (l.3). Then, Algorithm 5 outputs x ( k ) such that ∥∇ p ( x ( k ) ) ∥ Q -1 ( x ( k ) ) ≤ 3(1 -1 64 ) r η and all iterates computed during the procedure satisfy ∥ S -1 x (0) ( s ( x ( t ) ) -s ( x (0) )) ∥ 2 ≤ 1 10 .

Proof. As mentioned above, without computation errors, the result from [27] would apply directly. Here, we simply adapt the proof to the case with computational errors to show that it still applies. Denote Q = Q ( x (0) ) for convenience. Let η = 1 100 √ c e + µ ( x (0) ) . We prove by induction that ∥ x ( t ) -x (0) ∥ Q ≤ 9 η , ∥∇ p ( x ( t ) ) ∥ Q -1 ≤ (1 -1 64 ) t η for all t ≤ r . For a given iteration t , denote ˜ x ( t +1) = x ( k -1) -1 8 Q -1 ∇ p ( x ( k -1) ) the result of the exact computation. The same arguments as in the original proof give ∥ ˜ x ( t +1) -x (0) ∥ Q ≤ 9 η , and

Now because ∥ ˜ x ( t +1) -x ( t +1) ∥ Q ≤ η , we have ∥ ˜ x ( t +1) -x (0) ∥ Q , ∥ x ( t +1) -x (0) ∥ Q ≤ 10 η , so that [27, Lemma 11] gives ∇ 2 p ( y ( u )) ⪯ 8 Q ( y ( u )) ⪯ 16 Q , where y ( u ) = x ( t +1) + u (˜ x ( t +1) -x ( t +1) for u ∈ [0 , 1] . Thus,

Now by construction of the procedure, if the algorithm performed iteration t + 1 , we have ∥∇ p ( x ( t ) ) ∥ Q -1 ≥ (1 -1 64 ) r η . Combining this with the fact that ∥ ˜ x ( t +1) -x ( t +1) ∥ Q ≤ 1 2 10 r (1 -1 64 ) r η , obtain

We now write

The induction is now complete. When the algorithm stops, either the r steps were performed, in which case the induction already shows that ∥∇ p ( x ( r ) ) ∥ Q -1 ≤ (1 -1 64 ) r η . Otherwise, if the algorithm terminates at iteration k , because ∥∇ p ( x ( k ) ) ∥ Q -1 was computed to precision (1 -1 64 ) r η , we have (see l.2 of Algorithm 5)

The same argument as in the original proof shows that at each iteration t ,

This ends the proof of the lemma.

Because of rounding errors, Lemma B.1 has an extra factor 3 compared to the original guarantee in [27, Lemma 14]. To achieve the same guarantee, it suffices to perform 70 ≥ ln(3) / ln(1 / (1 -1 64 )) additional centering procedures at most. hence, instead of performing 200 centering procedures during the cutting plane method, we perform 270 (l.10 of Algorithm 6). We next turn to the numerical stability of the main Algorithm 6.

Lemma B.2. Suppose that throughout the algorithm, when checking the stopping criterion min i ∈ [ m ] s i ( x ) < 2 ϵ , the quantities s i ( x ) were computed with accuracy ϵ . Suppose that at each iteration of Algorithm 6, the leverage scores ψ ( x ( t ) ) are computed up to multiplicative precision c ∆ / 4 (l.3), that when a constraint is added, the response of the oracle a (l.7) is stored perfectly but b (l.8) is computed up to precision Ω( ϵ √ n ) . Further suppose that the centering Algorithm 5 is run with numerical approximations according to the assumptions in Lemma B.1. Then, all guarantees for the original algorithm in [27] hold, up to a factor 3 for ϵ .

Proof. We start with the termination criterion. Given the requirement on the computational accuracy, we know that the final output x satisfies min i ∈ [ m ] s i ( x ) ≤ 3 ϵ . Further, during the algorithm, if it does not stop, then one has min i ∈ [ m ] s i ( x ) ≥ ϵ , which is precisely the guarantee of the original algorithm in [27].

We next turn to the computation of the leverage scores in l.4. In the original algorithm, only a c ∆ -estimate is computed. Precisely, one computes a vector w ( t ) such that for all i ∈ [ d ] , ψ ( x ( t ) ) i ≤ w i ≤ (1 + c ∆ ) ψ ( x ( t ) ) i , then deletes a constraint when min i ∈ [ m ( t ) ] w ( t ) i ≤ c d . In the adapted algorithm, let ˜ ψ ( x ( t ) ) i denote the computed leverage scores for i ∈ [ d ] . By assumption, we have

Up to re-defining the constant c d as (1 -c ∆ / 4) c d , ˜ ψ ( x ( t ) ) is precisely within the guarantee bounds of the algorithm. For the accuracy on the separation oracle response and the second-term value b , [27] emphasizes that the algorithm always changes constraints by a δ amount where δ = Ω( ϵ √ d ) so that an inexact separation oracle with accuracy Ω( ϵ √ d ) suffices. Therefore, storing an Ω( ϵ √ d ) accuracy of the second term keeps the guarantees of the algorithm. Last, we checked in Lemma B.1 that the centering procedure Algorithm 5 satisfies all the requirements needed in the original proof [27].

For our recursive method, we need an efficient cutting-plane method that also provides a proof (certificate) of convergence. This is also provided by [27] that provide a proof that the feasible region has small width in one of the directions a i of the returned polyhedron.

Lemma B.3. [27, Lemma 28] Let ( P , x , ( λ i ) i ) be the output of Algorithm 7. Then, x is feasible, ∥ x ∥ 2 ≤ 3 √ d , λ j ≥ 0 for all j and ∑ i λ i = 1 . Further,

We are now ready to show that Algorithm 6 can be implemented with efficient memory and also provides a proof of the convergence of the algorithm.

Input: ϵ > 0 and a separation oracle O : C d → R d

1 Run Algorithm 6 to obtain a polyhedron P and a feasible point x

2 x ⋆ = Centering ( x , 64 ln 2 ϵ , c ∆ )

Output: ( P , x ⋆ , ( λ i ) i )

Algorithm 7: Cutting-plane algorithm with certified optimality

Proposition B.1. Provided that the output of the oracle are vectors discretized to precision poly ( ϵ d ) and have norm at most 1 , Algorithm 7 can be implemented with O ( d 2 ln d ϵ ) bits of memory to output a certified optimal point according to Lemma B.3. The algorithm performs O ( d ln d ϵ ) calls to the separation oracle and runs in O ( d 1+ ω ln O (1) d ϵ ) time.

Proof. We already checked the numerical stability of Algorithm 6 in Lemma B.2. It remains to check the next steps of the algorithm. The centering procedure is stable again via Lemma B.1. It also suffices to compute the coefficients λ j up to accuracy O ( ϵ/ ( √ d ) ln( d/ϵ )) to keep the guarantees desired since by construction all vectors a i have norm at most one.

It now remains to show that the algorithm can be implemented with efficient memory. We recall that at any point during the algorithm, the polyhedron P has at most O ( d ) constraints [27, Lemma 22]. Hence, since we assumed that each vector a i composing a constraint is discretized to precision poly ( ϵ d ) , we can store the polyhedron constraints with O ( d 2 ln d ϵ ) bits of memory. The second terms b are computed up to precision Ω( ϵ/ √ d ) hence only use O ( d ln d ϵ ) bits of memory. The algorithm also keeps the current iterate x ( t ) in memory. These are all bounded throughout the memory ∥ x ( t ) ∥ 2 = O ( √ d ) [27, Lemma 23], hence only require O ( d ln d ϵ ) bits of memory for the desired accuracy.

Next, the distances to the constraints are bounded at any step of the algorithm: s i ( x ( t ) ) ≤ O ( √ d ) [27, Lemma 24], hence computing s i ( x ( t ) ) to the required accuracy is memory-efficient. Recall that from the termination criterion, except for the last point, any point x during the algorithm satisfies s i ( x ) ≥ ϵ for all constraints i ∈ [ m ] . In particular, this bounds the eigenvalues of Q since λI ⪯ Q ( x ) ⪯ ( λ + m ( c e + 1) /ϵ 2 ) I . Thus, the matrix is sufficiently well-conditioned to achieve the accuracy guarantees from Lemma B.1 using O ( d 2 ln d ϵ ) memory during matrix inversions (and matrix multiplications). Similarly, for the computation of leverage scores, we use Ψ ( x ) = diag ( A x ( A ⊤ x A x + λ I ) -1 A ⊤ x ) , where λ I ⪯ A ⊤ x A x + λ I ⪯ ( λ + mϵ -2 ) I . This same matrix inversion appears when computing the second term of an added constraint. Overall, all linear algebra operations are well conditioned and implementable with required accuracy with O ( d 2 ln d ϵ ) memory. Using fast matrix multiplication, all these operations can be performed in ˜ O ( d ω ) time per iteration of the cutting-plane algorithm since these methods are also known to be numerically stable [15]. Thus, the total time complexity is O ( d 1+ ω ln O (1) d ϵ ) . The oracle-complexity still has optimal O ( d ln d ϵ ) oracle-complexity as in the original algorithm.

Up to changing ϵ to c · ϵ/ ( d ln d ϵ ) , the described algorithm finds constraints given by a i and b i , i ∈ [ m ] returned by the normalized separation oracle, coefficients λ i , i ∈ [ m ] , and a feasible point x ⋆ such that for any vector in the unit cube, z ∈ C d , one has

This effectively replaces Lemma 4.1.

B.2 Merging Algorithm 7 within the recursive algorithm

Algorithms 2 to 4 from the recursive procedure need to be slightly adapted to the new format of the cutting-plane method's output. In particular, the oracles do not take as input polyhedrons (and

eventually query their volumetric center as before), but directly take as input an point (which is an approximate volumetric center).

The same proof as for Algorithm 4 shows that Algorithm 10 run with the parameters in Theorem A.1 also outputs a successful vector using the same oracle-complexity. We only need to analyze the memory usage in more detail.

Proof of Theorem 3.2. As mentioned above, we will check that Algorithm 10 with the same parameters δ = ϵ 4 d and ξ = σ min ϵ 32 d 5 / 2 as in Theorem A.1 satisfies the desired requirements. We have already checked its correctness and oracle-complexity. Using the same arguments, the computational complexity is of the form O ( O ( ComplexityCuttingPlanes ) p ) where ComplexityCuttingPlanes is the computational complexity of the cutting-plane method used, i.e., here of Algorithm 7. Hence, the computational complexity is O (( C ( d/p ) 1+ ω ln O (1) d ϵ ) p ) for some universal constant C ≥ 2 . We now turn to the memory. In addition to the memory of Algorithm 4, described in Table 1, we need

a placement for all i ∈ [ p ] for the current iterate x ( i ) : O ( k i ln 1 ξ ) bits,

a placement for computations, that is shared for all layers (used to compute leverage scores, centering procedures, etc. By Proposition B.1, since the vectors are always discretized to precision ξ , this requires O (max i ∈ [ p ] k 2 i ln d ϵ ) bits,

the placement Q to perform queries is the concatenation of the placements ( x (1) , . . . , x ( p ) ) : no additional bits needed.

a placement N to store the precision needed for the oracle responses: O (ln 1 ξ ) bits

a placement R to receive the oracle responses: O ( d ln 1 ξ ) bits.

The new memory structure is summarized in Table 2.

With the same arguments as in the original proof of Theorem A.1, this memory is sufficient to run the algorithm and perform computations, thanks to the computation placement. The total number of bits used throughout the algorithm remains the same, O ( d 2 p ln d ϵ ) . This ends the proof of the theorem.

i

1

. . .

p

Oracle response

Precision

j

j

(1)

j

(

p

)

R

= (

R

1

, . . . , R

p

)

N

Iteration index

t

(1)

t

(

p

)

Polyhedron

P

(1)

=







k

1

,

a

1

, b

1

k

2

,

a

2

, b

2

. . .

k

m

,

a

m

, b

m





P

(

p

)

Computation

memory

Current

iterate

x

(1)

x

(

p

)

Computed

dual variables

(

k

⋆

,

λ

⋆

) =

(

k

⋆

1

, λ

⋆

1

k

⋆

2

, λ

⋆

2

. . .

)

(

k

⋆

(

p

)

,

λ

⋆

(

p

)

)

Working

separation vector

u

(1)

u

(

p

)

C Improved oracle-complexity/memory lower-bound trade-offs

We recall the three oracle-complexity/memory lower-bound trade-offs known in the literature.

First, [31] showed that any (including randomized) algorithm for convex optimization uses d 1 . 25 -δ memory or makes ˜ Ω( d 1+4 δ/ 3 ) queries.

Then, [5] showed that any deterministic algorithm for convex optimization uses d 2 -δ memory or makes ˜ Ω( d 1+ δ/ 3 ) queries.

Last, [5] show that any deterministic algorithm for the feasibility problem uses d 2 -δ memory or makes ˜ Ω( d 1+ δ ) queries.

Although these papers mainly focused on the regime ϵ = 1 / poly ( d ) and as a result ln 1 ϵ = O (ln d ) , neither of these lower bounds have an explicit dependence in ϵ . This can lead to sub-optimal lower bounds whenever ln 1 ϵ ≫ ln d . Furthermore, in the exponential regime ϵ ≤ 1 2 O ( d ) , these results do not effectively give useful lower bounds. Indeed, in this regime, one has d 2 = O ( d ln 1 ϵ ) and as a result, the lower bounds provided are weaker than the classical Ω( d ln 1 ϵ ) lower bounds for oracle-complexity [34] and memory [52]. In particular, in this exponential regime, these results fail to show that there is any trade-off between oracle-complexity and memory.

In this section, we aim to explicit the dependence in ϵ of these lower-bounds. We show with simple modifications and additional arguments that one can roughly multiply these oracle-complexity and memory lower bounds by a factor ln 1 ϵ each. We split the proofs in two. First we give arguments to improve the memory dependence by a factor ln 1 ϵ , which is achieved by modifying the sampling of the rows of the matrix A defining a wall term common to the functions considered in the lower bound proofs [31, 5]. Then we show how to improve the oracle-complexity dependence by an additional ln 1 ϵ / ln d factor, via a standard rescaling argument.

C.1 Improving the memory lower bound

We start with some concentration results on random vectors. [31] gave the following result for random vectors in the hypercube.

Lemma C.1 ([31]) . Let h ∼ U ( {± 1 } d ) . Then, for any t ∈ (0 , 1 / 2] and any matrix Z = [ z 1 , . . . , z k ] ∈ R d × k with orthonormal columns,

Instead, we will need a similar concentration result for random unit vectors in the unit sphere.

Lemma C.2. Let k ≤ d and x 1 , . . . , x k be k orthonormal vectors, and ζ ≤ 1 .

Proof. First, by isometry, we can suppose that the orthonormal vectors are simply e 1 , . . . , e k . We now prove the result by induction on d . For d = 1 , the result holds directly. Fix d ≥ 2 , and 1 ≤ k < d . Then, if S n is the surface area of S n the n -dimensional sphere, then

Conditionally on the value of y 1 , the vector ( y 2 , . . . , y d ) follows a uniform distribution on the ( d -2) -sphere of radius √ 1 -y 2 1 . Then,

Now recall that since | x 1 | ≤ 1 / √ d , we have d (1 -x 2 1 ) ≥ d -1 . Therefore, using the induction,

Combining this equation with Eq (7) ends the proof.

We next use the following lemma to partition the unit sphere S d -1 .

Lemma C.3 ([19] Lemma 21) . For any 0 < δ < π/ 2 , the sphere S d -1 can be partitioned into N ( δ ) = ( O (1) /δ ) d equal volume cells, each of diameter at most δ .

Following the notation from [5], we denote by V δ = { V i ( δ ) , i ∈ [ N ( δ )] } the corresponding partition, and consider a set of representatives D δ = { b i ( δ ) , i ∈ [ N ( δ )] } ⊂ S d -1 such that for all i ∈ [ N ( δ )] , b i ( δ ) ∈ V i ( δ ) . With these notations we can define the discretization function ϕ δ as follows

We then denote by U δ the distribution of ϕ δ ( z ) where z ∼ U ( S d -1 ) is sampled uniformly on the sphere. Note that because the cells of V δ have equal volume, U δ is simply the uniform distribution on the discretization D δ .

We are now ready to give the modifications necessary to the proofs, to include a factor ln 1 ϵ for the necessary memory. For their lower bounds, [31] exhibit a distribution of convex functions that are hard to optimize. Building upon their work [5] construct classes of convex functions that are hard to optimize, but that also depend adaptively on the considered optimization algorithm. For both, the functions considered a barrier term of the form ∥ Ax ∥ ∞ , where A is a matrix of ≈ d/ 2 rows that are independently drawn as uniform on the hypercube U ( {± 1 } d ) . The argument shows that memorizing A is necessary to a certain extent. As a result, the lower bounds can only apply for a memory of at most O ( d 2 ) bits, which is sufficient to memorize such a binary matrix. Instead, we draw rows independently according to the distribution U δ , where δ ≈ ϵ . We explicit the corresponding adaptations for each known trade-off. We start with the lower bounds from [5] for ease of exposition; although these build upon those of [31], their parametrization makes the adaptation more straightforward.

C.1.1 Lower bound of [5] for convex optimization and deterministic algorithms

For this lower bound, we use the exact same form of functions as they introduced,

with the difference that rows of A are take i.i.d. distributed according to U δ ′ instead of U ( {± 1 } d ) . As a remark, they use n = ⌈ d/ 4 ⌉ rows for A . Except for η , we keep all parameters γ 1 , γ 2 , etc as in the original proof, and we will take δ ′ = ϵ and η = 2 √ dϵ . The reason why we introduced δ ′ instead of δ is that the original construction also needs the discretization ϕ δ . This is used during the optimization procedure which constructs adaptively this class of functions, and only needs δ = poly (1 /d ) instead of δ of order ϵ .

Theorem C.1. For ϵ ≤ 1 / (2 d 4 . 5 ) and any δ ∈ [0 , 1] , a deterministic first-order algorithm guaranteed to minimize 1 -Lipschitz convex functions over the unit ball with ϵ accuracy uses at least d 2 -δ ln 1 ϵ bits of memory or makes ˜ Ω( d 1+ δ/ 3 ) queries.

With the changes defined above, we can easily check that all results from [5] which reduce convex optimization to the optimization procedure, then the optimization procedure to their Orthogonal Vector Game with Hints (OVGH) [5, Game 2], are not affected by our changes. The only modifications to perform are to the proof of query lower bound for the OVGH [5, Proposition 14]. We emphasize that the distribution of A is changed in the optimization procedure but also in OVGH as a result.

Proposition C.2. Let k ≥ 20 M +3 d log(2 d )+1 n log 2 ( √ 2( ζ + δ ′ √ d )) -1 . And let 0 < α, β ≤ 1 such that α ( √ d/β ) 5 / 4 ≤ ζ/ √ d where ζ ≤ 1 . If the Player wins the adapted OVGH with probability at least 1 / 2 , then m ≥ 1 8 (1 + 30 log 2 d log 2 ( √ 2( ζ + δ ′ √ d )) -1 ) -1 d .

Proof. We use the same proof and only highlight the modifications. The proof is unchanged until the step when the concentration result Lemma C.1 is used. Instead, we use Lemma C.2. With the same notations as in the original proof, we constructed ⌈ k/ 5 ⌉ orthonormal vectors Z = [ z 1 , . . . , z ⌈ k/ 5 ⌉ ] such that all rows a of A ′ (which is A up to some observed and unimportant rows) one has

Hence, using the same arguments as in the original proof, we obtain

where E is the event when the algorithm succeeds at the OVGH game. In the next step, we need to bound H ( A | V ) -H ( G , j , c ) where V stores hints received throughout the game, G stores observed rows of A during the game, and j , c are auxiliary variables. The latter can be treated as in the original proof. We obtain

Now the same arguments as in the original proof show that we still have I ( A , V ) ≤ 3 km log 2 d +1 , and that as a result, if M is the number of bits stored in memory,

Then, with the same arguments as in the original proof, we can conclude.

We are now ready to prove Theorem C.1. With the parameter k = ⌈ 20 M +3 d log(2 d )+1 n log 2 ( √ 2( ϵd 4 / 2+ δ ′ √ d )) -1 ⌉ and the same arguments, we show that an algorithm solving the convex optimization up to precision η/ (2 √ d ) = ϵ yields an algorithm solving the OVGH where the parameters α = 2 η γ 1 and β = γ 2 4 satisfy

We can then apply Proposition C.2 with ζ = d 4 ϵ/ 2 . Hence, if Q is the maximum number of queries of the convex optimization algorithm, we obtain

where in the last inequality we used ϵ ≤ 1 / (2 d 4 . 5 ) . As a result, with the same arguments, we obtain

This ends the proof of Theorem C.1.

C.1.2 Lower bound of [5] for feasibility problems and deterministic algorithms

We improve the memory dependence by showing the following result.

Theorem C.3. For ϵ = 1 / (48 d 3 ) and any δ ∈ [0 , 1] , a deterministic algorithm guaranteed to solve the feasibility problem over the unit ball with ϵ accuracy uses at least d 2 -δ ln 1 ϵ bits of memory or makes at least ˜ Ω( d 1+ δ ) queries.

We use the exact same class of feasibility problems and only change the parameter η 0 which constrained successful points to satisfy ∥ Ax ∥ ∞ ≤ η 0 , as well as the rows of A that are sampled i.i.d. from U δ . The other parameter η 1 = 1 / (2 √ d ) is unchanged. We also take δ ′ = ϵ . Because the rows of A are already normalized, we can take η 0 = ϵ directly. Then, the same proof as in [5] shows that if an algorithm solves feasibility problems with accuracy ϵ , there is an algorithm for OVGH for parameters α = η/η 1 and β = η 1 / 2 . Then, we have α ( √ d/β ) 5 / 4 ≤ 12 d 2 η 0 and we can apply Proposition C.2 with ζ = 12 d 2 . 5 η 0 = 12 d 2 . 5 ϵ . Similar computations as above then show that m ≥ d/ (8 · 61) , with k = Θ( M +ln d d ln 1 ϵ ) , so that the query lower bound finally becomes

Remark C.1. The more careful analysis-involving the discretization D δ of the unit sphere at scale δ instead of the hypercube {± 1 } d -allowed to add a ln 1 ϵ factor to the final query lower bound but also an additional ln d factor for both convex-optimization and feasibility-problem results. Indeed, the improved Proposition C.2 shows that the OVGH with adequate parameters requires O ( d ) queries, instead of O ( d/ ln d ) in [5, Proposition 14]. At a high level, each hint queried brings information O ( d ln d ) but memorizing a binary matrix A ∈ {± 1 } ⌈ d/ 4 ⌉× d only requires d 2 bits of memory: hence the query lower bound is limited to O ( d/ ln d ) . Instead, memorizing the matrix A where each row lies in D δ requires Θ( d 2 ln 1 ϵ ) memory, hence querying d hints (total information O ( d 2 ln d ) ) is not prohibitive for the lower bound.

C.1.3 Lower bound of [31] for convex optimization and randomized algorithms

We aim to improve the result to obtain the following.

Theorem C.4. For ϵ ≤ 1 /d 4 and any δ ∈ [0 , 1] , any (potentially randomized) algorithm guaranteed to minimize 1 -Lipschitz convex functions over the unit ball with ϵ accuracy uses at least d 1 . 25 -δ ln 1 ϵ bits of memory or makes ˜ Ω( d 1+4 δ/ 3 ) queries.

The distribution considered in [31] is given by the functions

where N ≤ d is a parameter, A has ⌊ d/ 2 ⌋ rows drawn i.i.d. from U ( {± 1 } d ) , and the vectors v i are drawn i.i.d. from the rescaled hypercube v i ∼ U ( d -1 / 2 {± 1 } d ) . We adapt the class of functions by simply changing pre-factors as follows

where A has the same number of rows but they are draw i.i.d. from U δ , and δ, µ > 0 are parameters to specify. We use the notation µ instead of η as in the previous sections because [31] already use a parameter η which in our context can be interpreted as η = 1 / ( µ √ d ) . We choose the parameters µ = 16 √ dϵ and δ ′ = ϵ .

Again, as for the previous sections, the original proof can be directly used to show that if an algorithm is guaranteed to find a µ 16 √ N ( ≥ ϵ ) -suboptimal point for the above function class, there is an algorithm that wins at their Orthogonal Vector Game (OVG) [31, Game 1], with the only difference that the parameter d -4 (l.8 of OVG) is replaced by √ dµ . OVG requires the output to be robustly-independent (defined in [31]) and effectively corresponds to β = 1 /d 2 in OVGH. As a result, there is a successful algorithm for the OVGH with parameters α = √ dµ and β = 1 /d 2 and that even completely ignores the hints. Hence, we can now directly use Proposition C.2 with ζ = d 1+25 / 16 µ (from the assumption ϵ ≤ d -4 we have ζ ≤ 1 / √ d ). This shows that with the adequate choice of k = Θ( M + d ln d d ln 1 ϵ ) , the query lower bound is Ω( d ) .

Putting things together, a potentially randomized algorithm for convex optimization that uses M memory makes at least the following number of queries

C.2 Proof sketch for improving the query-complexity lower bound

We now turn to improving the query-complexity lower bound by a factor ln 1 ϵ ln d . At the high level, the idea is to replicate these constructed 'difficult' class of functions at ln 1 ϵ ln d different scales or levels, similarly to the manner that the historical Ω( d ln 1 ϵ ) lower bound is obtained for convex optimization [34]. This argument is relatively standard and we only give details in the context of improving the bound from [31] for randomized algorithms in convex optimization for conciseness. This result uses a simpler class of functions, which greatly eases the exposition. We first present the construction with 2 levels, then present the generalization to p = Θ( ln 1 ϵ ln d ) levels. For convenience, we write

This is the query lower bound given in Theorem C.5 for convex optimization algorithms with memory M that optimize the defined class of functions (Eq (8)) to accuracy ϵ .

C.2.1 Construction of a bi-level class of functions F A , v 1 , v 2 to optimize

In the lower-bound proof, [31] introduce the point

where P A ⊥ is the projection onto the orthogonal space to the rows of A . They show that with failure probability at most 2 /d , ¯ x has good function value

x

G

A

,

v

1

(

x

)

¯

x

µξ

2

3

G

A

,

v

1

(¯

x

)

+

µξ

2

3

(1 +

∥

x

-

¯

x

∥

2

)

ξ

2

3

µξ

2

·

2

ξ

2

9

G

A

,

v

1

(¯

x

) +

µξ

2

3

+

µξ

2

2

18

+

µξ

2

2

54

i

∈

[

N

]

(

v

⊤

2

,i

(

x

-

¯

x

ξ

2

/

9

)

-

iγ

)

This is shown in [31, Lemma 25]. On the other hand, from Theorem C.4, during the first

queries of any algorithm, with probability at least 1 / 3 , all queries are at least µ/ (16 √ N ) -suboptimal compared to ¯ x in function value [31, Theorem 28, Lemma 14 and Theorem 16]. Precisely, if F A , v is the sampled function to optimize, with probability at least 1 / 3 ,

As a result, we can replicate the term max i ∈ [ N ] ( v ⊤ i x -iγ ) at a smaller scale within the ball B d (¯ x , 1 / (16 √ d )) . For convenience, we introduce ξ 2 = 1 / (16 √ d ) which will be the scale of the duplicate function. We separate the wall term ∥ Ax ∥ ∞ -µ for convenience. Hence, we define

An illustration of the construction is given in Fig. 4. The resulting optimization functions are given by adding the wall term:

We first explain the choice of parameters. First observe that since ∥ A ¯ x ∥ = 0 , we have G A , v 1 (¯ x ) = F A , v 1 (¯ x ) . We can then check that for all x ∈ B d (0 , 1) ,

Further, for any x ∈ B d (¯ x , ξ 2 / 3) , since F A , v 1 is 1 -Lipschitz, we can easily check that

Thus, G A , v 1 , v 2 ( x ) does not coincide with G A , v 1 ( x ) on B d (¯ x , ξ 2 / 3) . Then, the ∥ x -¯ x ∥ 2 term ensures that any minimizer of G A , v 1 , v 2 is contained within the closed ball B d (¯ x , ξ 2 / 3) . Also, to obtain a µξ 2 / 3 -suboptimal solution of F A , v 1 , v 2 , the algorithm needs to find what would be a µξ 2 -suboptimal solution of F A , v 1 , while receiving the same response as when optimizing the latter.

Next, for any x ∈ B d (¯ x , ξ 2 / 9) , the term max i ∈ [ N ] ( v ⊤ 2 ,i ( x -¯ x ξ 2 / 9 ) -iγ ) lies in [ -1 , 1] . Hence, we can check that for x ∈ B d (¯ x , ξ 2 / 9) ,

We now argue that F A , v 1 , v 2 acts as a duplicate function. Until the algorithm reaches a point with function value at most G A , v 1 (¯ x )+ µξ 2 , the optimization algorithm only receives responses consistent with the function F A , v 1 by Eq (9). Next, all minimizers of F A , v 1 , v 2 are contained in B d (¯ x , ξ 2 / 3) , which was the goal of introducing the term in ∥ x -¯ x ∥ 2 . As a result, optimizing F A , v 1 , v 2 on this ball is equivalent to minimizing

where y = x -¯ x ξ 2 / 9 . The function has been rescaled by a factor ξ 2 / 9 compared to F A , v 1 , v 2 so that µ 2 = 9 µ ξ 2 , ν 2 = µξ 2 6 , ν ′ 2 = 6 µ , c 2 = 9 ξ 2 G A , v 1 (¯ x ) + 3 µ + µξ 2 2 , and c ′ 2 = 9 ξ 2 G A , v 1 (¯ x ) + 3 µ . By Eq (10), the two first terms of ˜ F A , v 1 are preponderant for y ∈ B d (0 , 1) .

The form of ˜ F A , v 2 is very similar to the original form of functions

In fact, the same proof structure for the query-complexity/memory lower-bound can be applied in this case. The main difference is that originally one had µ ′ 1 = µ ′ 2 ; here we would instead have µ ′ 1 = µ 2 + c 2 = Θ( µ/ξ 2 ) and µ ′ 2 = ν 2 = Θ( µξ 2 ) . Intuitively, this corresponds to increasing the accuracy to Θ( ϵξ 2 2 ) -a factor ξ 2 is due to the fact that ˜ F A , v 2 was rescaled by a factor ξ 2 / 9 compared to F A , v 1 , v 2 , and a second factor ξ 2 is due to the fact that within ˜ F A , v 2 , we have µ ′ 2 = Θ( µξ 2 ) -while the query lower bound is similar to that obtained for Θ( ϵ/ξ 2 ) . As a result, during the first

queries of any algorithm optimizing ˜ F A , v 2 , with probability at least 1 / 3 on the sample of A and v 2 , all queries are at least Θ( ϵξ 2 ) -suboptimal compared to

We are now ready to give lower bounds on the queries of an algorithm minimizing F A , v 1 , v 2 to accuracy Θ( ϵξ 2 2 ) . Let T 2 be the index of the first query with function value at most G A , v 1 (¯ x ) + µξ 2 . We already checked that before that query, all responses of the oracle are consistent with minimizing F A , v 1 , hence on an event E 1 of probability at least 1 / 3 , one has T 2 ≥ Q 1 . Next, consider the hypothetical case when at time T 2 , the algorithm is also given the information of ¯ x and is allowed to store this vector. Given this information, optimizing F A , v 1 , v 2 reduces to optimizing ˜ F A , v 2 since we already know that the minimum is achieved within B d (¯ x , ξ 2 / 3) . Further, any query outside of this ball either

returns a vector v 1 ,i which does not give any useful information for the minimization ( v 1 and v 2 are sampled independently and ¯ x is given),

or returns a row from A , as covered by the original proof.

Hence, on an event E 2 of probability at least 1 / 3 , even with the extra information of ¯ x , during the next Q 2 queries starting from T 2 , the algorithm does not query a Θ( µξ 3 2 ) -suboptimal solution to F A , v 1 , v 2 . This holds a fortiori for the model when the algorithm is not given ¯ x at time T 2 .

C.2.2 Recursive construction of a p -level class of functions F A , v 1 ,..., v p

Similarly as in the last section, one can inductively construct the sequence of functions F A , v 1 , F A , v 1 , v 2 , F A , v 1 , v 2 , v 3 , etc. Formally, the induction is constructed as follows: let ( v p ) p ≥ 1 be an i.i.d. sequence of N i.i.d. vectors ( v k,i ) i ∈ [ N ] sampled from the rescaled hypercube d -1 / 2 {± 1 } d . Next, we pose

where µ (1) = µ , ¯ x (1) = 0 and s (1) = 1 . For k ≥ 1 , we pose

for a certain parameter ξ k +1 to be specified. We then define the next level as

We then pose µ ( k +1) := µ ( k ) ξ 2 k +1 / 54 and s ( k +1) := s ( k ) ξ k +1 / 9 , which closes the induction. The optimization functions are defined simply as

We checked before that we can use ξ 2 = 1 / (16 √ d ) . For general k ≥ 0 , given that the form of the function slightly changes to incorporate the absolute term (see ˜ F A , v 2 ), this constant may differ slightly. In any case, one has ξ k = Θ(1 / √ d ) . Now fix a construction level p ≥ 1 and for any k ∈ [ p ] , let T k be the first time that a point with function value at most F ( k ) is queried. For convenience let T 0 = 0 . Using the same arguments as above recursively, we can show that on an event E k with probability at least 1 / 3 ,

Next note that the sequence F ( k ) is decreasing and by construction, if one finds a µ ( p ) ξ p +1 -suboptimal point of F A , v 1 ,..., v p , then this point has value at most F ( p ) . As a result, for an algorithm that finds a µ ( p ) ξ p +1 -suboptimal point, the times T 0 , . . . , T p are all well defined and non-decreasing. We recall that µ = Θ( √ dϵ ) . Therefore, we can still have µ/s ( p ) ≤ √ ϵ and µ ( p ) ξ p +1 ≥ ϵ 2 for p = Θ( ln 1 ϵ ln d ) . Combining these observations, we showed that when optimizing the functions F A , v 1 ,..., v p to accuracy Θ( µ ( p ) ξ p +1 ) = Ω( ϵ 2 ) , the total number of queries Q satisfies

Changing ϵ to ϵ 2 proves the desired result.

Theorem C.5. For ϵ ≤ 1 /d 8 and any δ ∈ [0 , 1] , any (potentially randomized) algorithm guaranteed to minimize 1 -Lipschitz convex functions over the unit ball with ϵ accuracy uses at least d 1 . 25 -δ ln 1 ϵ bits of memory or makes ˜ Ω( d 1+4 δ/ 3 ln 1 ϵ ) queries.

The same recursive construction can be applied to the results from Theorems C.1 and C.3 to improve their oracle-complexity lower bounds by a factor ln 1 ϵ ln d , albeit with added technicalities due to the adaptivity of their class of functions. This yields Theorem 3.3.

Input: Number of iterations T , computation accuracy η ≤ 1 , target accuracy ϵ ≤ 1 Initialize: x = 0 ; for t = 0 , . . . , T do Query the oracle at x if x successful then return x ; Receive a separation vector g with accuracy η Update x as x -ϵ g up to accuracy η end return x

D Memory-constrained gradient descent for the feasibility problem

In this section, we prove a simple result showing that memory-constrained gradient descent applies to the feasibility problem. We adapt the algorithm described in [52].

We now prove that this memory-constrained gradient descent gives the desired result of Proposition 3.1.

Proof of Proposition 3.1. Denote by x t the state of x at iteration t , and g t (resp. ˜ g t ) the separation oracle without rounding errors (resp. with rounding errors) at x t . By construction,

As a result, recalling that ∥ g t ∥ = 1 ,

By assumption, Q contains a ball B d ( x ⋆ , ϵ ) for x ⋆ ∈ B d (0 , 1) . Then, because g t separates x t from B d ( x ⋆ , ϵ ) , one has g ⊤ t ( x ⋆ -x t ) ≥ ϵ . Therefore,

Then, take η = ϵ 2 / 40 and T = 8 ϵ 2 . If iteration T was performed, we have using the previous equation

Hence, x T is an ϵ -suboptimal solution.

We now turn to the memory usage of gradient descent. It only needs to store x and g up to the desired accuracy η = O ( ϵ 2 ) . Hence, this storage and the internal computations can be done with O ( d ln d ϵ ) memory. Because we suppose that ϵ ≤ 1 √ d , this gives the desired result.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main result of the paper is a theorem that establishes a trade-off between oracle-complexity and memory-usage for convex optimization problems. More specifically they show that given a convex optimization problem in R^d, then for any p between 1 and d there is a deterministic first-order algorithm tha solves the feasibility problem for accuracy \\epsilon <=1/sqrt{d} using O(d^2/p ln 1 \\epsilon) bits of memory with O((C (d/p) (ln 1/epsilon))^p) calls to the separation oracle.  In other words, by decreasing the number of bits of memory by a factor of p, one needs to call the oracle a number of times that is exponential in p. The basis of the exponential function is roughly (d/p)* ln(1/epsilon).\n\nSTRENGTHS:\nThe papers seems to make relevant advances in the understanding of tradeoffs between memory and number of oracle calls in convex optimization algorithms.\n\nWEAKNESSES:\nThe technique is heavily based on the Vaidya's cutting-plane method. It is not entirely clear what are really the advancements with respect to the original technique. \n\nThere are some minor presentation issues, such as statements of results with undefined parameters or parameters that are defined very far away.\n\nQUESTIONS:\n1) Please make theorem 3.2 more self contained by briefly describing two parameters before the statement of the theorem: d and omega. \n\nIt should not be necessary to read a big portion of the paper to be able to understand the meaning of the main statement (Theorem 3.2). For instance, you could add a line before the theorem stating that you are dealing with optimization problems in R^d. Same thing for omega. I only see it defined in the footnote of page 4. Although the notation is standard, it took some time to find the precise meaning of omega. So it would be good to write its meaning either in the theorem or in the text just before it.  \n\n2) What are the differences between your method and Vaidya's method? Is it the case that the only difference relies in a recursive application of Vaidya's method? Do you modify the method somehow? This should be better discussed in the introduction.\n\nLIMITATIONS:\nIn my opinion the paper discusses the limitations of the techniques in a fair way.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of memory constrained convex optimization. In this setting, the algorithm is given oracle access to the gradients of an unknown convex function and is tasked with finding an approximate minimizer. The additional constraint is that the algorithm is constrained to use as few bits of memory as possible. Though convex programming is a classical area, understanding the memory requirements of algorithms has only recently been considered and has become an active area of research. When considering memory, two standard algorithms witness a tradeoff between memory and oracle complexity: gradient descent makes $1/\\epsilon^2$ queries and uses $d \\log(1/\\epsilon)$ bits of memory while the center of gravity method makes $d \\log(1/\\epsilon) $ queries and uses $d^2 \nlog^2(1/ \\epsilon)$ bits of memory. A recent line of study has been trying to understand the tradeoff between these complexities. The paper presents a recursive implementation of a cutting plane method which uses $O( d^2 / p \\log (1/ \\epsilon)$ memory while making $O( (d/p \\log (1/ \\epsilon))^p$ queries. Here $p \\in [d]$ is a parameter of the algorithm. In particular, the paper shows that for $\\epsilon = d^{-d}$ the algorithm presented has improved query complexity than gradient descent while using the same amount of memory.\n\nSTRENGTHS:\nThe question considered by the paper is very interesting as memory constraints are a very natural \"simplicity\" condition on algorithms. To the best of my knowledge this is the first algorithm that improves on the memory complexity over the two benchmark algorithms. Furthermore the framework of recursive partitioning seems very interesting and could lead to improvements to other optimization algorithms,\n\nWEAKNESSES:\nThe regime of parameters that the algorithms improves over gradient descent is rather quant. In this regime both the algorithms make $d^O(d)$ calls to the oracle and the current algorithm improves the constant in $O(d)$ in the exponent. While this is of mathematical interest and does show the non Pareto optimality of gradient descent, justifying the interestingness of this range of parameters seems difficult.\n\nQUESTIONS:\n- More thorough description of the algorithm along with the intuition for why it works would be very helpful.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

comments: SUMMARY:
This paper studies solving feasibility problems, and hence optimization problems, and focusing on the trade-offs between the number of oracle calls and use of memory. The precise problem is to find a point in a convex set inside the unit cube, given access to a separation oracle, which reports if a query point is in the convex set, or otherwise returns a hyperplane separating the input query point and the convex set.

By segmenting the variables/dimensions and working on them sequentially, and using the variant of Vaidya’s cutting-plane method by Lee, Sidford, and Wong, this paper presents a new recursive algorithm with better trade-offs between the number of oracle calls and use of memory. In particular, this algorithm uses the same optimal memory but makes less oracle queries than gradient descent for low error regime $\epsilon \le \frac1{\sqrt d}$.

In addition to algorithms, this paper slightly improves on existing lower-bounds for the trade-offs between accuracy $\epsilon$ and dimension $d$ with a more careful analysis.

STRENGTHS:
This paper adapts an existing algorithm with a recursive decomposition to improve the trade-offs between number of oracle calls and memory usage, beating gradient descent for very low error regime.

WEAKNESSES:
The improvement appears incremental, and does not give new insights or understanding for solving feasibility or optimization problems.

QUESTIONS:
Are there geometric intuition (better than the repeated convexity argument applied to blocks of variables) to the recursive application of cutting plane method? Arguably, breaking up variables into blocks, while improving bounds, does not give new geometric understanding.

LIMITATIONS:
This theoretical paper does not have broader societal impacts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

comments: SUMMARY:
The authors propose an algorithm to solve a feasibility problem with constrained memory while minimizing the number of calls in separation oracle. The memory complexity and the calls to separation oracle is parametetrized by a parameter $p$.  They get $O(d^2/p \log 1/\epsilon)$ bits of memory with $O(d/p\log(1/\epsilon))^p$ calls to separation oracle. Their result basically improves the case where $\epsilon$ is much smaller than $1/d$. They also provide a lower bound of that shows the dependences on $\log(1/\epsilon)$ for several settings. The algorithm is based on the Vaidya's cutting plane algorithm.

STRENGTHS:
Contains new contributions for the regime where $\epsilon$ is much smaller than $1/d$.  The lower bound is also an interesting and a new contribution and potentially of independent interest. Overall, the paper is well written.

WEAKNESSES:
For me, the only weakness is that as you improve the memory, you increase the number of calls exponentially, which does not sound so intuitive to me. But still, the results are interesting.

QUESTIONS:
-

LIMITATIONS:
no limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses a parameterized family of memory-constrained algorithms for convex optimization and compares the memory and oracle call complexity of this family with existing algorithms. The authors show that with this parameterized family, one can trade between oracle call and memory complexity, and they claim that this is the first work in this direction.\n\nSTRENGTHS:\nThe paper is very well written and accessible even to readers slightly unfamiliar with the topic (but see also below for some comments). The clarity is particularly facilitated by the excellent setup in Section 2, and also the overview of related work in Section 2.1 is exemplary. The paper seems theoretically quite strong (although I admittedly did not check the proofs in the supplementary material) and the algorithms are easy to understand. Figure 1 excellently summarizes the significance of the paper in the sense that the known upper and lower bounds are non-trivially improved upon.\n\nWEAKNESSES:\nOne weakness of the paper is the introduction, which requires (in my opinion) more prior knowledge than what may be available from a non-expert. For example, the concept of a separation oracle in line 33 did not become clear to me until line 99, and similarly for the feasibility problem in line 42. In line 36 it is said that center-of-mass methods are \"quadratic\" in memory, from which I deduced quadratic in $d$ (but not in $-\\log\\epsilon$); then the statement that the algorithm improves upon center-of-mass for $p=1$ in line 54 is confusing. This is not a critical weakness, though, as the rest of the paper is quite accessible.\n\nAnother concern is whether the content of the paper fits well the scope of the venue. This is not immediately clear to me and shall be discussed in the discussion phase.\n\n## Minor:\n- line 115: \"can also be carried OUT\"?\n- line 122: \"all known lower boundS\"\n- line 139: \"one needs to store $O(d)$ cuts instead OF $O(...$\n\nQUESTIONS:\n- In Def. 2.2, you write that the \"the contents of $Q$ and $N$ are $q$ and $n$, respectively\" and that \"$R$ must contain at least $n$ bits\". Is the latter the same $n$ as the former?\n- What is the meaning of the notation $\\vee$ and $\\wedge$ in Corollary 3.1?\n\nLIMITATIONS:\nI do not see any immediate negative societal impacts, hence it is fine that the authors did not mention any. The limitations of the study are clearly discussed in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study memory vs oracle calls trade-off for convex optimization. The examples of problems they consider are: To find a point in the $d$-dimensional unit ball up to error $\\epsilon$, or to minimize $1$-Lipschitz convex functions over the unit ball up to error $\\epsilon$.\nThey provide a family of algorithms parametrised by $p \\in [d]$ that solve the problem using $O(d^2 \\log (1/\\epsilon) / p)$ bits of memory and make $(C d \\log (1/\\epsilon)/p)^p$ oracle calls, where $C$ is some absolute constant.\n\nMore precisely, the first result (the case $p=1$) is the proof that a memory-constrained Vaidya’s method has (optimal) oracle complexity $O(d\\log(1/\\epsilon))$ and memory complexity $O(d^2\\log(1/\\epsilon))$, that is a $\\log(1/\\epsilon)$-factor improvement over the state-of-the-art. For $p=d$, their algorithm has oracle complexity $(C\\log (1/\\epsilon))^d$ and (optimal) memory complexity $O(d\\log(1/\\epsilon))$, that improves over state-of-the art oracle complexity $O(1/\\epsilon^2)$ if $\\log(1/\\epsilon) \\gtrsim d\\log d$.\n\nSTRENGTHS:\nThe paper provides new algorithms with better memory vs oracle calls trade-off than the state-of-the art. The paper is well-written, and the contribution is clear. The algorithms are non-trivial and their analyses are sophisticated.\n\nWEAKNESSES:\nMy main concern is that the results only slightly improve over the state-of-the-art. \n\nThe first result improves over the state-of-the-art only by a $\\log(1/\\epsilon)$-factor. It is not surprising, since there are lower bounds that (for the feasibility problem) imply that $d^{2-\\Omega(1)}$ memory complexity can only be achieved with $> d^{1+\\Omega(1)}$ queries, but I am not sure if this result itself is strong and/or interesting enough to justify acceptance in NeurIPS.\n\nThe importance of the second result (with $p>1$) is not very clear to me. Let memory complexity of memory-constrained Vaidya’s method be $M$. In order to achieve memory complexity $o(M)$ (even $M/\\log \\log M$), one needs to increase the oracle complexity by a factor that is super-polynomial in $M$. I can hardly imagine settings when it might be reasonable. For comparison, there are some problems for which information-computation trade-offs are known (e.g. planted clique), and for those problems even a minor improvement over the state-of-the-art might be important, since there are basically only two options: Either we solve the problem (and perhaps spend a lot of computational resources), or do not solve it at all. Here it is not the case: the algorithm designer can always choose to use the memory-constrained Vaidya’s method, or to slightly decrease memory complexity and significantly increase the number of oracle calls (and hence the running time), and it seems to me that the huge price of memory complexity is not adequate here.  \n\nOne could argue that the trade-off that is not useful in practice can be interesting if it clarifies memory vs oracle complexity picture of the problem. But the upper bounds from the paper are very far from the currently known lower bounds, and it does not improve a high-level understanding of the complexity picture. For example, it is not clear if super-polynomial oracle complexity is necessary for sub-quadratic memory complexity.\n\nConsidering these strengths and weaknesses, I recommend borderline reject.\n\nUPDATE: After reading the rebuttal I decided to increase the score from 4 to 6. Please see the comments below.\n\nQUESTIONS:\nI have a question related to non-efficient algorithms: Since to match the optimal memory complexity your algorithm requires $p=d$ (and so the running time is exponential in $d$), it makes sense to compare your algorithms with inefficient approaches. Naive brute force for the feasibility problem (that only uses a weaker oracle for an indicator of the set, not a separation oracle) seems to require $(C/\\epsilon)^d$ oracle calls, which is too large, but maybe there exists some other simple brute force algorithm that uses the separation oracle and has comparable guarantees to your algorithm. Did you by chance think about such an algorithm?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}