SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma B.2. Suppose that throughout the algorithm, when checking the stopping criterion  $\min_{i\in [m]}s_i(\pmb {x}) < 2\epsilon$ , the quantities  $s_i(\pmb {x})$  were computed with accuracy  $\epsilon$ . Suppose that at each iteration of Algorithm 6, the leverage scores  $\psi (\pmb{x}^{(t)})$  are computed up to multiplicative precision  $c_{\Delta} / 4$  (l.3), that when a constraint is added, the response of the oracle  $\pmb{a}$  (l.7) is stored perfectly but b (l.8) is computed up to precision  $\Omega (\frac{\epsilon}{\sqrt{n}})$ . Further suppose that the centering Algorithm 5 is run with numerical approximations according to the assumptions in Lemma B.1. Then, all guarantees for the original algorithm in [27] hold, up to a factor 3 for  $\epsilon$ .
Proof. We start with the termination criterion. Given the requirement on the computational accuracy, we know that the final output  $\pmb{x}$  satisfies  $\min_{i\in [m]} s_i(\pmb{x}) \leq 3\epsilon$ . Further, during the algorithm, if it does not stop, then one has  $\min_{i\in [m]} s_i(\pmb{x}) \geq \epsilon$ , which is precisely the guarantee of the original algorithm in [27].
We next turn to the computation of the leverage scores in 1.4. In the original algorithm, only a  $c_{\Delta}$ -estimate is computed. Precisely, one computes a vector  $\boldsymbol{w}^{(t)}$  such that for all  $i \in [d]$ ,  $\psi(\boldsymbol{x}^{(t)})_i \leq w_i \leq (1 + c_{\Delta}) \psi(\boldsymbol{x}^{(t)})_i$ , then deletes a constraint when  $\min_{i \in [m^{(t)}} w_i^{(t)} \leq c_d$ . In the adapted algorithm, let  $\tilde{\psi}(\boldsymbol{x}^{(t)})_i$  denote the computed leverage scores for  $i \in [d]$ . By assumption, we have
$$
(1 - c _ {\Delta} / 4) \psi (\boldsymbol {x} ^ {(t)}) _ {i} \leq \tilde {\psi} (\boldsymbol {x} ^ {(t)}) _ {i} \leq (1 + c _ {\Delta} / 4) \psi (\boldsymbol {x} ^ {(t)}) _ {i}.
$$
Up to re-defining the constant  $c_d$  as  $(1 - c_{\Delta} / 4)c_d$ ,  $\tilde{\psi}(\boldsymbol{x}^{(t)})$  is precisely within the guarantee bounds of the algorithm. For the accuracy on the separation oracle response and the second-term value  $b$ , [27] emphasizes that the algorithm always changes constraints by a  $\delta$  amount where  $\delta = \Omega(\frac{\epsilon}{\sqrt{d}})$  so that an inexact separation oracle with accuracy  $\Omega(\frac{\epsilon}{\sqrt{d}})$  suffices. Therefore, storing an  $\Omega(\frac{\epsilon}{\sqrt{d}})$  accuracy of the second term keeps the guarantees of the algorithm. Last, we checked in Lemma B.1 that the centering procedure Algorithm 5 satisfies all the requirements needed in the original proof [27].
For our recursive method, we need an efficient cutting-plane method that also provides a proof (certificate) of convergence. This is also provided by [27] that provide a proof that the feasible region has small width in one of the directions  $\mathbf{a}_i$  of the returned polyhedron.
Lemma B.3. [27, Lemma 28] Let  $(\mathcal{P},\boldsymbol {x},(\lambda_i)_i)$  be the output of Algorithm 7. Then,  $\pmb{x}$  is feasible,  $\| \pmb {x}\| _2\leq 3\sqrt{d},\lambda_j\geq 0$  for all  $j$  and  $\sum_{i}\lambda_{i} = 1$  . Further,
$$
\left\| \sum_ {i} \lambda_ {i} \pmb {a} _ {i} \right\| _ {2} = \mathcal {O} \left(\epsilon \sqrt {d} \ln \frac {d}{\epsilon}\right), \quad a n d \quad \sum_ {i} \lambda_ {i} (\pmb {a} _ {i} ^ {\top} \pmb {x} - b _ {j}) \leq \mathcal {O} \left(d \epsilon \ln \frac {d}{\epsilon}\right).
$$
We are now ready to show that Algorithm 6 can be implemented with efficient memory and also provides a proof of the convergence of the algorithm.
Input:  $\epsilon > 0$  and a separation oracle  $O: \mathcal{C}_d \to \mathbb{R}^d$
1 Run Algorithm 6 to obtain a polyhedron  $\mathcal{P}$  and a feasible point  $x$
2  $\pmb{x}^{\star} =$  Centering  $(x,64\ln \frac{2}{\epsilon},c_{\Delta})$
3  $\lambda_{i} = \frac{c_{e} + \psi_{i}(\pmb{x}^{\star})}{s_{i}(\pmb{x}^{\star})}\left(\sum_{j}\frac{c_{e} + \psi_{j}(\pmb{x}^{\star})}{s_{j}(\pmb{x}^{\star})}\right)^{-1}$  for all  $i$
Output:  $(\mathcal{P},\pmb{x}^{\star},(\lambda_{i})_{i})$
Algorithm 7: Cutting-plane algorithm with certified optimality
Proposition B.1. Provided that the output of the oracle are vectors discretized to precision poly  $\left(\frac{\epsilon}{d}\right)$  and have norm at most 1, Algorithm 7 can be implemented with  $\mathcal{O}(d^2\ln \frac{d}{\epsilon})$  bits of memory to output a certified optimal point according to Lemma B.3. The algorithm performs  $\mathcal{O}(d\ln \frac{d}{\epsilon})$  calls to the separation oracle and runs in  $\mathcal{O}(d^{1 + \omega}\ln^{\mathcal{O}(1)}\frac{d}{\epsilon})$  time.
Proof. We already checked the numerical stability of Algorithm 6 in Lemma B.2. It remains to check the next steps of the algorithm. The centering procedure is stable again via Lemma B.1. It also suffices to compute the coefficients  $\lambda_{j}$  up to accuracy  $\mathcal{O}(\epsilon / (\sqrt{d})\ln (d / \epsilon))$  to keep the guarantees desired since by construction all vectors  $a_{i}$  have norm at most one.
It now remains to show that the algorithm can be implemented with efficient memory. We recall that at any point during the algorithm, the polyhedron  $\mathcal{P}$  has at most  $\mathcal{O}(d)$  constraints [27, Lemma 22]. Hence, since we assumed that each vector  $a_{i}$  composing a constraint is discretized to precision  $\mathrm{poly}\left(\frac{\epsilon}{d}\right)$ , we can store the polyhedron constraints with  $\mathcal{O}\left(d^{2} \ln \frac{d}{\epsilon}\right)$  bits of memory. The second terms  $b$  are computed up to precision  $\Omega(\epsilon / \sqrt{d})$  hence only use  $\mathcal{O}\left(d \ln \frac{d}{\epsilon}\right)$  bits of memory. The algorithm also keeps the current iterate  $x^{(t)}$  in memory. These are all bounded throughout the memory  $\| x^{(t)} \|_2 = \mathcal{O}(\sqrt{d})$  [27, Lemma 23], hence only require  $\mathcal{O}\left(d \ln \frac{d}{\epsilon}\right)$  bits of memory for the desired accuracy.
Next, the distances to the constraints are bounded at any step of the algorithm:  $s_i(\pmb{x}^{(t)}) \leq \mathcal{O}(\sqrt{d})$  [27, Lemma 24], hence computing  $s_i(\pmb{x}^{(t)})$  to the required accuracy is memory-efficient. Recall that from the termination criterion, except for the last point, any point  $\pmb{x}$  during the algorithm satisfies  $s_i(\pmb{x}) \geq \epsilon$  for all constraints  $i \in [m]$ . In particular, this bounds the eigenvalues of  $\pmb{Q}$  since  $\lambda I \preceq Q(x) \preceq (\lambda + m(c_e + 1) / \epsilon^2)I$ . Thus, the matrix is sufficiently well-conditioned to achieve the accuracy guarantees from Lemma B.1 using  $\mathcal{O}(d^2 \ln \frac{d}{\epsilon})$  memory during matrix inversions (and matrix multiplications). Similarly, for the computation of leverage scores, we use  $\Psi(x) = \text{diag}(\pmb{A}_x(\pmb{A}_x^\top \pmb{A}_x + \lambda I)^{-1}\pmb{A}_x^\top)$ , where  $\lambda I \preceq A_x^\top A_x + \lambda I \preceq (\lambda + m\epsilon^{-2})I$ . This same matrix inversion appears when computing the second term of an added constraint. Overall, all linear algebra operations are well conditioned and implementable with required accuracy with  $\mathcal{O}(d^2 \ln \frac{d}{\epsilon})$  memory. Using fast matrix multiplication, all these operations can be performed in  $\tilde{\mathcal{O}}(d^{\omega})$  time per iteration of the cutting-plane algorithm since these methods are also known to be numerically stable [15]. Thus, the total time complexity is  $\mathcal{O}(d^{1+\omega} \ln^{O(1)} \frac{d}{\epsilon})$ . The oracle-complexity still has optimal  $\mathcal{O}(d \ln \frac{d}{\epsilon})$  oracle-complexity as in the original algorithm.
Up to changing  $\epsilon$  to  $c\cdot \epsilon /(d\ln \frac{d}{\epsilon})$ , the described algorithm finds constraints given by  $\pmb{a}_i$  and  $b_{i}$ $i\in [m]$  returned by the normalized separation oracle, coefficients  $\lambda_{i}$ ,  $i\in [m]$ , and a feasible point  $\pmb{x}^{\star}$  such that for any vector in the unit cube,  $\pmb {z}\in \mathcal{C}_d$ , one has
$$
\min  _ {i \in [ m ]} \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {z} - b _ {i} \leq \sum_ {i \in [ m ]} \lambda_ {i} (\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {z} - b _ {i}) \leq \left(\sum_ {i \in [ m ]} \lambda \boldsymbol {a} _ {i}\right) ^ {\top} (\boldsymbol {x} ^ {\star} - \boldsymbol {z}) + \sum_ {i \in [ m ]} \lambda_ {i} (\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} ^ {\star} - b _ {i}) \leq \epsilon .
$$
This effectively replaces Lemma 4.1.
B.2 Merging Algorithm 7 within the recursive algorithm
Algorithms 2 to 4 from the recursive procedure need to be slightly adapted to the new format of the cutting-plane method's output. In particular, the oracles do not take as input polyhedrons (and
eventually query their volumetric center as before), but directly take as input an point (which is an approximate volumetric center).
Input:  $\delta, \xi, O_x: \mathcal{C}_n \to \mathbb{R}^m$  and  $O_y: \mathcal{C}_n \to \mathbb{R}^n$
1 Run Algorithm 7 with parameter  $c\cdot \delta /(d\ln \frac{d}{\delta}),\xi$  and  $O_{y}$  to obtain  $(\mathcal{P}^{\star},\pmb{x}^{\star},\pmb {\lambda})$
2 Store  $\pmb{k}^{\star} = (k_{i}, i \in [m])$  where  $m = |\mathcal{P}^{\star}|$ , and  $\lambda^{\star} \gets \text{Discretize}(\lambda^{\star}, \xi)$
3 Initialize  $\mathcal{P}_0\coloneqq \{(-1,\pmb {e}_i, - 1),(-1 - \pmb {e}_i, - 1),i\in [d]\}$ $\pmb{x}^{(0)} = \pmb{0}$  and let  $\pmb {u} = \pmb {0}\in \mathbb{R}^{m}$
4 for  $t = 0,1,\dots ,\max_{i}k_{i}$  do
5 if  $t = k_i^\star$  for some  $i\in [m]$  then 6  $\begin{array}{rl} & {\pmb {g}_x = O_x(\pmb{x}^{(t)})}\\ & {\pmb {u}\gets \mathrm{Discretize}_m(\pmb {u} + \lambda_i^\star \pmb {g}_x,\xi)}\\ & {\mathrm{Update}\mathcal{P}_t\mathrm{to}\mathrm{get}\mathcal{P}_{t + 1},\mathrm{and}\pmb{x}^{(t)}\mathrm{to}\mathrm{get}\pmb{x}^{(t + 1)}\mathrm{as in~Algorithm~6}} \end{array}$  7 8
10 return  $\pmb{u}$
Algorithm 8: ApproxSeparationVector $_{\delta, \xi}(O_x, O_y)$
Input:  $\delta, \xi, 1 \leq j \leq i \leq p$ ,  $\pmb{x}^{(r)} \in \mathcal{C}_{k_r}$  for  $r \in [i]$ ,  $O_S : \mathcal{C}_d \to \mathbb{R}^d$
1 if  $i = p$  then
2  $(\pmb {g}_1,\dots ,\pmb {g}_p) = O_S(\pmb {x}_1,\dots ,\pmb {x}_p)$  3 return Discretize  $k_{j}$ $(\pmb {g}_j,\xi)$
4 end
5 Define  $O_{x}:\mathcal{C}_{k_{i + 1}}\to \mathbb{R}^{k_{j}}$  as ApproxOracle  $\delta_{\xi ,\mathcal{O}_f}(i + 1,j,\pmb{x}^{(1)},\dots ,\pmb{x}^{(i)},\cdot)$
6 Define  $O_y: \mathcal{C}_{k_{i+1}} \to \mathbb{R}^{k_{i+1}}$  as ApproxOracle $_{\delta, \xi, \mathcal{O}_f}(i+1, i+1, \boldsymbol{x}^{(1)}, \ldots, \boldsymbol{x}^{(i)}, \cdot)$
7 return ApproxSeparationVector $_{\delta, \xi}(O_x, O_y)$
Algorithm 9: ApproxOracle $_{\delta, \xi, O_S}$ $(i, j, \boldsymbol{x}^{(1)}, \ldots, \boldsymbol{x}^{(i)})$
Check : Throughout the algorithm, if  $O_S$  returned Success to a query  $x$ , return  $x$
1 Run Algorithm 6 with parameters  $\delta$  and  $\xi$  and oracle  $\text{ApproxOracle}_{\delta, \xi, O_S}(1, 1, \cdot)$
Algorithm 10: Memory-constrained algorithm for convex optimization
The same proof as for Algorithm 4 shows that Algorithm 10 runs with the parameters in Theorem A.1 also outputs a successful vector using the same oracle-complexity. We only need to analyze the memory usage in more detail.
Proof of Theorem 3.2. As mentioned above, we will check that Algorithm 10 with the same parameters  $\delta = \frac{\epsilon}{4d}$  and  $\xi = \frac{\sigma_{\min} \epsilon}{32 d^{5/2}}$  as in Theorem A.1 satisfies the desired requirements. We have already checked its correctness and oracle-complexity. Using the same arguments, the computational complexity is of the form  $\mathcal{O}(\mathcal{O}(\text{ComplexityCuttingPlanes})^p)$  where  $\text{ComplexityCuttingPlanes}$  is the computational complexity of the cutting-plane method used, i.e., here of Algorithm 7. Hence, the computational complexity is  $\mathcal{O}((C(d/p)^{1+\omega} \ln^{\mathcal{O}(1)} \frac{d}{\epsilon})^p)$  for some universal constant  $C \geq 2$ . We now turn to the memory. In addition to the memory of Algorithm 4, described in Table 1, we need
1. a placement for all  $i\in [p]$  for the current iterate  $\pmb{x}^{(i)}\colon \mathcal{O}(k_i\ln \frac{1}{\xi})$  bits,
2. a placement for computations, that is shared for all layers (used to compute leverage scores, centering procedures, etc. By Proposition B.1, since the vectors are always discretized to precision  $\xi$ , this requires  $\mathcal{O}(\max_{i\in [p]}k_i^2\ln \frac{d}{\epsilon})$  bits,
3. the placement  $Q$  to perform queries is the concatenation of the placements  $(\pmb{x}^{(1)},\dots ,\pmb{x}^{(p)})$  .. no additional bits needed.
4. a placement  $N$  to store the precision needed for the oracle responses:  $\mathcal{O}(\ln \frac{1}{\xi})$  bits
5. a placement  $R$  to receive the oracle responses:  $\mathcal{O}(d\ln \frac{1}{\xi})$  bits.
The new memory structure is summarized in Table 2.
With the same arguments as in the original proof of Theorem A.1, this memory is sufficient to run the algorithm and perform computations, thanks to the computation placement. The total number of bits used throughout the algorithm remains the same,  $\mathcal{O}\left(\frac{d^2}{p}\ln \frac{d}{\epsilon}\right)$ . This ends the proof of the theorem.
Table 2: Memory structure for Algorithm 10
i	1	...	p	Oracle response	Precision
j	j(1)	j(p)	R=(R1,...,Rp)	N
Iteration index	t(1)	t(p)
Polyhedron	P(1)=(k1,a1,b1k2,a2,b2...km,am,bm)	P(p)	Computation memory
Current iterate	x(1)	x(p)
Computed dual variables	(k*,λ*)=(k1*,λ1*)k2*,λ2*)...	(k*(p),λ*(p))
Working separation vector	u(1)	u(p)
C Improved oracle-complexity/memory lower-bound trade-offs
We recall the three oracle-complexity/memory lower-bound trade-offs known in the literature.
1. First, [31] showed that any (including randomized) algorithm for convex optimization uses  $d^{1.25 - \delta}$  memory or makes  $\tilde{\Omega}(d^{1 + 4\delta / 3})$  queries.
2. Then, [5] showed that any deterministic algorithm for convex optimization uses  $d^{2 - \delta}$  memory or makes  $\tilde{\Omega}(d^{1 + \delta/3})$  queries.
3. Last, [5] show that any deterministic algorithm for the feasibility problem uses  $d^{2 - \delta}$  memory or makes  $\tilde{\Omega}(d^{1 + \delta})$  queries.
Although these papers mainly focused on the regime  $\epsilon = 1 / \mathrm{poly}(d)$  and as a result  $\ln \frac{1}{\epsilon} = \mathcal{O}(\ln d)$ , neither of these lower bounds have an explicit dependence in  $\epsilon$ . This can lead to sub-optimal lower bounds whenever  $\ln \frac{1}{\epsilon} \gg \ln d$ . Furthermore, in the exponential regime  $\epsilon \leq \frac{1}{2^{\mathcal{O}(d)}}$ , these results do not effectively give useful lower bounds. Indeed, in this regime, one has  $d^{2} = \mathcal{O}(d\ln \frac{1}{\epsilon})$  and as a result, the lower bounds provided are weaker than the classical  $\Omega(d\ln \frac{1}{\epsilon})$  lower bounds for oracle-complexity [34] and memory [52]. In particular, in this exponential regime, these results fail to show that there is any trade-off between oracle-complexity and memory.
In this section, we aim to explicit the dependence in  $\epsilon$  of these lower-bounds. We show with simple modifications and additional arguments that one can roughly multiply these oracle-complexity and memory lower bounds by a factor  $\ln \frac{1}{\epsilon}$  each. We split the proofs in two. First we give arguments to improve the memory dependence by a factor  $\ln \frac{1}{\epsilon}$ , which is achieved by modifying the sampling of the rows of the matrix  $A$  defining a wall term common to the functions considered in the lower bound proofs [31, 5]. Then we show how to improve the oracle-complexity dependence by an additional  $\ln \frac{1}{\epsilon} / \ln d$  factor, via a standard rescaling argument.
C.1 Improving the memory lower bound
We start with some concentration results on random vectors. [31] gave the following result for random vectors in the hypercube.
Lemma C.1 ([31]). Let  $\pmb{h} \sim \mathcal{U}(\{\pm 1\}^d)$ . Then, for any  $t \in (0,1/2]$  and any matrix  $Z = [z_1, \ldots, z_k] \in \mathbb{R}^{d \times k}$  with orthonormal columns,
$$
\mathbb {P} \left(\| Z ^ {\top} h \| _ {\infty} \leq t\right) \leq 2 ^ {- c _ {H} k}.
$$
Instead, we will need a similar concentration result for random unit vectors in the unit sphere.
Lemma C.2. Let  $k \leq d$  and  $x_{1}, \ldots, x_{k}$  be  $k$  orthonormal vectors, and  $\zeta \leq 1$ .
$$
\mathbb {P} _ {\boldsymbol {y} \sim \mathcal {U} (S ^ {d - 1})} \left(\left| \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {y} \right| \leq \frac {\zeta}{\sqrt {d}}, i \in [ k ]\right) \leq \left(\frac {2}{\sqrt {\pi}} \zeta\right) ^ {k} \leq (\sqrt {2} \zeta) ^ {k}.
$$
Proof. First, by isometry, we can suppose that the orthonormal vectors are simply  $e_1, \ldots, e_k$ . We now prove the result by induction on  $d$ . For  $d = 1$ , the result holds directly. Fix  $d \geq 2$ , and  $1 \leq k < d$ . Then, if  $S_n$  is the surface area of  $S^n$  the  $n$ -dimensional sphere, then
$$
\mathbb {P} \left(\left| y _ {1} \right| \leq \frac {\zeta}{\sqrt {d}}\right) \leq \frac {S _ {d - 2}}{S _ {d - 1}} \frac {2 \zeta}{\sqrt {d}} = \frac {2 \zeta}{\sqrt {\pi d}} \frac {\Gamma (d / 2)}{\Gamma (d / 2 - 1 / 2)} \leq \frac {2}{\sqrt {\pi}} \zeta . \tag {7}
$$
Conditionally on the value of  $y_{1}$ , the vector  $(y_{2},\ldots ,y_{d})$  follows a uniform distribution on the  $(d - 2)$ -sphere of radius  $\sqrt{1 - y_1^2}$ . Then,
$$
\mathbb {P} \left(| y _ {i} | \leq \frac {\zeta}{\sqrt {d}}, 2 \leq i \leq k \mid y _ {1}\right) = \mathbb {P} _ {\boldsymbol {z} \sim \mathcal {U} (S ^ {d - 2})} \left(| z _ {i} | \leq \frac {\zeta}{\sqrt {d (1 - y _ {1} ^ {2})}}, 2 \leq i \leq k\right)
$$
Now recall that since  $|x_1| \leq 1 / \sqrt{d}$ , we have  $d(1 - x_1^2) \geq d - 1$ . Therefore, using the induction,
$$
\mathbb {P} \left(\left| y _ {i} \right| \leq \frac {\zeta}{\sqrt {d}}, 2 \leq i \leq k \mid y _ {1}\right) \leq \mathbb {P} _ {\boldsymbol {z} \sim \mathcal {U} (S ^ {d - 2})} \left(\left| z _ {i} \right| \leq \frac {\zeta}{\sqrt {d - 1}}, 2 \leq i \leq k\right) \leq \left(\frac {2 \zeta}{\sqrt {\pi}}\right) ^ {k - 1}.
$$
Combining this equation with Eq (7) ends the proof.
We next use the following lemma to partition the unit sphere  $S^{d - 1}$ .
Lemma C.3 ([19] Lemma 21). For any  $0 < \delta < \pi /2$ , the sphere  $S^{d - 1}$  can be partitioned into  $N(\delta) = (\mathcal{O}(1) / \delta)^d$  equal volume cells, each of diameter at most  $\delta$ .
Following the notation from [5], we denote by  $\mathcal{V}_{\delta} = \{V_{i}(\delta), i \in [N(\delta)]\}$  the corresponding partition, and consider a set of representatives  $\mathcal{D}_{\delta} = \{\pmb{b}_i(\delta), i \in [N(\delta)]\} \subset S^{d-1}$  such that for all  $i \in [N(\delta)]$ ,  $\pmb{b}_i(\delta) \in V_i(\delta)$ . With these notations we can define the discretization function  $\phi_{\delta}$  as follows
$$
\phi_ {\delta} (\boldsymbol {x}) = \boldsymbol {b} _ {i} (\delta), \quad \boldsymbol {x} \in V _ {i} (\delta).
$$
We then denote by  $\mathcal{U}_{\delta}$  the distribution of  $\phi_{\delta}(z)$  where  $z \sim \mathcal{U}(S^{d-1})$  is sampled uniformly on the sphere. Note that because the cells of  $\mathcal{V}_{\delta}$  have equal volume,  $\mathcal{U}_{\delta}$  is simply the uniform distribution on the discretization  $\mathcal{D}_{\delta}$ .
We are now ready to give the modifications necessary to the proofs, to include a factor  $\ln \frac{1}{\epsilon}$  for the necessary memory. For their lower bounds, [31] exhibit a distribution of convex functions that are hard to optimize. Building upon their work [5] construct classes of convex functions that are hard to optimize, but that also depend adaptively on the considered optimization algorithm. For both, the functions considered a barrier term of the form  $\| A x \|_{\infty}$ , where  $A$  is a matrix of  $\approx d / 2$  rows that are independently drawn as uniform on the hypercube  $\mathcal{U}(\{\pm 1\}^d)$ . The argument shows that memorizing  $A$  is necessary to a certain extent. As a result, the lower bounds can only apply for a memory of at most  $\mathcal{O}(d^{2})$  bits, which is sufficient to memorize such a binary matrix. Instead, we draw rows independently according to the distribution  $\mathcal{U}_{\delta}$ , where  $\delta \approx \epsilon$ . We explicit the corresponding adaptations for each known trade-off. We start with the lower bounds from [5] for ease of exposition; although these build upon those of [31], their parametrization makes the adaptation more straightforward.
C.1.1 Lower bound of [5] for convex optimization and deterministic algorithms
For this lower bound, we use the exact same form of functions as they introduced,
$$
\left. \max  \left\{\| \boldsymbol {A} \boldsymbol {x} \| _ {\infty} - \eta , \eta \boldsymbol {v} _ {0} ^ {\top} \boldsymbol {x}, \eta \left(\max  _ {p \leq p _ {m a x}, l \leq l _ {p}} \boldsymbol {v} _ {p, l} ^ {\top} \boldsymbol {x} - p \gamma_ {1} - l \gamma_ {2}\right) \right\}, \right.
$$
with the difference that rows of  $\mathbf{A}$  are take i.i.d. distributed according to  $\mathcal{U}_{\delta'}$  instead of  $\mathcal{U}(\{\pm 1\}^d)$ . As a remark, they use  $n = \lceil d / 4 \rceil$  rows for  $\mathbf{A}$ . Except for  $\eta$ , we keep all parameters  $\gamma_1, \gamma_2$ , etc as in the original proof, and we will take  $\delta' = \epsilon$  and  $\eta = 2\sqrt{d}\epsilon$ . The reason why we introduced  $\delta'$  instead of  $\delta$  is that the original construction also needs the discretization  $\phi_{\delta}$ . This is used during the optimization procedure which constructs adaptively this class of functions, and only needs  $\delta = \mathrm{poly}(1 / d)$  instead of  $\delta$  of order  $\epsilon$ .
Theorem C.1. For  $\epsilon \leq 1 / (2d^{4.5})$  and any  $\delta \in [0,1]$ , a deterministic first-order algorithm guaranteed to minimize 1-Lipschitz convex functions over the unit ball with  $\epsilon$  accuracy uses at least  $d^{2 - \delta}\ln \frac{1}{\epsilon}$  bits of memory or makes  $\tilde{\Omega} (d^{1 + \delta /3})$  queries.
With the changes defined above, we can easily check that all results from [5] which reduce convex optimization to the optimization procedure, then the optimization procedure to their Orthogonal Vector Game with Hints (OVGH) [5, Game 2], are not affected by our changes. The only modifications to perform are to the proof of query lower bound for the OVGH [5, Proposition 14]. We emphasize that the distribution of  $A$  is changed in the optimization procedure but also in OVGH as a result.
Proposition C.2. Let  $k \geq 20\frac{M + 3d\log(2d) + 1}{n\log_2(\sqrt{2}(\zeta + \delta'\sqrt{d}))^{-1}}$ . And let  $0 < \alpha, \beta \leq 1$  such that  $\alpha(\sqrt{d} / \beta)^{5/4} \leq \zeta / \sqrt{d}$  where  $\zeta \leq 1$ . If the Player wins the adapted OVGH with probability at least  $1/2$ , then  $m \geq \frac{1}{8}(1 + \frac{30\log_2 d}{\log_2(\sqrt{2}(\zeta + \delta'\sqrt{d}))^{-1}})^{-1}d$ .
Proof. We use the same proof and only highlight the modifications. The proof is unchanged until the step when the concentration result Lemma C.1 is used. Instead, we use Lemma C.2. With the same notations as in the original proof, we constructed  $\lceil k/5\rceil$  orthonormal vectors  $Z = [z_1, \ldots, z_{\lceil k/5 \rceil}]$  such that all rows  $a$  of  $A'$  (which is  $A$  up to some observed and unimportant rows) one has
$$
\| \boldsymbol {Z} ^ {\top} \boldsymbol {a} \| _ {\infty} \leq \frac {\zeta}{\sqrt {d}}.
$$
Next, by Lemma C.2, we have