SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next note that the sequence  $F^{(k)}$  is decreasing and by construction, if one finds a  $\mu^{(p)}\xi_{p + 1}$ -suboptimal point of  $F_{A,\mathbf{v}_1,\dots,\mathbf{v}_p}$ , then this point has value at most  $F^{(p)}$ . As a result, for an algorithm that finds a  $\mu^{(p)}\xi_{p + 1}$ -suboptimal point, the times  $T_0,\ldots ,T_p$  are all well defined and non-decreasing. We recall that  $\mu = \Theta (\sqrt{d}\epsilon)$ . Therefore, we can still have  $\mu /s^{(p)}\leq \sqrt{\epsilon}$  and  $\mu^{(p)}\xi_{p + 1}\geq \epsilon^2$  for  $p = \Theta (\frac{\ln\frac{1}{\epsilon}}{\ln d})$ . Combining these observations, we showed that when optimizing the functions  $F_{A,\mathbf{v}_1,\dots,\mathbf{v}_p}$  to accuracy  $\Theta (\mu^{(p)}\xi_{p + 1}) = \Omega (\epsilon^2)$ , the total number of queries  $Q$  satisfies
$$
\mathbb {E} [ Q ] \geq \frac {1}{3} \sum_ {k \in [ p ]} Q _ {k} \geq \frac {p}{3} Q (\sqrt {\epsilon}; M, d) = \Theta \left(\frac {d ^ {4 / 3} \ln \frac {1}{\epsilon}}{\ln^ {4 / 3} d} \left(\frac {d \ln \frac {1}{\epsilon}}{M + d \ln d}\right) ^ {4 / 3}\right).
$$
Changing  $\epsilon$  to  $\epsilon^2$  proves the desired result.
Theorem C.5. For  $\epsilon \leq 1 / d^8$  and any  $\delta \in [0,1]$ , any (potentially randomized) algorithm guaranteed to minimize 1-Lipschitz convex functions over the unit ball with  $\epsilon$  accuracy uses at least  $d^{1.25 - \delta}\ln \frac{1}{\epsilon}$  bits of memory or makes  $\tilde{\Omega} (d^{1 + 4\delta /3}\ln \frac{1}{\epsilon})$  queries.
The same recursive construction can be applied to the results from Theorems C.1 and C.3 to improve their oracle-complexity lower bounds by a factor  $\frac{\ln\frac{1}{c}}{\ln d}$ , albeit with added technicalities due to the adaptivity of their class of functions. This yields Theorem 3.3.
Input: Number of iterations  $T$ , computation accuracy  $\eta \leq 1$ , target accuracy  $\epsilon \leq 1$
Initialize:  $\pmb{x} = \mathbf{0}$
for  $t = 0,\dots ,T$  do
Query the oracle at  $\pmb{x}$   
if  $\pmb{x}$  successful then return  $x$ ;  
Receive a separation vector  $\pmb{g}$  with accuracy  $\eta$ ;  
Update  $\pmb{x}$  as  $x - \epsilon g$  up to accuracy  $\eta$
return  $\pmb{x}$
Algorithm 11: Memory-constrained gradient descent
D Memory-constrained gradient descent for the feasibility problem
In this section, we prove a simple result showing that memory-constrained gradient descent applies to the feasibility problem. We adapt the algorithm described in [52].
We now prove that this memory-constrained gradient descent gives the desired result of Proposition 3.1.
Proof of Proposition 3.1. Denote by  $\pmb{x}_t$  the state of  $\pmb{x}$  at iteration  $t$ , and  $\pmb{g}_t$  (resp.  $\tilde{\pmb{g}}_t$ ) the separation oracle without rounding errors (resp. with rounding errors) at  $\pmb{x}_t$ . By construction,
$$
\left\| \boldsymbol {x} _ {t + 1} - \left(\boldsymbol {x} _ {t} + \epsilon \tilde {\boldsymbol {g}} _ {t}\right) \right\| \leq \eta \quad \text {a n d} \quad \left\| \tilde {\boldsymbol {g}} _ {t} - \boldsymbol {g} _ {t} \right\| \leq \eta . \tag {11}
$$
As a result, recalling that  $\| g_t\| = 1$
$$
\left\| \boldsymbol {x} _ {t + 1} - \boldsymbol {x} ^ {\star} \right\| ^ {2} \leq \left(\left\| \boldsymbol {x} _ {t} + \epsilon \tilde {\boldsymbol {g}} _ {t} - \boldsymbol {x} ^ {\star} \right\| + \eta\right) ^ {2} \leq \left(\left\| \boldsymbol {x} _ {t} + \epsilon \boldsymbol {g} _ {t} - \boldsymbol {x} ^ {\star} \right\| + (1 + \epsilon) \eta\right) ^ {2} \leq \left\| \boldsymbol {x} _ {t} + \epsilon \boldsymbol {g} _ {t} - \boldsymbol {x} ^ {\star} \right\| ^ {2} + 2 0 \eta .
$$
By assumption,  $Q$  contains a ball  $B_{d}(\pmb{x}^{\star}, \epsilon)$  for  $\pmb{x}^{\star} \in B_{d}(0,1)$ . Then, because  $\pmb{g}_{t}$  separates  $\pmb{x}_{t}$  from  $B_{d}(\pmb{x}^{\star}, \epsilon)$ , one has  $\pmb{g}_{t}^{\top}(\pmb{x}^{\star} - \pmb{x}_{t}) \geq \epsilon$ . Therefore,
$$
\begin{array}{l} \left\| \boldsymbol {x} _ {t + 1} - \boldsymbol {x} ^ {\star} \right\| ^ {2} \leq \left\| \boldsymbol {x} _ {t} - \boldsymbol {x} ^ {\star} \right\| ^ {2} + 2 \epsilon \boldsymbol {g} _ {t} ^ {\top} \left(\boldsymbol {x} _ {t} - \boldsymbol {x} ^ {\star}\right) + \epsilon^ {2} \left\| \boldsymbol {g} _ {t} \right\| ^ {2} + 2 0 \eta \\ \leq \left\| \boldsymbol {x} _ {t} - \boldsymbol {x} ^ {\star} \right\| ^ {2} - \epsilon^ {2} + 2 0 \eta . \\ \end{array}
$$
Then, take  $\eta = \epsilon^2 / 40$  and  $T = \frac{8}{\epsilon^2}$ . If iteration  $T$  was performed, we have using the previous equation
$$
\left\| \boldsymbol {x} _ {T} - \boldsymbol {x} ^ {\star} \right\| ^ {2} \leq \left\| \boldsymbol {x} _ {0} - \boldsymbol {x} ^ {\star} \right\| ^ {2} - \frac {\epsilon^ {2}}{2} T \leq 4 - \frac {\epsilon^ {2}}{2} T \leq 0.
$$
Hence,  $\pmb{x}_T$  is an  $\epsilon$ -suboptimal solution.
We now turn to the memory usage of gradient descent. It only needs to store  $\pmb{x}$  and  $\pmb{g}$  up to the desired accuracy  $\eta = \mathcal{O}(\epsilon^2)$ . Hence, this storage and the internal computations can be done with  $\mathcal{O}(d\ln \frac{d}{\epsilon})$  memory. Because we suppose that  $\epsilon \leq \frac{1}{\sqrt{d}}$ , this gives the desired result.