SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| \left\{\boldsymbol {a} \in \mathcal {D} _ {\delta^ {\prime}}: \| \boldsymbol {Z} ^ {\top} \boldsymbol {a} \| _ {\infty} \leq \frac {\zeta}{\sqrt {d}} \right\} \right| \leq | \mathcal {D} _ {\delta^ {\prime}} | \cdot \mathbb {P} _ {\boldsymbol {a} \sim \mathcal {U} _ {\delta^ {\prime}}} \left(\| \boldsymbol {Z} ^ {\top} \boldsymbol {a} \| _ {\infty} \leq \frac {\zeta}{\sqrt {d}}\right) \\ \leq \left| \mathcal {D} _ {\delta^ {\prime}} \right| \cdot \mathbb {P} _ {\boldsymbol {z} \sim \mathcal {U} (S ^ {d - 1})} \left(\| \boldsymbol {Z} ^ {\top} \boldsymbol {z} \| _ {\infty} \leq \frac {\zeta}{\sqrt {d}} + \delta^ {\prime}\right) \\ \leq \left| \mathcal {D} _ {\delta^ {\prime}} \right| \cdot \left(\sqrt {2} \left(\zeta + \delta^ {\prime} \sqrt {d}\right)\right) ^ {\lceil k / 5 \rceil}. \\ \end{array}
$$
Hence, using the same arguments as in the original proof, we obtain
$$
H \left(\boldsymbol {A} ^ {\prime} \mid \boldsymbol {Y}\right) \leq (n - m) \left(\log_ {2} \left| \mathcal {D} _ {\delta^ {\prime}} \right| + \mathbb {P} (\mathcal {E}) \cdot \frac {k}{5} \log_ {2} \left(\sqrt {2} \left(\zeta + \delta^ {\prime} \sqrt {d}\right)\right)\right),
$$
where  $\mathcal{E}$  is the event when the algorithm succeeds at the OVGH game. In the next step, we need to bound  $H(\boldsymbol{A} \mid \boldsymbol{V}) - H(\boldsymbol{G}, \boldsymbol{j}, \boldsymbol{c})$  where  $\boldsymbol{V}$  stores hints received throughout the game,  $\boldsymbol{G}$  stores observed rows of  $\boldsymbol{A}$  during the game, and  $\boldsymbol{j}, \boldsymbol{c}$  are auxiliary variables. The latter can be treated as in the original proof. We obtain
$$
\begin{array}{l} H (\boldsymbol {A} \mid \boldsymbol {V}) - H (\boldsymbol {G}, \boldsymbol {j}, \boldsymbol {c}) \geq H (\boldsymbol {A}) - H (\boldsymbol {G}) - I (\boldsymbol {A}; \boldsymbol {V}) - 3 m \log_ {2} (2 d) \\ \geq (n - m) \log_ {2} | \mathcal {D} _ {\delta^ {\prime}} | - 3 m \log_ {2} (2 d) - I (\boldsymbol {A}, \boldsymbol {V}). \\ \end{array}
$$
Now the same arguments as in the original proof show that we still have  $I(\mathbf{A}, \mathbf{V}) \leq 3km\log_2d + 1$ , and that as a result, if  $M$  is the number of bits stored in memory,
$$
M \geq \frac {k}{1 0} \log_ {2} \left(\frac {1}{\sqrt {2} (\zeta + \delta^ {\prime} \sqrt {d})}\right) (n - m) - 3 k m \log_ {2} d - 1 - 3 d \log_ {2} (2 d).
$$
Then, with the same arguments as in the original proof, we can conclude.
We are now ready to prove Theorem C.1. With the parameter  $k = \lceil 20\frac{M + 3d\log(2d) + 1}{n\log_2(\sqrt{2}(\epsilon d^4 / 2 + \delta'\sqrt{d}))^{-1}} \rceil$  and the same arguments, we show that an algorithm solving the convex optimization up to precision  $\eta /(2\sqrt{d}) = \epsilon$  yields an algorithm solving the OVGH where the parameters  $\alpha = \frac{2\eta}{\gamma_1}$  and  $\beta = \frac{\gamma_2}{4}$  satisfy
$$
\alpha \left(\frac {\sqrt {d}}{\beta}\right) ^ {5 / 4} \leq \frac {\eta d ^ {3}}{4} = \frac {d ^ {3 . 5} \epsilon}{2}.
$$
We can then apply Proposition C.2 with  $\zeta = d^4\epsilon /2$ . Hence, if  $Q$  is the maximum number of queries of the convex optimization algorithm, we obtain
$$
\lceil Q / p _ {m a x} \rceil + 1 \geq \frac {1}{8} \left(1 + \frac {3 0 \log_ {2} d}{\log_ {2} \frac {1}{d ^ {4} \epsilon} - 1 / 2}\right) ^ {- 1} d \geq \frac {d}{8 \cdot 6 1},
$$
where in the last inequality we used  $\epsilon \leq 1 / (2d^{4.5})$ . As a result, with the same arguments, we obtain
$$
Q = \Omega \left(\frac {d ^ {5 / 3} \ln^ {1 / 3} \frac {1}{\epsilon}}{(M + \ln d) ^ {1 / 3} \ln^ {2 / 3} d}\right).
$$
This ends the proof of Theorem C.1.
C.1.2 Lower bound of [5] for feasibility problems and deterministic algorithms
We improve the memory dependence by showing the following result.
Theorem C.3. For  $\epsilon = 1 / (48d^3)$  and any  $\delta \in [0,1]$ , a deterministic algorithm guaranteed to solve the feasibility problem over the unit ball with  $\epsilon$  accuracy uses at least  $d^{2 - \delta}\ln \frac{1}{\epsilon}$  bits of memory or makes at least  $\tilde{\Omega} (d^{1 + \delta})$  queries.
We use the exact same class of feasibility problems and only change the parameter  $\eta_0$  which constrained successful points to satisfy  $\| A\pmb{x}\|_{\infty} \leq \eta_0$ , as well as the rows of  $\pmb{A}$  that are sampled i.i.d. from  $\mathcal{U}_{\delta}$ . The other parameter  $\eta_1 = 1 / (2\sqrt{d})$  is unchanged. We also take  $\delta' = \epsilon$ . Because the rows of  $\pmb{A}$  are already normalized, we can take  $\eta_0 = \epsilon$  directly. Then, the same proof as in [5] shows that if an algorithm solves feasibility problems with accuracy  $\epsilon$ , there is an algorithm for OVGH for parameters  $\alpha = \eta / \eta_1$  and  $\beta = \eta_1 / 2$ . Then, we have  $\alpha (\sqrt{d} / \beta)^{5/4} \leq 12d^2 \eta_0$  and we can apply Proposition C.2 with  $\zeta = 12d^{2.5} \eta_0 = 12d^{2.5} \epsilon$ . Similar computations as above then show that  $m \geq d / (8 \cdot 61)$ , with  $k = \Theta\left(\frac{M + \ln d}{d \ln \frac{1}{\epsilon}}\right)$ , so that the query lower bound finally becomes
$$
Q \geq \Omega \left(\frac {d ^ {3} \ln \frac {1}{\epsilon}}{(M + \ln d) \ln^ {2} d}\right).
$$
Remark C.1. The more careful analysis--involving the discretization  $\mathcal{D}_{\delta}$  of the unit sphere at scale  $\delta$  instead of the hypercube  $\{\pm 1\}^{d}$  --allowed to add a  $\ln \frac{1}{\epsilon}$  factor to the final query lower bound but also an additional  $\ln d$  factor for both convex-optimization and feasibility-problem results. Indeed, the improved Proposition C.2 shows that the OVGH with adequate parameters requires  $\mathcal{O}(d)$  queries, instead of  $\mathcal{O}(d / \ln d)$  in [5, Proposition 14]. At a high level, each hint queried brings information  $\mathcal{O}(d\ln d)$  but memorizing a binary matrix  $A\in \{\pm 1\}^{[d / 4]\times d}$  only requires  $d^2$  bits of memory: hence the query lower bound is limited to  $\mathcal{O}(d / \ln d)$ . Instead, memorizing the matrix  $A$  where each row lies in  $\mathcal{D}_{\delta}$  requires  $\Theta (d^{2}\ln \frac{1}{\epsilon})$  memory, hence querying  $d$  hints (total information  $\mathcal{O}(d^2\ln d)$ ) is not prohibitive for the lower bound.
C.1.3 Lower bound of [31] for convex optimization and randomized algorithms
We aim to improve the result to obtain the following.
Theorem C.4. For  $\epsilon \leq 1 / d^4$  and any  $\delta \in [0,1]$ , any (potentially randomized) algorithm guaranteed to minimize 1-Lipschitz convex functions over the unit ball with  $\epsilon$  accuracy uses at least  $d^{1.25 - \delta}\ln \frac{1}{\epsilon}$  bits of memory or makes  $\tilde{\Omega} (d^{1 + 4\delta /3})$  queries.
The distribution considered in [31] is given by the functions
$$
\frac{1}{d^{6}}\max \left\{d^{5}\| \boldsymbol {A}\boldsymbol {x}\|_{\infty} - 1,\max_{i\in [N]}(\boldsymbol{v}_{i}^{\top}\boldsymbol {x} - i\gamma)\right\} ,
$$
where  $N \leq d$  is a parameter,  $\mathbf{A}$  has  $\lfloor d / 2\rfloor$  rows drawn i.i.d. from  $\mathcal{U}(\{\pm 1\} ^d)$ , and the vectors  $\pmb{v}_i$  are drawn i.i.d. from the rescaled hypercube  $\pmb{v}_i \sim \mathcal{U}(d^{-1 / 2}\{\pm 1\}^d)$ . We adapt the class of functions by simply changing pre-factors as follows
$$
\mu \max  \left\{\frac {1}{\mu} \| \boldsymbol {A} \boldsymbol {x} \| _ {\infty} - 1, \max  _ {i \in [ N ]} \left(\boldsymbol {v} _ {i} ^ {\top} \boldsymbol {x} - i \gamma\right) \right\}, \tag {8}
$$
where  $\mathbf{A}$  has the same number of rows but they are drawn i.i.d. from  $\mathcal{U}_{\delta}$ , and  $\delta, \mu > 0$  are parameters to specify. We use the notation  $\mu$  instead of  $\eta$  as in the previous sections because [31] already use a parameter  $\eta$  which in our context can be interpreted as  $\eta = 1 / (\mu \sqrt{d})$ . We choose the parameters  $\mu = 16\sqrt{d}\epsilon$  and  $\delta' = \epsilon$ .
Again, as for the previous sections, the original proof can be directly used to show that if an algorithm is guaranteed to find a  $\frac{\mu}{16\sqrt{N}} (\geq \epsilon)$ -suboptimal point for the above function class, there is an algorithm that wins at their Orthogonal Vector Game (OVG) [31, Game 1], with the only difference that the parameter  $d^{-4}$  (1.8 of OVG) is replaced by  $\sqrt{d}\mu$ . OVG requires the output to be robustly-independent (defined in [31]) and effectively corresponds to  $\beta = 1 / d^2$  in OVGH. As a result, there is a successful algorithm for the OVGH with parameters  $\alpha = \sqrt{d}\mu$  and  $\beta = 1 / d^2$  and that even completely ignores the hints. Hence, we can now directly use Proposition C.2 with  $\zeta = d^{1 + 25 / 16}\mu$  (from the assumption  $\epsilon \leq d^{-4}$  we have  $\zeta \leq 1 / \sqrt{d}$ ). This shows that with the adequate choice of  $k = \Theta \left( \frac{M + d \ln d}{d \ln \frac{1}{\epsilon}} \right)$ , the query lower bound is  $\Omega(d)$ .
Putting things together, a potentially randomized algorithm for convex optimization that uses  $M$  memory makes at least the following number of queries
$$
Q \geq \Omega \left(\frac {N d}{k}\right) = \Omega \left(\frac {d ^ {4 / 3}}{\ln^ {1 / 3} d} \left(\frac {d \ln \frac {1}{\epsilon}}{M + d \ln d}\right) ^ {4 / 3}\right).
$$
C.2 Proof sketch for improving the query-complexity lower bound
We now turn to improving the query-complexity lower bound by a factor  $\frac{\ln\frac{1}{\epsilon}}{\ln d}$ . At the high level, the idea is to replicate these constructed "difficult" class of functions at  $\frac{\ln\frac{1}{\epsilon}}{\ln d}$  different scales or levels, similarly to the manner that the historical  $\Omega(d\ln\frac{1}{\epsilon})$  lower bound is obtained for convex optimization [34]. This argument is relatively standard and we only give details in the context of improving the bound from [31] for randomized algorithms in convex optimization for conciseness. This result uses a simpler class of functions, which greatly eases the exposition. We first present the construction with 2 levels, then present the generalization to  $p = \Theta\left(\frac{\ln\frac{1}{\epsilon}}{\ln d}\right)$  levels. For convenience, we write
$$
Q (\epsilon ; M, d) = \Omega \left(\frac {d ^ {4 / 3}}{\ln^ {1 / 3} d} \left(\frac {d \ln \frac {1}{\epsilon}}{M + d \ln d}\right) ^ {4 / 3}\right).
$$
This is the query lower bound given in Theorem C.5 for convex optimization algorithms with memory  $M$  that optimize the defined class of functions (Eq (8)) to accuracy  $\epsilon$ .
C.2.1 Construction of a bi-level class of functions  $F_{A,v_1,v_2}$  to optimize
In the lower-bound proof, [31] introduce the point
$$
\bar {\boldsymbol {x}} = - \frac {1}{2 \sqrt {N}} \sum_ {i \in [ N ]} P _ {\boldsymbol {A} ^ {\perp}} (\boldsymbol {v} _ {i}),
$$
where  $P_{\mathbf{A}^{\perp}}$  is the projection onto the orthogonal space to the rows of  $\mathbf{A}$ . They show that with failure probability at most  $2 / d$ ,  $\bar{x}$  has good function value
$$
F _ {\boldsymbol {A}, \boldsymbol {v}} (\bar {\boldsymbol {x}}) := \mu \max \left\{\frac {1}{\mu} \| \boldsymbol {A} \bar {\boldsymbol {x}} \| _ {\infty} - 1, \max _ {i \in [ N ]} (\boldsymbol {v} _ {i} ^ {\top} \bar {\boldsymbol {x}} - i \gamma) \right\} \leq - \frac {\mu}{8 \sqrt {N}}.
$$
Figure 4: Representation of the procedure to rescale the optimization function.
This is shown in [31, Lemma 25]. On the other hand, from Theorem C.4, during the first
$$
Q _ {1} = Q (\epsilon ; M, d)
$$
queries of any algorithm, with probability at least  $1/3$ , all queries are at least  $\mu/(16\sqrt{N})$ -suboptimal compared to  $\bar{x}$  in function value [31, Theorem 28, Lemma 14 and Theorem 16]. Precisely, if  $F_{A,v}$  is the sampled function to optimize, with probability at least  $1/3$ ,
$$
F _ {\boldsymbol {A}, \boldsymbol {v}} (\boldsymbol {x} _ {t}) \geq F _ {\boldsymbol {A}, \boldsymbol {v}} (\bar {\boldsymbol {x}}) + \frac {\mu}{1 6 \sqrt {N}} \geq F _ {\boldsymbol {A}, \boldsymbol {v}} (\bar {\boldsymbol {x}}) + \frac {\mu}{1 6 \sqrt {d}}, \quad \forall t \leq Q _ {1}.
$$
As a result, we can replicate the term  $\max_{i\in [N]}(\pmb{v}_i^\top \pmb{x} - i\gamma)$  at a smaller scale within the ball  $B_{d}(\bar{\pmb{x}},1 / (16\sqrt{d}))$ . For convenience, we introduce  $\xi_2 = 1 / (16\sqrt{d})$  which will be the scale of the duplicate function. We separate the wall term  $\| \pmb {A}\pmb {x}\|_{\infty} - \mu$  for convenience. Hence, we define
$$
G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}} (\boldsymbol {x}) := \mu \max  _ {i \in [ N ]} \left(\boldsymbol {v} _ {1, i} ^ {\top} \boldsymbol {x} - i \gamma\right)
$$
$$
G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \boldsymbol {v} _ {2}} (\boldsymbol {x}) := \max  \left\{G _ {\boldsymbol {A}, \boldsymbol {v} ^ {(1)}} (\boldsymbol {x}), G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}} (\bar {\boldsymbol {x}}) + \frac {\mu \xi_ {2}}{3} \cdot \right.
$$
$$
\left. \right.\left.\left. \max  \left\{1 + \| \boldsymbol {x} - \bar {\boldsymbol {x}} \| _ {2}, 1 + \frac {\xi_ {2}}{6} + \frac {\xi_ {2}}{1 8} \max  _ {i \in [ N ]} \left(\boldsymbol {v} _ {2, i} ^ {\top} \left(\frac {\boldsymbol {x} - \bar {\boldsymbol {x}}}{\xi_ {2} / 9}\right) - i \gamma\right)\right\}\right\}\right\}
$$
An illustration of the construction is given in Fig. 4. The resulting optimization functions are given by adding the wall term:
$$
F _ {\boldsymbol {A}, \boldsymbol {v} _ {1}} (\boldsymbol {x}) = \max  \left\{\| \boldsymbol {A} \boldsymbol {x} \| _ {\infty} - \mu , G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}} (\boldsymbol {x}) \right\}
$$
$$
F _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \boldsymbol {v} _ {2}} (\boldsymbol {x}) = \max  \left\{\| \boldsymbol {A} \boldsymbol {x} \| _ {\infty} - \mu , G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \boldsymbol {v} _ {2}} (\boldsymbol {x}) \right\}
$$
We first explain the choice of parameters. First observe that since  $\| A\bar{x} \| = 0$ , we have  $G_{A,v_1}(\bar{x}) = F_{A,v_1}(\bar{x})$ . We can then check that for all  $x \in B_d(0,1)$ ,
$$
G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \boldsymbol {v} _ {2}} (\boldsymbol {x}) \leq \max  \left\{G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}} (\boldsymbol {x}), G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}} (\bar {\boldsymbol {x}}) + \frac {2}{3} \mu \xi_ {2} \right\}. \tag {9}
$$
Further, for any  $\pmb{x} \in B_d(\bar{x}, \xi_2/3)$ , since  $F_{\pmb{A}, \pmb{v}_1}$  is 1-Lipschitz, we can easily check that
$$
\begin{array}{l} G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \boldsymbol {v} _ {2}} (\boldsymbol {x}) - G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}} (\bar {\boldsymbol {x}}) \\ = \frac {\mu \xi_ {2}}{3} \max  \left\{1 + \| \boldsymbol {x} - \bar {\boldsymbol {x}} \| _ {2}, 1 + \frac {\xi_ {2}}{6} + \frac {\xi_ {2}}{1 8} \max  _ {i \in [ N ]} \left(\boldsymbol {v} _ {2, i} ^ {\top} \left(\frac {\boldsymbol {x} - \bar {\boldsymbol {x}}}{\xi_ {2} / 9}\right) - i \gamma\right) \right\} \leq \frac {2}{3} \mu \xi_ {2}. \\ \end{array}
$$
Thus,  $G_{\mathbf{A},v_1,v_2}(\pmb{x})$  does not coincide with  $G_{\mathbf{A},v_1}(\pmb{x})$  on  $B_d(\bar{\pmb{x}},\xi_2 / 3)$ . Then, the  $\| \pmb{x} - \bar{\pmb{x}} \|_2$  term ensures that any minimizer of  $G_{\mathbf{A},v_1,v_2}$  is contained within the closed ball  $B_d(\bar{\pmb{x}},\xi_2 / 3)$ . Also, to obtain a  $\mu \xi_2 / 3$ -suboptimal solution of  $F_{\mathbf{A},v_1,v_2}$ , the algorithm needs to find what would be a  $\mu \xi_2$ -suboptimal solution of  $F_{\mathbf{A},v_1}$ , while receiving the same response as when optimizing the latter.
Next, for any  $\pmb{x} \in B_d(\bar{\pmb{x}}, \xi_2 / 9)$ , the term  $\max_{i \in [N]} \left( \pmb{v}_{2,i}^\top \left( \frac{\pmb{x} - \bar{\pmb{x}}}{\xi_2 / 9} \right) - i\gamma \right)$  lies in  $[-1,1]$ . Hence, we can check that for  $\pmb{x} \in B_d(\bar{\pmb{x}}, \xi_2 / 9)$ ,
$$
G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \boldsymbol {v} _ {2}} (\boldsymbol {x}) = G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}} (\bar {\boldsymbol {x}}) + \frac {\mu \xi_ {2}}{3} + \frac {\mu \xi_ {2} ^ {2}}{1 8} + \frac {\mu \xi_ {2} ^ {2}}{5 4} \max  _ {i \in [ N ]} \left(\boldsymbol {v} _ {2, i} ^ {\top} \left(\frac {\boldsymbol {x} - \bar {\boldsymbol {x}}}{\xi_ {2} / 9}\right) - i \gamma\right). \tag {10}
$$
We now argue that  $F_{A,v_1,v_2}$  acts as a duplicate function. Until the algorithm reaches a point with function value at most  $G_{A,v_1}(\bar{x}) + \mu \xi_2$ , the optimization algorithm only receives responses consistent with the function  $F_{A,v_1}$  by Eq (9). Next, all minimizers of  $F_{A,v_1,v_2}$  are contained in  $B_d(\bar{x},\xi_2 / 3)$  which was the goal of introducing the term in  $\| x - \bar{x}\| _2$ . As a result, optimizing  $F_{A,v_1,v_2}$  on this ball is equivalent to minimizing
$$
\tilde {F} _ {\boldsymbol {A}, \boldsymbol {v} _ {2}} (\boldsymbol {y}) = \max  \left\{\| \boldsymbol {A} \boldsymbol {y} \| _ {\infty} - \mu_ {2}, c _ {2} + \nu_ {2} \max  _ {i \in [ N ]} (\boldsymbol {v} _ {2, i} ^ {\top} \boldsymbol {y} - i \gamma), c _ {2} ^ {\prime} + \nu_ {2} ^ {\prime} \| \boldsymbol {y} \| \right\}, \quad \boldsymbol {y} \in B _ {d} (0, 3),
$$
where  $\pmb{y} = \frac{\pmb{x} - \bar{\pmb{x}}}{\xi_2 / 9}$ . The function has been rescaled by a factor  $\xi_2 / 9$  compared to  $F_{\mathbf{A},\pmb{v}_1,\pmb{v}_2}$  so that  $\mu_2 = \frac{9\mu}{\xi_2}$ ,  $\nu_2 = \frac{\mu\xi_2}{6}$ ,  $\nu_2' = 6\mu$ ,  $c_2 = \frac{9}{\xi_2} G_{\mathbf{A},\pmb{v}_1}(\bar{\pmb{x}}) + 3\mu + \frac{\mu\xi_2}{2}$ , and  $c_2' = \frac{9}{\xi_2} G_{\mathbf{A},\pmb{v}_1}(\bar{\pmb{x}}) + 3\mu$ . By Eq (10), the two first terms of  $\tilde{F}_{\mathbf{A},\pmb{v}_1}$  are preponderant for  $\pmb{y} \in B_d(0,1)$ .
The form of  $\tilde{F}_{A,v_2}$  is very similar to the original form of functions
$$
F _ {\boldsymbol {A}, \boldsymbol {v} _ {2}} = \max  \left\{\| \boldsymbol {A} \boldsymbol {y} \| _ {\infty} - \mu_ {1} ^ {\prime}, \mu_ {2} ^ {\prime} \max  _ {i \in [ N ]} (\boldsymbol {v} _ {2, i} ^ {\top} \boldsymbol {y} - i \gamma) \right\},
$$
In fact, the same proof structure for the query-complexity/memory lower-bound can be applied in this case. The main difference is that originally one had  $\mu_1' = \mu_2'$ ; here we would instead have  $\mu_1' = \mu_2 + c_2 = \Theta(\mu/\xi_2)$  and  $\mu_2' = \nu_2 = \Theta(\mu\xi_2)$ . Intuitively, this corresponds to increasing the accuracy to  $\Theta(\epsilon\xi_2^2)$ —a factor  $\xi_2$  is due to the fact that  $\tilde{F}_{A,v_2}$  was rescaled by a factor  $\xi_2/9$  compared to  $F_{A,v_1,v_2}$ , and a second factor  $\xi_2$  is due to the fact that within  $\tilde{F}_{A,v_2}$ , we have  $\mu_2' = \Theta(\mu\xi_2)$ —while the query lower bound is similar to that obtained for  $\Theta(\epsilon/\xi_2)$ . As a result, during the first
$$
Q _ {2} = Q \left(\Theta \left(\frac {\epsilon}{\xi_ {2}}\right); M, d\right)
$$
queries of any algorithm optimizing  $\tilde{F}_{\mathbf{A},\mathbf{v}_2}$ , with probability at least  $1/3$  on the sample of  $\mathbf{A}$  and  $\mathbf{v}_2$ , all queries are at least  $\Theta(\epsilon \xi_2)$ -suboptimal compared to
$$
\bar {\boldsymbol {y}} = - \frac {1}{2 \sqrt {N}} \sum_ {i \in [ N ]} P _ {\boldsymbol {A} ^ {\perp}} (\boldsymbol {v} _ {2, i}).
$$
We are now ready to give lower bounds on the queries of an algorithm minimizing  $F_{A,v_1,v_2}$  to accuracy  $\Theta(\epsilon \xi_2^2)$ . Let  $T_2$  be the index of the first query with function value at most  $G_{A,v_1}(\bar{x}) + \mu \xi_2$ . We already checked that before that query, all responses of the oracle are consistent with minimizing  $F_{A,v_1}$ , hence on an event  $\mathcal{E}_1$  of probability at least  $1/3$ , one has  $T_2 \geq Q_1$ . Next, consider the hypothetical case when at time  $T_2$ , the algorithm is also given the information of  $\bar{x}$  and is allowed to store this vector. Given this information, optimizing  $F_{A,v_1,v_2}$  reduces to optimizing  $\tilde{F}_{A,v_2}$  since we already know that the minimum is achieved within  $B_d(\bar{x},\xi_2/3)$ . Further, any query outside of this ball either
- returns a vector  $\mathbf{v}_{1,i}$  which does not give any useful information for the minimization  $(\mathbf{v}_1$  and  $\mathbf{v}_2$  are sampled independently and  $\bar{x}$  is given),
- or returns a row from  $A$ , as covered by the original proof.
Hence, on an event  $\mathcal{E}_2$  of probability at least  $1/3$ , even with the extra information of  $\bar{\pmb{x}}$ , during the next  $Q_2$  queries starting from  $T_2$ , the algorithm does not query a  $\Theta(\mu \xi_2^3)$ -suboptimal solution to  $F_{\pmb{A},\pmb{v}_1,\pmb{v}_2}$ . This holds a fortiori for the model when the algorithm is not given  $\bar{\pmb{x}}$  at time  $T_2$ .
C.2.2 Recursive construction of a  $p$ -level class of functions  $F_{A,v_1,\dots,v_p}$
Similarly as in the last section, one can inductively construct the sequence of functions  $F_{A,v_1}$ ,  $F_{A,v_1,v_2}$ ,  $F_{A,v_1,v_2,v_3}$ , etc. Formally, the induction is constructed as follows: let  $(v_p)_{p\geq 1}$  be an i.i.d. sequence of  $N$  i.i.d. vectors  $(v_{k,i})_{i\in [N]}$  sampled from the rescaled hypercube  $d^{-1 / 2}\{\pm 1\}^d$ . Next, we pose
$$
G _ {\pmb {A}, \pmb {v} _ {1}} (\pmb {x}) = \mu^ {(1)} \max _ {i \in [ N ]} \left(\pmb {v} _ {1, i} ^ {\top} \left(\frac {\pmb {x} - \bar {\pmb {x}} ^ {(1)}}{s ^ {(1)}}\right) - i \gamma\right),
$$
where  $\mu^{(1)} = \mu, \bar{\pmb{x}}^{(1)} = \mathbf{0}$  and  $s^{(1)} = 1$ . For  $k \geq 1$ , we pose
$$
\bar {\boldsymbol {x}} ^ {(k + 1)} = \bar {\boldsymbol {x}} ^ {(k)} - \frac {s ^ {(k)}}{2 \sqrt {N}} \sum_ {i \in [ N ]} P _ {\boldsymbol {A} ^ {\perp}} (\boldsymbol {v} _ {k, i}), \quad \text {a n d} \quad F ^ {(k)} := G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \dots , \boldsymbol {v} _ {k}} (\bar {\boldsymbol {x}} ^ {(k)}) + \mu^ {(k)} \xi_ {k + 1},
$$
for a certain parameter  $\xi_{k + 1}$  to be specified. We then define the next level as
$$
\begin{array}{l} G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \dots , \boldsymbol {v} _ {k + 1}} (\boldsymbol {x}) := \max  \left\{G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \dots , \boldsymbol {v} _ {k}} (\boldsymbol {x}), G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \dots , \boldsymbol {v} _ {k}} (\bar {\boldsymbol {x}} ^ {(k + 1)}) + \frac {\mu^ {(k)} \xi_ {k + 1}}{3} \cdot \right. \\ \left. \max  \left\{1 + \frac {\| \boldsymbol {x} - \bar {\boldsymbol {x}} ^ {(k + 1)} \| _ {2}}{s ^ {(k)}}, 1 + \frac {\xi_ {k + 1}}{6} + \frac {\xi_ {k + 1}}{1 8} \max  _ {i \in [ N ]} \left(\boldsymbol {v} _ {k + 1, i} ^ {\top} \left(\frac {\boldsymbol {x} - \bar {\boldsymbol {x}} ^ {(k + 1)}}{s ^ {(k)} \xi_ {k + 1} / 9}\right) - i \gamma\right) \right\} \right\}. \\ \end{array}
$$
We then pose  $\mu^{(k + 1)}\coloneqq \mu^{(k)}\xi_{k + 1}^2 /54$  and  $s^{(k + 1)}\coloneqq s^{(k)}\xi_{k + 1} / 9$ , which closes the induction. The optimization functions are defined simply as
$$
F _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \dots , \boldsymbol {v} _ {k + 1}} (\boldsymbol {x}) = \max  \left\{\| \boldsymbol {A} \boldsymbol {x} \| _ {\infty} - \mu , G _ {\boldsymbol {A}, \boldsymbol {v} _ {1}, \dots , \boldsymbol {v} _ {k + 1}} (\boldsymbol {x}) \right\}.
$$
We checked before that we can use  $\xi_{2} = 1 / (16\sqrt{d})$ . For general  $k \geq 0$ , given that the form of the function slightly changes to incorporate the absolute term (see  $\tilde{F}_{A,\mathbf{v}_2}$ ), this constant may differ slightly. In any case, one has  $\xi_{k} = \Theta(1 / \sqrt{d})$ . Now fix a construction level  $p \geq 1$  and for any  $k \in [p]$ , let  $T_{k}$  be the first time that a point with function value at most  $F^{(k)}$  is queried. For convenience let  $T_{0} = 0$ . Using the same arguments as above recursively, we can show that on an event  $\mathcal{E}_{k}$  with probability at least  $1/3$ ,
$$
T _ {k} - T _ {k - 1} \geq Q _ {k} = Q \left(\Theta \left(\frac {\mu}{s ^ {(k)}}\right); M, d\right)
$$