SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To illustrate the power of Theorem 1, we specialize it to several models to obtain concrete uniform signal recovery results. Starting with Theorem 1, the remaining work is to select parameters that justify Assumptions 2-4. We summarize the strategy as follows: (i) Determine the parameters in Assumption 2 by the measurement model; (ii) Set  $T$  that verifies (2.5) (see Lemmas 8-11 for the following models); (iii) Set the parameters in Assumption 3, for which bounding the norm of Gaussian vector is useful; (iv) Set  $\beta_{1}$  to guarantee (2.6) based on some standard probability argument. We only provide suitable parameters for the following concrete models due to space limit, while leaving more details to Appendix E.
(A) 1-bit GCS. Assume that we have the 1-bit observations  $y_{i} = \mathrm{sign}(\pmb{a}_{i}^{\top}\pmb{x}^{*})$ ; then  $f_{i}(\cdot) = f(\cdot) = \mathrm{sign}(\cdot)$  satisfies Assumption 2 with  $(B_0,L_0,\beta_0) = (2,0,\infty)$ . In this model, it is hopeless to recover the norm of  $\| x^{*}\|_{2}$ ; as done in previous work, we assume  $x^{*}\in \mathcal{K}\subset \mathbb{S}^{n - 1}$  [31, Remark 1]. We set  $T = \sqrt{2 / \pi}$  and take the parameters in Assumption 3 as  $A_{g}^{(1)}\asymp 1,U_{g}^{(1)}\asymp \sqrt{n},P_{0}^{(1)}\asymp \exp (-\Omega (n)),A_{g}^{(2)}\asymp 1,U_{g}^{(2)}\asymp 1,P_{0}^{(2)} = 0$ . We take  $\beta = \beta_{1}\asymp \frac{k}{m}$  to guarantee (2.6). With these choices, Theorem 1 specializes to the following:
Corollary 1. Consider Assumption 1 with  $f_{i}(\cdot) = \mathrm{sign}(\cdot)$  and  $\mathcal{K} \subset \mathbb{S}^{n - 1}$ , let  $\epsilon \in (0,1)$  be any given recovery accuracy. If  $m \gtrsim \frac{k}{\epsilon^2} \log \left( \frac{Lr\sqrt{mn}}{\epsilon \wedge (k / m)} \right)$ , then with probability at least  $1 - 2m \exp(-cn) - m \exp(-\Omega(k))$  on a single draw of  $(\pmb{a}_i)_{i=1}^m$ , we have the uniform signal recovery guarantee  $\| \hat{\pmb{x}} - \sqrt{\frac{2}{\pi}} \pmb{x}^* \|_2 \leq \epsilon$  for all  $\pmb{x}^* \in \mathcal{K}$ , where  $\hat{\pmb{x}}$  is the solution to (2.1) with  $\pmb{y} = \mathrm{sign}(\pmb{A}\pmb{x}^*)$  and  $T = \sqrt{\frac{2}{\pi}}$ .
Remark 1. A uniform recovery guarantee for generalized Lasso in 1-bit GCS was obtained in [33, Section 5]. Their proof relies on the local embedding property in [31]. Note that such geometric property is often problem-dependent and highly nontrivial. By contrast, our argument is free of geometric properties of this kind.
Remark 2. For traditional 1-bit CS, [17, Corollary 2] requires  $m \gtrsim \tilde{O}(k/\epsilon^4)$  to achieve uniform  $\ell_2$ -accuracy of  $\epsilon$  for all  $k$ -sparse signals, which is inferior to our  $\tilde{O}(k/\epsilon^2)$ . This is true for all remaining examples. To obtain such a sharper rate, the key technique is to use our Theorem 2 (rather than [36]) to obtain tighter bound for the product processes, as will be discussed in Remark 8.
(B) 1-bit GCS with dithering. Assume that the  $\pmb{a}_i^\top \pmb{x}^*$  is quantized to 1-bit with  $\text{dither}^5 \tau_i \stackrel{i:d}{\sim} \mathcal{U}[-\lambda, \lambda]$  for some  $\lambda$  to be chosen, i.e., we observe  $y_i = \text{sign}(\pmb{a}_i^\top \pmb{x}^* + \tau_i)$ . Following [45] we assume  $\mathcal{K} \subset \mathbb{B}_2^n(R)$  for some  $R > 0$ . Here, using dithering allows the recovery of signal norm  $\| \pmb{x}^* \|_2$ , so we do not need to assume  $\mathcal{K} \subset \mathbb{S}^{n-1}$  as in Corollary 1. We set  $\lambda = CR\sqrt{\log m}$  with sufficiently large  $C$ , and  $T = \lambda^{-1}$ . In Assumption 3, we take  $A_g^{(1)} \asymp 1$ ,  $U_g^{(1)} \asymp \sqrt{n}$ ,  $P_0^{(1)} \asymp \exp(-\Omega(n))$ ,  $A_g^{(2)} \asymp 1$ ,  $U_g^{(2)} \asymp 1$ , and  $P_0^{(2)} = 0$ . Moreover, we take  $\beta = \beta_1 = \frac{\lambda k}{m}$  to guarantee (2.6). Now we can invoke Theorem 1 to get the following.
Corollary 2. Consider Assumption 1 with  $f_{i}(\cdot) = \mathrm{sign}(\cdot +\tau_{i})$ ,  $\tau_{i}\sim \mathcal{U}[-\lambda ,\lambda ]$  and  $\kappa \subset \mathbb{B}_2^n (R)$  and  $\lambda = CR\sqrt{\log m}$  with sufficiently large  $C$ . Let  $\epsilon \in (0,1)$  be any given recovery accuracy. If  $m\gtrsim \frac{k}{\epsilon^2}\log \left(\frac{Lr\sqrt{m}n}{\lambda(\epsilon\wedge(k / m))}\right)$ , then with probability at least  $1 - 2m\exp (-cn) - m\exp (-\Omega (k))$  on a single draw of  $(\pmb {a}_i,\tau_i)_{i = 1}^m$ , we have the uniform signal recovery guarantee  $\| \hat{\pmb{x}} -\lambda^{-1}\pmb{x}^{*}\|_{2}\leq \epsilon$  for all  $x^{*}\in \mathcal{K}$ , where  $\hat{\pmb{x}}$  is the solution to (2.1) with  $y = \mathrm{sign}(A\pmb{x}^{*} + \pmb {\tau})$  (here,  $\pmb {\tau} = [\tau_1,\dots,\tau_m]^\top$ ) and  $T = \lambda^{-1}$ .
Remark 3. To our knowledge, the only related prior result is in [45, Theorem 3.2]. However, their result is restricted to ReLU networks. By contrast, we deal with the more general Lipschitz generative models; by specializing our result to the ReLU network that is typically  $(n^{\Theta (d)})$ -Lipschitz [2] ( $d$  is
Here and in other similar statements, we implicitly assume a large enough implied constant.
5Throughout this work, the random dither is independent of the  $\{a_i\}_{i = 1}^m$
the number of layers), our error rate coincides with theirs up to a logarithmic factor. Additionally, as already mentioned in the Introduction Section, our result can be generalized to a sensing vector with an unknown covariance matrix, unlike theirs which is restricted to isotropic sensing vectors. The advantage of their result is in allowing sub-exponential sensing vectors.
(C) Lipschitz-continuous SIM with generative prior. Assume that any realization of  $f$  is unconditionally  $\hat{L}$ -Lipschitz, which implies Assumption 2 with  $(B_0, L_0, \beta_0) = (0, \hat{L}, \infty)$ . We further assume  $\mathbb{P}(f(0) \leq \hat{B}) \geq 1 - P_0'$  for some  $(\hat{B}, P_0')$ . Because the norm of  $x^*$  is absorbed into the unknown  $f(\cdot)$ , we assume  $\mathcal{K} \subset \mathbb{S}^{n-1}$ . We set  $\beta = 0$  so that  $f_{i,\beta} = f_i$ . We introduce the quantities  $\mu = \mathbb{E}[f(g)g]$ ,  $\psi = \| f(g)\|_{\psi_2}$ , where  $g \sim \mathcal{N}(0,1)$ . We choose  $T = \mu$  and set parameters in Assumption 3 as  $A_g^{(1)} \asymp \psi + \mu$ ,  $U_g^{(1)} \asymp (\hat{L} + \mu)\sqrt{n} + \hat{B}$ ,  $P_0^{(1)} \asymp P_0' + \exp(-\Omega(n))$ ,  $A_g^{(2)} \asymp \psi + \mu$ ,  $U_g^{(2)} = 0$ ,  $P_0^{(2)} = 0$ . Now we are ready to apply Theorem 1 to this model. We obtain:
Corollary 3. Consider Assumption 1 with  $\hat{L}$ -Lipschitz  $f$ , suppose that  $\mathbb{P}(f(0) \leq \hat{B}) \geq 1 - P_0'$  and define the parameters  $\mu = \mathbb{E}[f(g)g]$ ,  $\psi = \| f(g)\|_{\psi_2}$  with  $g \sim \mathcal{N}(0,1)$ . Let  $\epsilon \in (0,1)$  be any given recovery accuracy. If  $m \gtrsim \frac{(\mu + \psi)k}{\epsilon^2} \log \left(\frac{Lr\sqrt{m}[n(\mu + \epsilon)(\hat{L} + \mu) + \sqrt{n}\mu\hat{B} + \psi]}{(\mu + \psi)\epsilon}\right)$ , then with probability at least  $1 - 2m\exp(-cn) - mP_0' - c_1\exp(-\Omega(k))$  on a single draw of  $(a_i, f_i)_{i=1}^m$ , we have the uniform signal recovery guarantee  $\|\hat{\pmb{x}} - \mu\pmb{x}^*\|_2 \leq \epsilon$  for all  $\pmb{x}^* \in \mathcal{K}$ , where  $\hat{\pmb{x}}$  is the solution to (2.1) with  $\pmb{y} = \pmb{f}(\pmb{A}\pmb{x}^*)$  and  $T = \mu$ .
Remark 4. While the main result of [33] is non-uniform, it was noted in [33, Section 5] that a similar uniform error rate can be established for any deterministic 1-Lipschitz  $f$ . Our result here is more general in that the  $\hat{L}$ -Lipschitz  $f$  is possibly random. Note that randomness on  $f$  is significant because it provides much more flexibility (e.g., additive random noise).
Remark 5. For SIM with unknown  $f_{i}$  it may seem impractical to use (2.1) as it requires  $\mu = \mathbb{E}[f(g)g]$  where  $g \sim \mathcal{N}(0,1)$ . However, by assuming  $\mu x^{*} \in \mathcal{K} = G(\mathbb{B}_{2}^{k}(r))$  as in [33], which is natural for sufficiently expressive  $G(\cdot)$ , we can simply use  $x \in \mathcal{K}$  as constraint in (2.1). Our Corollary 3 remains valid in this case under some inessential changes of  $\log \mu$  factors in the sample complexity.
(D) Uniformly quantized GCS with dithering. The uniform quantizer with resolution  $\delta > 0$  is defined as  $\mathcal{Q}_{\delta}(a) = \delta \left( \left\lfloor \frac{a}{\delta} \right\rfloor + \frac{1}{2} \right)$  for  $a \in \mathbb{R}$ . Using dithering  $\tau_{i} \sim \mathcal{U}[-\frac{\delta}{2}, \frac{\delta}{2}]$ , we suppose that the observations are  $y_{i} = \mathcal{Q}_{\delta}(\pmb{a}_{i}^{\top} \pmb{x}^{*} + \tau_{i})$ . This satisfies Assumption 2 with  $(B_{0}, L_{0}, \beta_{0}) = (\delta, 0, \delta)$ . We set  $T = 1$  and take parameters for Assumption 3 as follows:  $A_{g}^{(1)}, U_{g}^{(1)}, A_{g}^{(2)}, U_{g}^{(2)} \asymp \delta$ , and  $P_{0}^{(1)} = P_{0}^{(2)} = 0$ . We take  $\beta = \beta_{1} \asymp \frac{k\delta}{m}$  to confirm (2.6). With these parameters, we obtain the following from Theorem 1.
Corollary 4. Consider Assumption 1 with  $f(\cdot) = \mathcal{Q}_{\delta}(\cdot + \tau)$ ,  $\tau \sim \mathcal{U}[-\frac{\delta}{2}, \frac{\delta}{2}]$  for some quantization resolution  $\delta > 0$ . Let  $\epsilon > 0$  be any given recovery accuracy. If  $m \gtrsim \frac{\delta^2 k}{\epsilon^2} \log \left( \frac{L r \sqrt{m n}}{\epsilon \wedge [k \delta / (m \sqrt{n})]} \right)$ , then with probability at least  $1 - 2m \exp(-c n) - c_1 \exp(-\Omega(k))$  on a single draw of  $(\pmb{a}_i, \tau_i)_{i=1}^m$ , we have the uniform recovery guarantee  $\| \hat{\pmb{x}} - \pmb{x}^* \|_2 \leq \epsilon$  for all  $\pmb{x}^* \in \mathcal{K}$ , where  $\hat{\pmb{x}}$  is the solution to (2.1) with  $\pmb{y} = \mathcal{Q}_{\delta}(\pmb{A}\pmb{x} + \pmb{\tau})$  and  $T = 1$  (here,  $\pmb{\tau} = [\tau_1, \dots, \tau_m]^\top$ ).
Remark 6. While this dithered uniform quantized model has been widely studied in traditional CS (e.g., non-uniform recovery [8, 48], uniform recovery [17, 52]), it has not been investigated in GCS even for non-uniform recovery. Thus, this is new to the best of our knowledge.
A simple extension to the noisy model  $\pmb{y} = \pmb{f}(\pmb{A}\pmb{x}^{*}) + \pmb{\eta}$  where  $\pmb{\eta} \in \mathbb{R}^{m}$  has i.i.d. sub-Gaussian entries can be obtained by a fairly straightforward extension of our analysis; see Appendix F.
3 Proof Sketch
To provide a sketch of our proof, we begin with the optimality condition  $\| \pmb{y} - \pmb{A}\hat{\pmb{x}}\|_2^2 \leq \| \pmb{y} - \pmb{A}(T\pmb{x}^*)\|_2^2$ . We expand the square and plug in  $\pmb{y} = \pmb{f}(\pmb{A}\pmb{x}^*)$  to obtain
$$
\left\| \frac {\boldsymbol {A}}{\sqrt {m}} (\hat {\boldsymbol {x}} - T \boldsymbol {x} ^ {*}) \right\| _ {2} ^ {2} \leq \frac {2}{m} \langle \boldsymbol {f} (\boldsymbol {A} \boldsymbol {x} ^ {*}) - T \boldsymbol {A} \boldsymbol {x} ^ {*}, \boldsymbol {A} (\hat {\boldsymbol {x}} - T \boldsymbol {x} ^ {*}) \rangle . \tag {3.1}
$$
For the final goal  $\| \hat{\pmb{x}} - T\pmb{x}^{*}\|_{2} \leq \epsilon$ , up to rescaling, it is enough to prove  $\| \hat{\pmb{x}} - T\pmb{x}^{*}\|_{2} \leq 3\epsilon$ . We assume for convenience that  $\| \hat{\pmb{x}} - T\pmb{x}^{*}\|_{2} > 2\epsilon$ , without loss of generality. Combined with
$\hat{\pmb{x}}, T\pmb{x}^{*} \in T\mathcal{K}$ , we know  $\hat{\pmb{x}} - T\pmb{x}^{*} \in \mathcal{K}_{\epsilon}^{-}$ , where  $\mathcal{K}_{\epsilon}^{-} \coloneqq (T\mathcal{K}^{-}) \cap \left(\mathbb{B}_2^n(2\epsilon)\right)^c$ ,  $\mathcal{K}^{-} = \mathcal{K} - \mathcal{K}$ . We further define
$$
\left(\mathcal {K} _ {\epsilon} ^ {-}\right) ^ {*} := \left\{z / \| z \| _ {2}: z \in \mathcal {K} _ {\epsilon} ^ {-} \right\} \tag {3.2}
$$
where the normalized error lives, i.e.  $\frac{\hat{x} - T\pmb{x}^*}{\|\hat{x} - T\pmb{x}^*\|_2}\in (\mathcal{K}_\epsilon^-)^*$ . Our strategy is to establish a uniform lower bound (resp., upper bound) for the left-hand side (resp., the right-hand side) of (3.1). We emphasize that these bounds must hold uniformly for all  $\pmb{x}^{*}\in \mathcal{K}$ .
It is relatively easy to use set-restricted eigenvalue condition (S-REC) [2] to establish a uniform lower bound for the left-hand side of (3.1), see Appendix B.1 for more details. It is significantly more challenging to derive an upper bound for the right-hand side of (3.1). As the upper bound must hold uniformly for all  $x^{*}$ , we first take the supremum over  $x^{*}$  and  $\hat{x}$  and consider bounding the following:
$$
\begin{array}{l} \mathscr {R} := \frac {1}{m} \left\langle \boldsymbol {f} (\boldsymbol {A} \boldsymbol {x} ^ {*}) - T \boldsymbol {A} \boldsymbol {x} ^ {*}, \boldsymbol {A} \left(\hat {\boldsymbol {x}} - T \boldsymbol {x} ^ {*}\right) \right\rangle \\ = \frac {1}{m} \sum_ {i = 1} ^ {m} \left(f _ {i} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} ^ {*}\right) - T \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} ^ {*}\right) \cdot \left(\boldsymbol {a} _ {i} ^ {\top} [ \hat {\boldsymbol {x}} - T \boldsymbol {x} ^ {*} ]\right) \tag {3.3} \\ \leq \| \hat {\boldsymbol {x}} - T \boldsymbol {x} ^ {*} \| _ {2} \cdot \sup  _ {\boldsymbol {x} \in \mathcal {K}} \sup  _ {\boldsymbol {v} \in (\mathcal {K} _ {\epsilon} ^ {-}) ^ {*}} \frac {1}{m} \sum_ {i = 1} ^ {m} \left(f _ {i} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x}\right) - T \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x}\right) \cdot \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {v}\right) := \| \hat {\boldsymbol {x}} - T \boldsymbol {x} ^ {*} \| _ {2} \cdot \mathscr {R} _ {u}, \\ \end{array}
$$
where  $(\mathcal{K}_{\epsilon}^{-})^{*}$  is defined in (3.2). Clearly,  $\mathcal{R}_u$  is the supremum of a product process, whose factors are indexed by  $\pmb{x} \in \mathcal{K}$  and  $\pmb{v} \in (\mathcal{K}_{\epsilon}^{-})^{*}$ . It is, in general, challenging to control a product process, and existing results often require both factors to satisfy a certain "sub-Gaussian increments" condition (e.g., [36, 37]). However, the first factor of  $\mathcal{R}_u$  (i.e.,  $f_{i}(\pmb{a}_{i}^{\top}\pmb{x}^{*}) - T\pmb{a}_{i}^{\top}\pmb{x}^{*}$ ) does not admit such a condition when  $f_{i}$  is not continuous (e.g., the 1-bit model  $f_{i} = \mathrm{sign}(\cdot)$ ). We will construct the Lipschitz approximation of  $f_{i}$  to overcome this difficulty shortly in Section 3.1.
Remark 7. We note that these challenges stem from our pursuit of uniform recovery. In fact, a non-uniform guarantee for SIM was presented in [33, Theorem 1]. In its proof, the key ingredient is [33, Lemma 3] that bounds  $\mathcal{R}_u$  without the supremum on  $x$ . This can be done as long as  $f_{i}(\pmb{a}_{i}^{\top}\pmb{x}^{*})$  is sub-Gaussian, while the potential discontinuity of  $f_{i}$  is totally unproblematic.
3.1 Lipschitz Approximation
For any  $x_0 \in \mathcal{D}_{f_i}$  we define the one-sided limits as  $f_{i}^{-}(x_{0}) = \lim_{x\to x_{0}^{-}}f_{i}(x)$  and  $f_{i}^{+}(x_{0}) = \lim_{x\to x_{0}^{+}}f_{i}(x)$ , and write their average as  $f_{i}^{a}(x_{0}) = \frac{1}{2} (f_{i}^{-}(x_{0}) + f_{i}^{+}(x_{0}))$ . Given any approximation accuracy  $\beta \in (0,\frac{\beta_0}{2})$ , we construct the Lipschitz continuous function  $f_{i,\beta}$  as:
$$
f _ {i, \beta} (x) = \left\{ \begin{array}{c l} f _ {i} (x) & , \quad \text {i f} x \notin \mathcal {D} _ {f _ {i}} + \left[ - \frac {\beta}{2}, \frac {\beta}{2} \right] \\ f _ {i} ^ {a} (x _ {0}) - \frac {2 \left[ f _ {i} ^ {a} (x _ {0}) - f _ {i} \left(x _ {0} - \frac {\beta}{2}\right) \right] (x _ {0} - x)}{\beta}, & \text {i f} \exists x _ {0} \in \mathcal {D} _ {f _ {i}} \text {s . t .} x \in \left[ x _ {0} - \frac {\beta}{2}, x _ {0} \right] \\ f _ {i} ^ {a} (x _ {0}) + \frac {2 \left[ f _ {i} \left(x _ {0} + \frac {\beta}{2}\right) - f _ {i} ^ {a} (x _ {0}) \right] (x - x _ {0})}{\beta}, & \text {i f} \exists x _ {0} \in \mathcal {D} _ {f _ {i}}, \text {s . t .} x \in \left[ x _ {0}, x _ {0} + \frac {\beta}{2} \right] \end{array} . \right. \tag {3.4}
$$
We have defined the approximation error  $\varepsilon_{i,\beta}(\cdot) = f_{i,\beta}(\cdot) - f_i(\cdot)$  in Assumption 3. An important observation is that both  $f_{i,\beta}$  and  $|\varepsilon_{i,\beta}|$  are Lipschitz continuous (see Lemma 1 below). Here, it is crucial to consider  $|\varepsilon_{i,\beta}|$  rather than  $\varepsilon_{i,\beta}$  as the latter is not continuous; see Figure 1 for an intuitive graphical illustration and more explanations in Appendix B.2.
Lemma 1. With  $B_0, L_0, \beta_0$  given in Assumption 2, for any  $\beta \in (0, \frac{\beta_0}{2})$ ,  $f_{i,\beta}$  is  $(L_0 + \frac{B_0}{\beta})$ -Lipschitz over  $\mathbb{R}$ , and  $|\varepsilon_{i,\beta}|$  is  $(2L_0 + \frac{B_0}{\beta})$ -Lipschitz over  $\mathbb{R}$ .
3.2 Bounding the product process
We now present our technique to bound  $\mathcal{R}_u$ . Recall that  $\xi_{i,\beta}(a)$  and  $\varepsilon_{i,\beta}(a)$  were defined in (2.2). By Lemma 1,  $\xi_{i,\beta}$  is  $(L_0 + T + \frac{B_0}{\beta})$ -Lipschitz. Now we use  $f_i(a) - Ta = \xi_{i,\beta}(a) - \varepsilon_{i,\beta}$  to decompose  $\mathcal{R}_u$  (in the following, we sometimes shorten “ $\sup_{\boldsymbol{x} \in \mathcal{K}} \sup_{\boldsymbol{v} \in (\mathcal{K}_{\epsilon}^{-})^{*}}$ ” as “ $\sup_{\boldsymbol{x}, \boldsymbol{v}}$ ”:
$$
\mathcal {R} _ {u} \leq \underbrace {\sup  _ {\boldsymbol {x} , \boldsymbol {v}} \frac {1}{m} \sum_ {i = 1} ^ {m} \xi_ {i , \beta} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x}\right) \cdot \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {v}\right)} _ {\mathcal {R} _ {u 1}} + \underbrace {\sup  _ {\boldsymbol {x} , \boldsymbol {v}} \frac {1}{m} \sum_ {i = 1} ^ {m} \left| \varepsilon_ {i , \beta} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x}\right) \right| \left| \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {v} \right|} _ {\mathcal {R} _ {u 2}}. \tag {3.5}
$$
Figure 1: (Left):  $f_{i}$  and its approximation  $f_{i,0.5}$ ; (Right): approximation error  $\varepsilon_{i,0.5}$ ,  $|\varepsilon_{i,0.5}|$ .
It remains to control  $\mathcal{R}_{u1}$  and  $\mathcal{R}_{u2}$ . By the Lipschitz continuity of  $\xi_{i,\beta}$  and  $|\varepsilon_{i,\beta}|$ , the factors of  $\mathcal{R}_{u1}$  and  $\mathcal{R}_{u2}$  admit sub-Gaussian increments, so it is natural to first center them and then invoke the concentration inequality for product process due to Mendelson [36, Theorem 1.13], which we restate in Lemma 5 (Appendix A). However, this does not produce a tight bound and would eventually require  $\tilde{O}(k/\epsilon^4)$  to achieve a uniform  $\ell_2$ -error of  $\epsilon$ , as is the case in [17, Section 4].
In fact, Lemma 5 is based on Gaussian width and hence blind to the fact that  $\mathcal{K}, (\mathcal{K}_{\epsilon}^{-})^{*}$  here have low metric entropy (Lemma 6). By characterizing the low intrinsic dimension of index sets via metric entropy, we develop the following concentration inequality that can produce tighter bound for  $\mathcal{R}_{u1}$  and  $\mathcal{R}_{u2}$ . This also allows us to derive uniform error rates sharper than those in [17, Section 4].
Theorem 2. Let  $g_{\pmb{x}} = g_{\pmb{x}}(\pmb{a})$  and  $h_{\pmb{v}} = h_{\pmb{v}}(\pmb{a})$  be stochastic processes indexed by  $\pmb{x} \in \mathcal{X} \subset \mathbb{R}^{p_1}$ ,  $\pmb{v} \in \mathcal{V} \subset \mathbb{R}^{p_2}$ , both defined with respect to a common random variable  $\pmb{a}$ . Assume that:
- (A1.)  $g_{\pmb{x}}(\pmb{a})$ ,  $h_{\pmb{v}}(\pmb{a})$  are sub-Gaussian for some  $(A_g, A_h)$  and admit sub-Gaussian increments regarding  $\ell_2$  distance for some  $(M_g, M_h)$ :
$$
\left\| g _ {\boldsymbol {x}} (\boldsymbol {a}) - g _ {\boldsymbol {x} ^ {\prime}} (\boldsymbol {a}) \right\| _ {\psi_ {2}} \leq M _ {g} \| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| _ {2}, \left\| g _ {\boldsymbol {x}} (\boldsymbol {a}) \right\| _ {\psi_ {2}} \leq A _ {g}, \forall \boldsymbol {x}, \boldsymbol {x} ^ {\prime} \in \mathcal {X};
$$
$$
\left\| h _ {\boldsymbol {v}} (\boldsymbol {a}) - h _ {\boldsymbol {v} ^ {\prime}} (\boldsymbol {a}) \right\| _ {\psi_ {2}} \leq M _ {h} \left\| \boldsymbol {v} - \boldsymbol {v} ^ {\prime} \right\| _ {2}, \left\| h _ {\boldsymbol {v}} (\boldsymbol {a}) \right\| _ {\psi_ {2}} \leq A _ {h}, \forall \boldsymbol {v}, \boldsymbol {v} ^ {\prime} \in \mathcal {V}. \tag {3.6}
$$
- (A2.) On a single draw of  $\mathbf{a}$ , for some  $(L_g, U_g, L_h, U_h)$  the following events simultaneously hold with probability at least  $1 - P_0$ :
$$
\left| g _ {\boldsymbol {x}} (\boldsymbol {a}) - g _ {\boldsymbol {x} ^ {\prime}} (\boldsymbol {a}) \right| \leq L _ {g} \| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| _ {2}, \left| g _ {\boldsymbol {x}} (\boldsymbol {a}) \right| \leq U _ {g}, \forall \boldsymbol {x}, \boldsymbol {x} ^ {\prime} \in \mathcal {X};
$$
$$
\left| h _ {\boldsymbol {v}} (\boldsymbol {a}) - h _ {\boldsymbol {v} ^ {\prime}} (\boldsymbol {a}) \right| \leq L _ {h} \| \boldsymbol {v} - \boldsymbol {v} ^ {\prime} \| _ {2}, \left| h _ {\boldsymbol {v}} (\boldsymbol {a}) \right| \leq U _ {h}, \forall \boldsymbol {v}, \boldsymbol {v} ^ {\prime} \in \mathcal {V}.
$$
Let  $\pmb{a}_1, \dots, \pmb{a}_m$  be i.i.d. copies of  $\pmb{a}$ , and introduce the shorthand  $S_{g,h} = L_gU_h + M_gA_h$  and  $T_{g,h} = L_hU_g + M_hA_g$ . If  $m \gtrsim \mathcal{H}\left(\mathcal{X},\frac{A_gA_h}{\sqrt{m}S_{g,h}}\right) + \mathcal{H}\left(\mathcal{V},\frac{A_gA_h}{\sqrt{m}T_{g,h}}\right)$ , where  $\mathcal{H}(\cdot ,\cdot)$  is the metric entropy defined in Definition 2, then with probability at least  $1 - mP_0 - 2\exp \left[-\Omega \big(\mathcal{H}(\mathcal{X},\frac{A_gA_h}{\sqrt{m}S_{g,h}}) + \mathcal{H}(\mathcal{V},\frac{A_gA_h}{\sqrt{m}T_{g,h}})\big)\right]$  we have  $I \lesssim \frac{A_gA_h}{\sqrt{m}}\sqrt{\mathcal{H}(\mathcal{X},\frac{A_gA_h}{\sqrt{m}S_{g,h}}) + \mathcal{H}(\mathcal{V},\frac{A_gA_h}{\sqrt{m}T_{g,h}})}$ , where  $I := \sup_{\pmb{x} \in \mathcal{X}}\sup_{\pmb{v} \in \mathcal{V}}\left|\frac{1}{m}\sum_{i = 1}^{m}\big(g_{\pmb{x}}(\pmb{a}_{i})h_{\pmb{v}}(\pmb{a}_{i}) - \mathbb{E}[g_{\pmb{x}}(\pmb{a}_{i})h_{\pmb{v}}(\pmb{a}_{i})]\right|$  is the supremum of a product process.
Remark 8. We use  $\mathcal{R}_{u2}$  as an example to illustrate the advantage of Theorem 2 over Lemma 5. The key step is on bounding the centered process
$$
\mathscr{R}_{u2,c}:= \sup_{\boldsymbol {x}\in \mathcal{K}}\sup_{\boldsymbol {v}\in (\mathcal{K}_{\epsilon}^{-})^{*}}\bigl\{|\varepsilon_{i,\beta}(\boldsymbol{a}_{i}^{\top}\boldsymbol {x})||\boldsymbol{a}_{i}^{\top}\boldsymbol {v}| - \mathbb{E}[|\varepsilon_{i,\beta}(\boldsymbol{a}_{i}^{\top}\boldsymbol {x})||\boldsymbol{a}_{i}^{\top}\boldsymbol {v}|]\bigr \} .
$$
Let  $g_{\pmb{x}}(\pmb{a}_i) = |\varepsilon_{i,\beta}(\pmb{a}_i^\top \pmb{x})|$  and  $h_{\pmb{v}}(\pmb{a}_i) = |\pmb{a}_i^\top \pmb{v}|$ , then one can use Theorem 2 or Lemma 5 to bound  $\mathcal{R}_{u2,c}$ . Note that  $\| \pmb{a}_i^\top \pmb{v}\|_{\psi_2} = O(1)$  justifies the choice  $A_h = O(1)$ , and both  $\mathcal{H}(\mathcal{K},\eta)$  and  $\mathcal{H}((\mathcal{K}_{\epsilon}^{-})^{*},\eta)$  depend linearly on  $k$  but only logarithmically on  $\eta$  (Lemma 6), so Theorem 2 could bound  $\mathcal{R}_{u2,c}$  by  $\tilde{O}\big(A_g\sqrt{k / m}\big)$  that depends on  $M_g$  in a logarithmic manner. However, the bound produced by Lemma 5 depends linearly on  $M_g$ ; see term  $\frac{M_gA_h\omega(\mathcal{K})}{\sqrt{m}}$  in (A.1). From (3.6),  $M_g$  should be proportional to the Lipschitz constant of  $|\varepsilon_{i,\beta}|$ , which scales as  $\frac{1}{\beta}$  (Lemma 1). The issue is that in many cases we need to take extremely small  $\beta$  to guarantee that (2.6) holds true (e.g., we take  $\beta \asymp k / m$  in 1-bit GCS). Thus, Lemma 5 produces a worse bound compared to our Theorem 2.