SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this work, we built a unified framework for uniform signal recovery in nonlinear generative compressed sensing. We showed that using generalized Lasso, a sample size of  $\tilde{O}(k/\epsilon^2)$  suffices to uniformly recover all  $x \in G(\mathbb{B}_2^k(r))$  up to an  $\ell_2$ -error of  $\epsilon$ . We specialized our main theorem to 1-bit GCS with/without dithering, single index model, and uniformly quantized GCS, deriving uniform guarantees that are new or exhibit some advantages over existing ones. Unlike [33], our proof is free of any non-trivial embedding property. As part of our technical contributions, we constructed the Lipschitz approximation to handle potential discontinuity in the observation model, and also developed a concentration inequality to derive tighter bound for the product processes arising in the proof, allowing us to obtain a uniform error rate faster than [17]. Possible future directions include extending our framework to handle the adversarial noise and representation error.
Acknowledgment. J. Chen was supported by a Hong Kong PhD Fellowship from the Hong Kong Research Grants Council (RGC). J. Scarlett was supported by the Singapore National Research Foundation (NRF) under grant A-0008064-00-00. M. K. Ng was partially supported by the HKRGC GRF 17201020, 17300021, CRF C7004-21GF and Joint NSFC-RGC N-HKU76921.
[1] M. Asim, M. Daniels, O. Leong, A. Ahmed, and P. Hand, "Invertible generative models for inverse problems: Mitigating representation error and dataset bias," in International Conference on Machine Learning. PMLR, 2020, pp. 399-409.
[2] A. Bora, A. Jalal, E. Price, and A. G. Dimakis, "Compressed sensing using generative models," in International Conference on Machine Learning. PMLR, 2017, pp. 537-546.
[3] P. T. Boufounos and R. G. Baraniuk, "1-bit compressive sensing," in Annual Conference on Information Sciences and Systems. IEEE, 2008, pp. 16-21.
[4] T. T. Cai and A. Zhang, "Sparse representation of a polytope and recovery of sparse signals and low-rank matrices," IEEE Transactions on Information Theory, vol. 60, no. 1, pp. 122-132, 2013.
[5] E. J. Candes and T. Tao, "Decoding by linear programming," IEEE Transactions on Information Theory, vol. 51, no. 12, pp. 4203-4215, 2005.
[6] E. J. Candes and T. Tao, "Near-optimal signal recovery from random projections: Universal encoding strategies?" IEEE Transactions on Information Theory, vol. 52, no. 12, pp. 5406-5425, 2006.
[7] J. Chen and M. K. Ng, "Uniform exact reconstruction of sparse signals and low-rank matrices from phase-only measurements," IEEE Transactions on Information Theory, vol. 69, no. 10, pp. 6739-6764, 2023.
[8] J. Chen, M. K. Ng, and D. Wang, "Quantizing heavy-tailed data in statistical estimation: (Near) Minimax rates, covariate quantization, and uniform recovery," arXiv preprint arXiv:2212.14562, 2022.
[9] J. Chen, C.-L. Wang, M. K. Ng, and D. Wang, "High dimensional statistical estimation under uniformly dithered one-bit quantization," IEEE Transactions on Information Theory, vol. 69, no. 8, pp. 5151-5187, 2023.
[10] H. Chung, J. Kim, M. T. McCann, M. L. Klasky, and J. C. Ye, "Diffusion posterior sampling for general noisy inverse problems," in International Conference on Learning Representations, 2022.
[11] G. Daras, J. Dean, A. Jalal, and A. Dimakis, "Intermediate layer optimization for inverse problems using deep generative models," in International Conference on Machine Learning. PMLR, 2021, pp. 2421-2432.
[12] M. Dhar, A. Grover, and S. Ermon, "Modeling sparse deviations for compressed sensing using generative models," in International Conference on Machine Learning. PMLR, 2018, pp. 1214-1223.
A Unified Framework for Uniform Signal Recovery in Nonlinear Generative Compressed Sensing (NeurIPS 2023)
Lemma 2. (Lemma 2.7.7, [50]). Let  $X,Y$  be sub-Gaussian, then  $XY$  is sub-exponential with  $\| XY\|_{\psi_1}\leq \| X\|_{\psi_2}\| Y\|_{\psi_2}$
Lemma 3. (Centering, [50, Exercise 2.7.10]). For some absolute constant  $C$ ,  $\| X - \mathbb{E}X\|_{\psi_1}\leq C\| X\|_{\psi_1}$ .
Lemma 4. (Bernstein's inequality, [50, Theorem 2.8.1]). Let  $X_{1}, \ldots, X_{N}$  be independent, zero-mean, sub-exponential random variables. Then for every  $t \geq 0$ , for some absolute constant  $c$  we have
$$
\mathbb {P} \left(\left| \sum_ {i = 1} ^ {N} X _ {i} \right| \geq t\right) \leq 2 \exp \left(- c \min  \left\{\frac {t ^ {2}}{\sum_ {i = 1} ^ {N} \| X _ {i} \| _ {\psi_ {1}} ^ {2}}, \frac {t}{\max  _ {1 \leq i \leq N} \| X _ {i} \| _ {\psi_ {1}}} \right\}\right)
$$
Lemma 5. ([36], statement adapted from [16, Theorem 8]). Let  $g_{\mathbf{x}} = g_{\mathbf{x}}(\mathbf{a})$  and  $h_{\mathbf{v}} = h_{\mathbf{v}}(\mathbf{a})$  be stochastic processes indexed by  $\mathbf{x} \in \mathcal{X} \subset \mathbb{R}^{p_1}$ ,  $\mathbf{v} \in \mathcal{V} \subset \mathbb{R}^{p_2}$ , both defined on some common random variable  $\mathbf{a}$ . Assume that (A1.) in Theorem 2 holds, and let  $\mathbf{a}_1, \ldots, \mathbf{a}_m$  be i.i.d. copies of  $\mathbf{a}$ . Then for any  $u \geq 1$ , with probability at least  $1 - 2\exp(-cu^2)$  we have the bound
$$
\begin{array}{l} \sup_{\substack{\boldsymbol {x}\in \mathcal{X}\\ \boldsymbol {v}\in \mathcal{V}}}\left|\frac{1}{m}\sum_{i = 1}^{m}\big(g_{\boldsymbol{x}}(\boldsymbol{a}_{i})h_{\boldsymbol{v}}(\boldsymbol{a}_{i}) - \mathbb{E}[g_{\boldsymbol{x}}(\boldsymbol{a}_{i})h_{\boldsymbol{v}}(\boldsymbol{a}_{i})]\big)\right| \\ \leq C \left(\frac {\left(M _ {g} \cdot \omega (\mathcal {X}) + u \cdot A _ {g}\right) \cdot \left(M _ {h} \cdot \omega (\mathcal {V}) + u \cdot A _ {h}\right)}{m} \right. \tag {A.1} \\ + \frac {A _ {g} \cdot M _ {h} \cdot \omega (\mathcal {V}) + A _ {h} \cdot M _ {g} \cdot \omega (\mathcal {X}) + u \cdot A _ {g} A _ {h}}{\sqrt {m}}), \\ \end{array}
$$
where  $\omega (\cdot)$  is the Gaussian width defined as  $\omega (\mathcal{X}) = \mathbb{E}\sup_{\boldsymbol {x}\in \mathcal{X}}\boldsymbol{\vartheta}^{\top}\boldsymbol{x}$  where  $\pmb {g}\sim \mathcal{N}(0,\pmb{I}_{p_1})$
The proofs of the remaining lemmas will be provided in Appendix D. (Some simple facts such as Lemma 8 were already used in prior works; while we provide the proofs for completeness.)
Lemma 6. (Metric entropy of some constraint sets). Assume  $\mathcal{K} = G(\mathbb{B}_2^k (r))$  for some  $L$ -Lipschitz generative model  $G$ . Let  $\mathcal{K}^{-} = \mathcal{K} - \mathcal{K}$ , for some  $T > 0, \epsilon \in (0,1)$  let  $\mathcal{K}_{\epsilon}^{-} := (TK^{-}) \cap (\mathbb{B}_2^n (2\epsilon))^c$ , and further define  $(\mathcal{K}_{\epsilon}^{-})^{*} = \{\frac{z}{\|\pmb{z}\|_{2}} : z \in \mathcal{K}_{\epsilon}^{-}\}$ . Then for any  $\eta \in (0,Lr)$ , we have
$$
\mathcal {H} (\mathcal {K}, \eta) \leq k \log \frac {3 L r}{\eta}, \mathcal {H} (\mathcal {K} ^ {-}, \eta) \leq 2 k \log \frac {6 L r}{\eta},
$$
$$
\mathcal {H} (\mathcal {K} _ {\epsilon} ^ {-}, \eta) \leq 2 k \log \frac {1 2 T L r}{\eta}, \mathcal {H} \big ((\mathcal {K} _ {\epsilon} ^ {-}) ^ {*}, \eta \big) \leq 2 k \log \frac {1 2 T L r}{\epsilon \eta},
$$
where  $\mathcal{H}(\cdot ,\cdot)$  is the metric entropy defined in Definition 2.
Lemma 7. (Bound the  $\ell_2$ -norm of Gaussian vector). If  $\pmb{a} \sim \mathcal{N}(0, \pmb{I}_n)$ , then  $\mathbb{P}\left(|\| \pmb{a} \|_2 - \sqrt{n} | \geq t\right) \leq 2 \exp(-C t^2)$ . In particular, setting  $t \asymp \sqrt{n}$  yields  $\mathbb{P}(\| \pmb{a} \|_2 \geq \sqrt{n}) \leq 2 \exp(-\Omega(n))$ .
In the following, Lemmas 8-11 indicate suitable choices of  $T$  in the concrete models we consider. These choices can make  $\rho(\pmb{x})$  in (2.3) sufficiently small or even zero.
Lemma 8. (Choice of  $T$  in 1-bit GCS). If  $\pmb{a} \sim \mathcal{N}(0, \pmb{I}_n)$ , then for any  $\pmb{x} \in \mathbb{S}^{n-1}$  it holds that  $\mathbb{E}[\mathrm{sign}(\pmb{a}^\top \pmb{x})\pmb{a}] = \sqrt{\frac{2}{\pi}}\pmb{x}$ .
Lemma 9. (Choice of  $T$  in 1-bit GCS with dithering). If  $\pmb{a} \sim \mathcal{N}(0, I_n)$  and  $\tau \sim \mathcal{U}[-\lambda, \lambda]$  are independent, and  $\lambda = CR\sqrt{\log m}$  with sufficiently large  $C$ , then for any  $\pmb{x} \in \mathbb{B}_2^n(R)$  it holds that  $\|\mathbb{E}[\mathrm{sign}(\pmb{a}^\top \pmb{x} + \tau)\pmb{a}] - \frac{\pmb{x}}{\lambda}\|_2 = O(m^{-9})$ .
Lemma 10. (Choice of  $T$  in SIM). If  $\pmb{a} \sim \mathcal{N}(0, I_n)$ , for some function  $f$  and any  $\pmb{x} \in \mathbb{S}^{n-1}$  it holds that  $\mathbb{E}[f(\pmb{a}^\top \pmb{x})\pmb{a}] = \mu \pmb{x}$  for  $\mu = \mathbb{E}[f(g)g]$  with  $g \sim \mathcal{N}(0,1)$ .
Lemma 11. (Choice of  $T$  in uniformly quantized GCS with dithering). Given any  $\delta >0$ , let  $\tau \sim \mathcal{U}[-\frac{\delta}{2},\frac{\delta}{2} ]$  and  $\mathcal{Q}_{\delta}(\cdot) = \delta \left(\lfloor \dot{\bar{\sigma}}\rfloor +\frac{1}{2}\right)$ . Then, for any  $a\in \mathbb{R}$ , it holds that  $\mathbb{E}[\mathcal{Q}_\delta (a + \tau)] = a$ . In particular, let  $\pmb {a}\in \mathbb{R}^n$  be a random vector satisfying  $\mathbb{E}(\pmb {a}\pmb{a}^{\top}) = \pmb{I}_n$ , and  $\tau \sim \mathcal{U}[-\frac{\delta}{2},\frac{\delta}{2} ]$  be independent of  $\pmb{a}$ , then we have  $\mathbb{E}[\mathcal{Q}_{\delta}(\pmb{a}^{\top}\pmb {x} + \tau)\pmb {a}] = \pmb{x}$ .
Lemma 12 facilitates our analysis of the uniform quantizer.
Lemma 12. Let  $f_{i}(\cdot) = \delta \left( \left\lfloor \frac{\cdot + \tau_{i}}{\delta} \right\rfloor + \frac{1}{2} \right)$  for  $\tau_{i} \sim \mathcal{U}[-\frac{\delta}{2}, \frac{\delta}{2}]$ , and  $f_{i,\beta}(\cdot)$  be defined in (3.4) for some  $0 < \beta < \frac{\delta}{2}$ . Moreover, let  $\xi_{i,\beta}(a) = f_{i,\beta}(a) - a$ ,  $\varepsilon_{i,\beta}(a) = f_{i,\beta}(a) - f_{i}(a)$ , then for any  $a \in \mathbb{R}$ ,  $|\xi_{i,\beta}(a)| \leq 2\delta$ ,  $|\varepsilon_{i,\beta}(a)| \leq \delta$  holds deterministically.
More generally, the approximation error  $|\varepsilon_{i,\beta}(a)|$  can always be bounded as follows.
Lemma 13. Suppose that  $f_{i}$  satisfies Assumption 2, and for any  $\beta \in [0,\frac{\beta_0}{2}]$  we construct  $f_{i,\beta}$  as in (3.4). Then, for any  $a\in \mathbb{R}$ , we have  $|\varepsilon_{i,\beta}(a)|\leq \left(\frac{3L_0\beta}{2} +B_0\right)\mathbb{1}(a\in \mathcal{D}_{f_i} + [-\frac{\beta}{2},\frac{\beta}{2}])$ .
B More Details of the Proof Sketch
B.1 Set-Restricted Eigenvalue Condition
Definition 4. Let  $S \subset \mathbb{R}^n$ . For parameters  $\gamma, \delta > 0$ , a matrix  $A \in \mathbb{R}^{m \times n}$  is said to satisfy  $S-REC(S, \gamma, \delta)$  if the following holds:
$$
\| \boldsymbol {A} (\boldsymbol {x} _ {1} - \boldsymbol {x} _ {2}) \| _ {2} \geq \gamma \| \boldsymbol {x} _ {1} - \boldsymbol {x} _ {2} \| _ {2} - \delta , \forall \boldsymbol {x} _ {1}, \boldsymbol {x} _ {2} \in \mathcal {S}.
$$
It was proved in [2] that  $\frac{1}{\sqrt{m}} A$  satisfies the S-REC with high probability if the entries of  $A$  are i.i.d. standard Gaussian.
Lemma 14. (Lemma 4.1 in [2]). Let  $G: \mathbb{B}_2^k(r) \to \mathbb{R}^n$  be  $L$ -Lipschitz for some  $r, L > 0$ , and define  $\mathcal{K} = G(\mathbb{B}_2^k(r))$ . For any  $\alpha \in (0,1)$ , if  $\mathbf{A} \in \mathbb{R}^{m \times n}$  has i.i.d.  $\mathcal{N}(0,1)$  entries, and  $m = \Omega\left(\frac{k}{\alpha^2}\log \frac{Lr}{\delta}\right)$ , then  $\frac{1}{\sqrt{m}}\mathbf{A}$  satisfies S-REC(  $\mathcal{K}, 1 - \alpha, \delta$  ) with probability at least  $1 - \exp(-\Omega(\alpha^2m))$ .
B.2 Lipschitz Approximation
The approximation error  $\varepsilon_{i,\beta}(\cdot)$  can be expanded as:
$$
\varepsilon_ {i, \beta} (x) = \left\{ \begin{array}{c l} 0 & , \quad \text {i f} x \notin \mathcal {D} _ {f _ {i}} + [ - \frac {\beta}{2}, \frac {\beta}{2} ] \\ f _ {i} ^ {a} (x _ {0}) - f _ {i} (x) - \frac {2 [ f _ {i} ^ {a} (x _ {0}) - f _ {i} (x _ {0} - \frac {\beta}{2}) ] (x _ {0} - x)}{\beta}, & \text {i f} x \in [ x _ {0} - \frac {\beta}{2}, x _ {0} ], (\exists x _ {0} \in \mathcal {D} _ {f _ {i}}) \\ f _ {i} ^ {a} (x _ {0}) - f _ {i} (x) + \frac {2 [ f _ {i} (x _ {0} + \frac {\beta}{2}) - f _ {i} ^ {a} (x _ {0}) ] (x - x _ {0})}{\beta}, & \text {i f} x \in [ x _ {0}, x _ {0} + \frac {\beta}{2} ], (\exists x _ {0} \in \mathcal {D} _ {f _ {i}}) \end{array} \right..
$$
Although  $|\varepsilon_{i,\beta}|$  is Lipschitz continuous,  $\varepsilon_{i,\beta}$  is not. In particular, given  $x_0\in \mathcal{D}_{f_i}$  we note that
$$
\varepsilon_ {i, \beta} ^ {-} (x _ {0}) = \lim  _ {x \to x _ {0} ^ {-}} \varepsilon_ {i, \beta} (x) = f _ {i} ^ {a} (x _ {0}) - f _ {i} ^ {-} (x _ {0}) = \frac {1}{2} \big (f _ {i} ^ {+} (x _ {0}) - f _ {i} ^ {-} (x _ {0}) \big),
$$
$$
\varepsilon_ {i, \beta} ^ {+} (x _ {0}) = \lim  _ {x \to x _ {0} ^ {+}} \varepsilon_ {i, \beta} (x) = f _ {i} ^ {a} (x _ {0}) - f _ {i} ^ {+} (x _ {0}) = \frac {1}{2} \big (f _ {i} ^ {-} (x _ {0}) - f _ {i} ^ {+} (x _ {0}) \big).
$$
Thus, it is crucial to include the absolute value for rendering the continuity.
C Proofs of Main Results
Proof. Up to rescaling, we only need to prove that  $\| \hat{\pmb{x}} - T\pmb{x}^* \|_2 \leq 3\epsilon$  holds uniformly for all  $\pmb{x}^* \in \mathcal{K}$ . We can assume  $\| \hat{\pmb{x}} - T\pmb{x}^* \|_2 \geq 2\epsilon$ ; otherwise, the desired bound is immediate.
(1) Lower bounding the left-hand side of (3.1).
We use S-REC to find a lower bound for  $\left\| \frac{\pmb{A}}{\sqrt{m}} (\hat{\pmb{x}} - T\pmb{x}^{*})\right\|_{2}^{2}$ . Specifically, we invoke Lemma 14 with  $\alpha = \frac{1}{2}$  and  $\delta = \frac{\epsilon}{2T}$ , which gives that under  $m = \Omega \left(k\log \frac{LTr}{\epsilon}\right)$ , with probability at least  $1 - \exp (-cm)$ , the following holds:
$$
\left\| \frac {\boldsymbol {A}}{\sqrt {m}} \left(\boldsymbol {x} _ {1} - \boldsymbol {x} _ {2}\right) \right\| _ {2} \geq \frac {1}{2} \| \boldsymbol {x} _ {1} - \boldsymbol {x} _ {2} \| _ {2} - \frac {\epsilon}{2 T}, \forall \boldsymbol {x} _ {1}, \boldsymbol {x} _ {2} \in \mathcal {K}. \tag {C.1}
$$
Recall that we assume  $\| \hat{\pmb{x}} - T\pmb{x}^* \|_2 \geq 2\epsilon$ ,  $T^{-1}\hat{\pmb{x}} \in \mathcal{K}$ , and  $\pmb{x}^* \in \mathcal{K}$ , so we set  $\pmb{x}_1 = \frac{\hat{\pmb{x}}}{T}$ ,  $\pmb{x}_2 = \pmb{x}^*$  in (C.1) to obtain
$$
\left\| \frac {\boldsymbol {A}}{\sqrt {m}} \Big (\frac {\hat {\boldsymbol {x}}}{T} - \boldsymbol {x} ^ {*} \Big) \right\| _ {2} \geq \frac {1}{2} \left\| \frac {\hat {\boldsymbol {x}}}{T} - \boldsymbol {x} ^ {*} \right\| _ {2} - \frac {\epsilon}{2 T} \geq \frac {1}{4 T} \left\| \hat {\boldsymbol {x}} - T \boldsymbol {x} ^ {*} \right\| _ {2}.
$$
Thus, the left-hand side of (3.1) can be lower bounded by  $\Omega \left(\|\hat{\boldsymbol{x}} - T\boldsymbol{x}^{*}\|_{2}^{2}\right)$ .
(2) Upper bounding the right-hand side of (3.1).
As analysed in (3.3) and (3.5), the right-hand side of (3.1) is bounded by  $2\|\hat{\pmb{x}} - T\pmb{x}^{*}\|_{2} \cdot (\mathcal{R}_{u1} + \mathcal{R}_{u2})$ , so all that remains is to bound  $\mathcal{R}_{u1}, \mathcal{R}_{u2}$ . In the rest of the proof, we simply write  $\sup_{\pmb{x}, \pmb{v}} := \sup_{\pmb{x} \in \mathcal{K}, \pmb{v} \in (\mathcal{K}_{\epsilon}^{-})^{*}}$  and recall the shorthand  $\xi_{i,\beta}(a) = f_{i,\beta}(a) - Ta$ . Thus, the first factor of  $\mathcal{R}_{u1}$  is given by  $\xi_{i,\beta}(a_i^\top \pmb{x})$ . By centering, we have
$$
\mathcal {R} _ {u 1} \leq \underbrace {\sup  _ {\boldsymbol {x} , \boldsymbol {v}} \frac {1}{m} \sum_ {i = 1} ^ {m} \left\{\left[ \xi_ {i , \beta} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x}\right) \right] \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {v}\right) - \mathbb {E} \left[ \left[ \xi_ {i , \beta} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x}\right) \right] \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {v}\right) \right] \right\}} _ {\mathcal {R} _ {u 1, c}} + \underbrace {\sup  _ {\boldsymbol {x} , \boldsymbol {v}} \mathbb {E} \left[ \left[ \xi_ {i , \beta} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x}\right) \right] \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {v}\right) \right]} _ {\mathcal {R} _ {u 1, e}}, \tag {C.2}
$$
$$
\mathcal {R} _ {u 2} \leq \underbrace {\sup  _ {\boldsymbol {x} , \boldsymbol {v}} \left\{\frac {1}{m} \sum_ {i = 1} ^ {m} \left| \varepsilon_ {i , \beta} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x}\right) \right| \left| \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {v} \right| - \mathbb {E} \left[ \left| \varepsilon_ {i , \beta} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x}\right) \right| \left| \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {v} \right| \right] \right\}} _ {\mathcal {R} _ {u 2, c}} + \underbrace {\sup  _ {\boldsymbol {x} , \boldsymbol {v}} \mathbb {E} \left[ \left| \varepsilon_ {i , \beta} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x}\right) \right| \left| \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {v} \right| \right]} _ {\mathcal {R} _ {u 2, e}}. \tag {C.3}
$$
We will invoke Theorem 2 multiple times to derive the required bounds.
(2.1) Bounding the centered product process  $\mathcal{R}_{u1,c}$ .
We let  $g_{\pmb{x}}(\pmb{a}_i) = \xi_{i,\beta}(\pmb{a}_i^\top \pmb{x})$  and  $h_\pmb{v}(\pmb{a}_i) = \pmb{a}_i^\top \pmb{v}$ , and write
$$
\mathscr {R} _ {u 1, c} = \sup  _ {\boldsymbol {x}, \boldsymbol {v}} \frac {1}{m} \sum_ {i = 1} ^ {m} \left\{g _ {\boldsymbol {x}} \left(\boldsymbol {a} _ {i}\right) h _ {\boldsymbol {v}} \left(\boldsymbol {a} _ {i}\right) - \mathbb {E} \left[ g _ {\boldsymbol {x}} \left(\boldsymbol {a} _ {i}\right) h _ {\boldsymbol {v}} \left(\boldsymbol {a} _ {i}\right) \right] \right\}.
$$
We verify conditions in Theorem 2 as follows:
- For any  $\pmb{x}, \pmb{x}' \in \mathcal{K}$ , because  $\xi_{i,\beta}$  is  $\left(L_0 + \frac{B_0}{\beta} + T\right)$ -Lipschitz continuous (Lemma 1), we have
$$
\begin{array}{l} \left\| g _ {\boldsymbol {x}} \left(\boldsymbol {a} _ {i}\right) - g _ {\boldsymbol {x} ^ {\prime}} \left(\boldsymbol {a} _ {i}\right) \right\| _ {\psi_ {2}} \leq \left(L _ {0} + T + \frac {B _ {0}}{\beta}\right) \left\| \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} - \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} ^ {\prime} \right\| _ {\psi_ {2}} \\ = O \left(L _ {0} + T + \frac {B _ {0}}{\beta}\right) \| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| _ {2}. \\ \end{array}
$$
- Since  $\pmb{a}_i \sim \mathcal{N}(\pmb{0}, \pmb{I}_n)$ , by Lemma 7, with probability  $1 - 2\exp(-\Omega(n))$  we have  $\| \pmb{a}_i \|_2 = O(\sqrt{n})$ . On this event, we have
$$
\begin{array}{l} \left| g _ {\boldsymbol {x}} \left(\boldsymbol {a} _ {i}\right) - g _ {\boldsymbol {x} ^ {\prime}} \left(\boldsymbol {a} _ {i}\right) \right| \leq \left(L _ {0} + T + \frac {B _ {0}}{\beta}\right) \left| \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} - \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} ^ {\prime} \right| \\ \leq \left(L _ {0} + T + \frac {B _ {0}}{\beta}\right) \| \boldsymbol {a} _ {i} \| _ {2} \| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| _ {2} \\ = O \left(\sqrt {n} \left[ L _ {0} + T + \frac {B _ {0}}{\beta} \right]\right) \| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| _ {2}. \\ \end{array}
$$
- Recall  $(\mathcal{K}_{\epsilon}^{-})^{*}$  in (3.2). Since  $(\mathcal{K}_{\epsilon}^{-})^{*} \subset \mathbb{S}^{n - 1}$ , for any  $\pmb{v}, \pmb{v}' \in (\mathcal{K}_{\epsilon}^{-})^{*}$ , we have  $\| \pmb{a}_i^\top \pmb{v} - \pmb{a}_i^\top \pmb{v}' \|_{\psi_2} = O(1) \| \pmb{v} - \pmb{v}' \|_2$ , and when  $\| \pmb{a}_i \| = O(\sqrt{n})$  we have  $|\pmb{a}_i^\top \pmb{v} - \pmb{a}_i^\top \pmb{v}'| \leq \| \pmb{a}_i \|_2 \| \pmb{v} - \pmb{v}' \|_2 = O(\sqrt{n}) \| \pmb{v} - \pmb{v}' \|_2$ . Moreover, because  $(\mathcal{K}_{\epsilon}^{-})^{*} \subset \mathbb{B}_2^n$ , for any  $\pmb{v} \in (\mathcal{K}_{\epsilon}^{-})^{*}$  we have  $\| \pmb{a}_i^\top \pmb{v} \|_{\psi_2} = O(1)$ ,  $|\pmb{a}_i^\top \pmb{v}| \leq \| \pmb{a}_i \|_2 = O(\sqrt{n})$ .
Combined with Assumption 3 and its parameters  $(A_{g}^{(1)},U_{g}^{(1)},P_{0}^{(1)})$  and  $(A_{g}^{(2)},U_{g}^{(2)},P_{0}^{(2)}),\mathcal{R}_{u1,c}$  satisfies the conditions of Theorem 2 with the following parameters
$$
M _ {g} \asymp L _ {0} + T + \frac {B _ {0}}{\beta}, A _ {g} = A _ {g} ^ {(1)}, M _ {h} \asymp 1, A _ {h} \asymp 1
$$
$$
L _ {g} \asymp \sqrt {n} \left(L _ {0} + T + \frac {B _ {0}}{\beta}\right), U _ {g} = U _ {g} ^ {(1)}, L _ {h} \asymp \sqrt {n}, U _ {h} \asymp \sqrt {n}
$$
and  $P_0 = P_0^{(1)} + 2\exp (-\Omega (n))$  . Now suppose that we have
$$
m \gtrsim \mathcal {H} \left(\mathcal {K}, \frac {A _ {g} ^ {(1)}}{\sqrt {m} n \left[ L _ {0} + T + \frac {B _ {0}}{\beta} \right]}\right) + \mathcal {H} \left(\left(\mathcal {K} _ {\epsilon} ^ {-}\right) ^ {*}, \frac {A _ {g} ^ {(1)}}{\sqrt {m} \left(\sqrt {n} U _ {g} ^ {(1)} + A _ {g} ^ {(1)}\right)}\right), \tag {C.4}
$$
and note that by using Lemma 6, (C.4) can be guaranteed by
$$
m \gtrsim k \log \left(\frac {L r \sqrt {m}}{A _ {g} ^ {(1)}} \left[ n \left(L _ {0} + T + \frac {B _ {0}}{\beta}\right) + \frac {T \left(\sqrt {n} U _ {g} ^ {(1)} + A _ {g} ^ {(1)}\right)}{\epsilon} \right]\right). \tag {C.5}
$$
Then Theorem 2 yields that the following bound holds with probability at least  $1 - mP_0^{(1)} - C\exp (-\Omega (k)) - m\exp (-\Omega (n))$  ..
$$
\begin{array}{l} \left| \mathscr {R} _ {u 1, c} \right| \lesssim \frac {A _ {g} ^ {1}}{\sqrt {m}} \sqrt {\mathscr {H} \left(\mathcal {K} , \frac {A _ {g} ^ {(1)}}{\sqrt {m} n \left[ L _ {0} + T + \frac {B _ {0}}{\beta} \right]}\right) + \mathscr {H} \left(\left(\mathcal {K} _ {\epsilon} ^ {-}\right) ^ {*}, \frac {A _ {g} ^ {(1)}}{\sqrt {m} (\sqrt {n} U _ {g} ^ {(1)} + A _ {g} ^ {(1)})}\right)} \tag {C.6} \\ \lesssim A _ {g} ^ {(1)} \sqrt {\frac {k}{m} \log \left(\frac {L r \sqrt {m}}{A _ {g} ^ {(1)}} \Big [ n (L _ {0} + T + \frac {B _ {0}}{\beta}) + \frac {T (\sqrt {n} U _ {g} ^ {(1)} + A _ {g} ^ {(1)})}{\epsilon} \Big ]\right)}. \\ \end{array}
$$
(2.2) Bounding the centered product process  $\mathcal{R}_{u2,c}$ .
We let  $g_{\pmb{x}}(\pmb{a}_i) = |\varepsilon_{i,\beta}(\pmb{a}_i^\top \pmb{x})|$  and  $h_{\pmb{v}}(\pmb{a}_i) = |\pmb{a}_i^\top \pmb{v}|$ , and write
$$
\mathcal {R} _ {u 2, c} = \sup  _ {\boldsymbol {x}, \boldsymbol {v}} \frac {1}{m} \sum_ {i = 1} ^ {m} \left\{g _ {\boldsymbol {x}} (\boldsymbol {a} _ {i}) h _ {\boldsymbol {v}} (\boldsymbol {a} _ {i}) - \mathbb {E} \left[ g _ {\boldsymbol {x}} (\boldsymbol {a} _ {i}) h _ {\boldsymbol {v}} (\boldsymbol {a} _ {i}) \right] \right\}.
$$
We verify the conditions in Theorem 2 as follows:
- For any  $\pmb{x}, \pmb{x}' \in \mathcal{K}$ , because  $|\varepsilon_{i,\beta}|$  is  $(2L_0 + \frac{B_0}{\beta})$ -Lipschitz continuous (Lemma 1), we have
$$
\begin{array}{l} \left\| g _ {\boldsymbol {x}} (\boldsymbol {a} _ {i}) - g _ {\boldsymbol {x} ^ {\prime}} (\boldsymbol {a} _ {i}) \right\| _ {\psi_ {2}} \leq \left(2 L _ {0} + \frac {B _ {0}}{\beta}\right) \left\| \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} - \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} ^ {\prime} \right\| _ {\psi_ {2}} \\ = O \left(L _ {0} + \frac {B _ {0}}{\beta}\right) \| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| _ {2}. \\ \end{array}
$$
- By Lemma 7, with probability at least  $1 - 2\exp(-\Omega(n))$  we have  $\| \pmb{a}_i \|_2 = O(\sqrt{n})$ . On this event, we have
$$
\begin{array}{l} \left| g _ {\boldsymbol {x}} \left(\boldsymbol {a} _ {i}\right) - g _ {\boldsymbol {x} ^ {\prime}} \left(\boldsymbol {a} _ {i}\right) \right| \leq \left(2 L _ {0} + \frac {B _ {0}}{\beta}\right) \left| \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} - \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {x} ^ {\prime} \right| \\ \leq \left(2 L _ {0} + \frac {B _ {0}}{\beta}\right) \| \boldsymbol {a} _ {i} \| _ {2} \| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| _ {2} \\ = O \left(\sqrt {n} \left[ L _ {0} + \frac {B _ {0}}{\beta} \right]\right) \| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| _ {2}. \\ \end{array}
$$