SlowGuess/NeurIPS_2023_OCR huggingface dataset:
F.1 Learning Algorithm and Group Equivalence 35
F.2 Verifying the Group Equivalence of SGD and Adam 37
G CNNs vs. LCNs: Proofs of Section 5.2 39
G.1 Proof of Theorem 5.4 39
G.2 Proof of Theorem 5.2 41
H LCNs vs. FCNs: Proofs of Section 5.3 43
H.1 Proof of Theorem 5.7 43
H.2 Proof of Theorem 5.5 45
I Auxiliary Lemmas for Appendix G and H 46
I.1 Proof of Lemma G.2 47
I.2 Proof of Lemma H.3 48
Approximation and estimation. Zhou (2020b) established the first universality theorem for CNNs, which, however, requires the depth to increase as approximation accuracy improved. Later, He et al. (2022) showed a depth of  $\mathcal{O}(d)$  is sufficient when multichanneling is utilized. In this paper, we further reduce the required depth to  $\mathcal{O}(\log d)$  by combining multichanneling with downsampling. We also prove that without downsampling, CNNs require a minimum depth of  $\Omega(d)$  to achieve universality, highlighting the necessity of downsampling for deep CNNs.
Deza et al. (2020) designed some synthetic tasks to show the benefit of depth in CNNs. Okumoto and Suzuki (2022) analyzed dilated CNNs (Yu and Koltun, 2016) by considering target functions with mixed and anisotropic smoothness. They showed that dilated CNNs can capture long-range sparse signals if the dilation rate is on the order of  $\Omega(d)$ . In contrast, we show that for a similar task, vanilla CNNs can capture the long-range sparse correlations with only  $\mathcal{O}(\log d)$  depth and  $\widetilde{\mathcal{O}}(\log^2 d)$  samples if downsampling is appropriately combined with multichanneling.
Long and Sedghi (2019); Lin and Zhang (2019) established the upper bounds of the covering number and Rademacher complexity of deep CNNs, where the effect of weight sharing was incorporated. We design concrete tasks to instantiate these bounds to demonstrate the superiority of CNNs over other architectures such as FCNs. Beyond that, we also improve the covering number bound of Long and Sedghi (2019) in various aspects, including improved dependence on parameter norms (from exponential to polynomial). See Appendix C.3 for a detailed discussion.
Understanding the inductive bias. Mei et al. (2021); Bietti et al. (2021) studied the superiority of convolutional kernels by exploiting the translation invariance of convolution. Later, under similar setups, Misiakiewicz and Mei (2022) studied the benefit of using local kernels in convolutions and the effect of downsampling; Bietti (2021); Cagnetta et al. (2022) studied the inductive bias of kernel composition (mimicking the effect of depth in neural networks). However, these studies are limited to kernel methods and it is generally unclear how the results are relevant for understanding CNNs. In addition, Gunasekar et al. (2018); Jagadeesan et al. (2022); Razin et al. (2022); Jiang et al. (2021); Du et al. (2018) studied the inductive biases in linear CNNs. By contrast, we examine vanilla nonlinear CNNs and show that the nonlinearity of activation functions and the network adaptivity are critical for reaping the benefits of the CNN-specific structures.
We highlight the work Li et al. (2020), where a provable separation between CNNs and FCNs was established by exploiting the equivariance groups of the learning algorithm such as SGD and Adam. However, Li et al. (2020) neither disentangled the effects of locality and weight sharing nor studied the effects of increasing network depth, multichanneling, and downsampling. Xiao and Pennington (2022) took a similar idea to explain the inductive bias of CNNs and other architectures but the authors focused on empirical investigations without providing theoretical guarantees. It is worth noting that the idea of studying a learning algorithm through its equivariance group was first introduced in Ng (2004) for linear regression and was also applied to study other hardness problems in deep learning (Abbe and Boix-Adserà, 2022).
B Technical Background
B.1 Toolbox for Concentrations
Theorem B.1. (Hoeffding's inequality, (Vershynin, 2018, Theorem 2.2.6)) Let  $X_{1}, X_{2}, \ldots, X_{n}$  be i.i.d. random variables with mean  $\mu$  and satisfy  $X_{i} \in [a, b]$ . Then,
$$
\mathbb {P} \left\{\left| \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {i} - \mu \right| \geqslant t \right\} \leqslant 2 e ^ {- \frac {2 n t ^ {2}}{(b - a) ^ {2}}}.
$$
Definition B.2 (Sub-exponential Random Variable). For a random variable  $X$ , define
$$
\| X \| _ {\psi_ {1}} = \inf  \{t > 0: \mathbb {E} \exp (| X | / t) \leqslant 2 \} \tag {11}
$$
to be its sub-exponential norm. A random variable  $X$  that satisfies  $\| X\|_{\psi_1} < +\infty$  is called a sub-exponential random variable.
Lemma B.3. (Bernstein's inequality, (Vershynin, 2018, Corollary 2.8.3)) Let  $X_{1},\ldots ,X_{N}$  be independent, mean zero, sub-exponential random variables. Then, for every  $t\geqslant 0$  we have
$$
\mathbb {P} \left\{\left| \frac {1}{N} \sum_ {i = 1} ^ {N} X _ {i} \right| \geqslant t \right\} \leqslant 2 \exp \left[ - c _ {2} \min  \left(\frac {t ^ {2}}{K ^ {2}}, \frac {t}{K}\right) N \right] \tag {12}
$$
where  $K = \max_{i}\| X_{i}\|_{\psi_{1}}$  and  $c_{2} > 0$  is an absolute constant.
We will use covering number, packing number, and Rademacher complexity to measure the complexity of a function class.
Definition B.4 (Covering number). Let  $(T,\rho)$  be a metric space. Consider a subset  $K\subset T$  and let  $\varepsilon >0$ . A subset  $K_{\varepsilon}\subseteq K$  is called an  $\varepsilon$ -covering of  $K$  if every point in  $K$  is within distance  $\varepsilon$  of some point of  $K_{\varepsilon}$ , i.e.  $\forall x\in K,\exists x_0\in K_{\varepsilon}:\rho (x,x_0)\leqslant \varepsilon$ . The smallest possible cardinality of an  $\varepsilon$ -covering of  $K$  is called the covering number of  $K$  and is denoted by  $\mathcal{N}(K,\rho ,\varepsilon)$ .
Definition B.5 (Packing number). Let  $(T,\rho)$  be a metric space and  $K\subset T$ . Given  $\varepsilon >0$ ,  $K_{\varepsilon}\subset K$  is said to be an  $\varepsilon$ -packing of  $K$  if  $\rho (x,y) > \varepsilon$  for all distinct points  $x,y\in K_{\varepsilon}$ . The largest possible cardinality of an  $\varepsilon$ -packing of  $K$  is called the packing number of  $K$  and is denoted by  $\mathcal{P}(K,\rho ,\varepsilon)$ .
Definition B.6 (Rademacher complexity). The empirical Rademacher complexity of a function class  $\mathcal{F}$  on finite samples is defined as
$$
\widehat {\operatorname {R a d}} _ {n} (\mathcal {F}) = \mathbb {E} _ {\xi} \left[ \sup  _ {f \in \mathcal {F}} \frac {1}{n} \sum_ {i = 1} ^ {n} \xi_ {i} f \left(X _ {i}\right) \right] \tag {13}
$$
where  $\xi_1, \xi_2, \ldots, \xi_n$  are i.i.d. Rademacher random variables:  $\mathbb{P}(\xi_i = 1) = \mathbb{P}(\xi_i = -1) = \frac{1}{2}$ . Let  $\operatorname{Rad}_n(\mathcal{F}) = \mathbb{E}[\widehat{\operatorname{Rad}}(\mathcal{F})]$  be the population Rademacher complexity.
The following lemma provides an upper bound for the covering number of a general ball.
Lemma B.7. (Long and Sedghi, 2019, Lemma A.8.) Let  $p$  be a positive integer and  $\| \cdot \|$  be a norm in  $\mathbb{R}^p$ . Denote by  $B_r = \{\theta \in \mathbb{R}^p : \| \theta \| \leqslant r\}$ . Then,  $\mathcal{N}(B_r, \| \cdot \|, t) \leqslant \left(\frac{3r}{t}\right)^p$  provided  $t \leqslant r$ .
Lemma B.8 (Covering number of Lipschitz class). Let  $g: \mathcal{X} \times \Theta \mapsto \mathbb{R}$  with  $\Theta = \mathbb{R}^p$  be a general parametric model and  $\| \cdot \|$  be a norm over  $\Theta$ . Define  $\Theta_r = \{\theta : \| \theta \| \leqslant r\}$  and  $\mathcal{G}_r = \{g_\theta : \| \theta \| \leqslant r\}$ . Suppose for any  $\theta, \theta' \in \Theta_r$ , we have  $|g_\theta(z) - g_{\theta'}(z)| \leqslant B(z) \| \theta - \theta' \|$ . Given  $z_1, z_2, \ldots, z_n \in \mathcal{X}$ , let  $\hat{B}_n = \sqrt{\frac{1}{n} \sum_{i=1}^{n} B^2(z_i)}$  and  $\hat{\rho}_n(f, g) = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (f(z_i) - g(z_i))^2}$  for any  $f, g \in \mathcal{F}$ . Then for  $t \leqslant r\hat{B}_n$ , we have
$$
\mathcal {N} \left(\mathcal {G} _ {r}, \hat {\rho} _ {n}, t\right) \leqslant \left(\frac {3 \hat {B} _ {n} r}{t}\right) ^ {p}. \tag {14}
$$
$$
\hat {\rho} _ {n} \left(g _ {\theta}, g _ {\theta^ {\prime}}\right) = \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \left(g _ {\theta} \left(z _ {i}\right) - g _ {\theta^ {\prime}} \left(z _ {i}\right)\right) ^ {2}} \leqslant \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} B \left(z _ {i}\right) ^ {2}} \| \theta - \theta^ {\prime} \| = \hat {B} _ {n} \| \theta - \theta^ {\prime} \|.
$$
This implies that for any  $t > 0$ , an  $t / \hat{B}_n$ -cover of  $\Theta_r$  w.r.t.  $\| \cdot \|$  induces a  $t$ -cover of  $\mathcal{G}_r$  w.r.t.  $\hat{\rho}_n$ . Therefore,
$$
\mathcal {N} \left(\mathcal {G} _ {r}, \hat {\rho} _ {n}, t\right) \leqslant \mathcal {N} \left(\Theta_ {r}, \| \cdot \|, t / \hat {B} _ {n}\right) \leqslant \left(\frac {3 r}{t / \hat {B} _ {n}}\right) ^ {p},
$$
where the last step follows from Lemma B.7. Thus, we complete the proof.
Then we recall the uniform law of large number via Rademacher complexity, which can be found in (Wainwright, 2019, Theorem 4.10).
Lemma B.9. Assume that  $f$  ranges in  $[0, R]$  for all  $f \in \mathcal{F}$ . For any  $n \geqslant 1$ , for any  $\delta \in (0, 1)$ , w.p. at least  $1 - \delta$  over the choice of the i.i.d. training set  $S = \{X_1, \ldots, X_n\}$ , we have
$$
\sup  _ {f \in \mathcal {F}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(X _ {i}\right) - \mathbb {E} f (X) \right| \leqslant 2 \operatorname {R a d} _ {n} (\mathcal {F}) + R \sqrt {\frac {\log (4 / \delta)}{n}} \tag {15}
$$
Next, the Dudley's integral entropy bound given below shows that the Rademacher complexity can be further upper bounded by using the covering number.
Lemma B.10. Given  $n$  samples  $X_{1},\ldots ,X_{n}$ , let  $\hat{\rho}_n(f,g)\coloneqq \sqrt{\frac{1}{n}\sum_{i = 1}^{n}(f(X_i) - g(X_i))^2}$  for  $f,g\in \mathcal{F}$  and  $D = \sup_{f,g\in \mathcal{F}}\hat{\rho}_n(f,g)$  be the diameter of  $\mathcal{F}$  with respect to  $\hat{\rho}_n$ . Then,
$$
\widehat {\operatorname {R a d}} _ {n} (\mathcal {F}) \leqslant 1 2 \int_ {0} ^ {D} \sqrt {\frac {\log \mathcal {N} (\mathcal {F} , \hat {\rho} _ {n} , t)}{n}} \mathrm {d} t \tag {16}
$$
Corollary B.11. Given  $R \geqslant 1$ , let  $\mathcal{F}$  be a set of functions from  $\mathcal{X}$  to  $[0, R]$  and  $P$  be an arbitrary probability distribution over  $\mathcal{X}$ . Given  $n \in \mathbb{N}$ , let  $X_1, X_2, \ldots, X_n$  be i.i.d. samples drawn from  $P$ . Assume that there exists a  $\hat{K}_n$ , which may depend on the training data  $X_1, X_2, \ldots, X_n$ , such that  $\mathcal{N}(\mathcal{F}, \hat{\rho}_n, t) \leqslant \left( \frac{3\hat{K}_n}{t} \right)^p$  for all  $0 < t \leqslant \hat{K}_n$ . Then, for any  $\delta \in (0, 1)$ , w.p. at least  $1 - \delta$  we have
$$
\sup  _ {f \in \mathcal {F}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(X _ {i}\right) - \mathbb {E} f (X) \right| \lesssim R \sqrt {\frac {p \log \left(\mathbb {E} \left[ \hat {K} _ {n} \right] + 3\right)}{n}} + R \sqrt {\frac {\log (4 / \delta)}{n}} \tag {17}
$$
Proof. When  $2R \leqslant \hat{K}_n$ ,
$$
\begin{array}{l} \int_ {0} ^ {D} \sqrt {\frac {\log \mathcal {N} (\mathcal {F} , \hat {\rho} _ {n} , t)}{n}} d t \leqslant \sqrt {\frac {p}{n}} \int_ {0} ^ {2 R} \sqrt {\log (3 \hat {K} _ {n} / t)} d t \\ \leqslant \sqrt {\frac {p}{n}} \int_ {0} ^ {2 R} \left(\sqrt {\log \left(3 \hat {K} _ {n} + 1\right)}\right) d t + \sqrt {\frac {p}{n}} \int_ {0} ^ {2 R} \sqrt {\log (1 / t)} d t \\ \leqslant \left(2 R \sqrt {\log \left(3 \hat {K} _ {n} + 1\right)} + 4 R \log (3)\right) \sqrt {\frac {p}{n}} \tag {18} \\ \lesssim R \sqrt {\frac {p \log (\hat {K} _ {n} + 3)}{n}} \\ \end{array}
$$
When  $2R\geqslant \hat{K}_n$
$$
\begin{array}{l} \int_ {0} ^ {D} \sqrt {\frac {\log \mathcal {N} (\mathcal {F} , \hat {\rho} _ {n} , t)}{n}} d t \leqslant \int_ {0} ^ {\hat {K} _ {n}} \sqrt {\frac {\log \mathcal {N} (\mathcal {F} , \hat {\rho} _ {n} , t)}{n}} d t + \int_ {\hat {K} _ {n}} ^ {2 R} \sqrt {\frac {\log \mathcal {N} (\mathcal {F} , \hat {\rho} _ {n} , t)}{n}} d t \\ \leqslant \left(\hat {K} _ {n} \sqrt {\log \left(3 \hat {K} _ {n} + 1\right)} + 2\right) \sqrt {\frac {p}{n}} + \int_ {\hat {K} _ {n}} ^ {2 R} \sqrt {\frac {p \log (3)}{n}} d t \\ \leqslant \left(2 R \sqrt {\log \left(3 \hat {K} _ {n} + 1\right)} + 4 R \log (3)\right) \sqrt {\frac {p}{n}} \tag {19} \\ \lesssim R \sqrt {\frac {p \log (\hat {K} _ {n} + 3)}{n}} \\ \end{array}
$$
By applying Lemma B.10, we have
$$
\operatorname {R a d} _ {n} (\mathcal {F}) = \mathbb {E} [ \widehat {\operatorname {R a d}} _ {n} (\mathcal {F}) ] \lesssim R \mathbb {E} \left[ \sqrt {\frac {p \log \left(\hat {K} _ {n} + 3\right)}{n}} \right] \leqslant R \sqrt {\frac {p \log \left(\mathbb {E} \left[ \hat {K} _ {n} + 3 \right]\right)}{n}}, \tag {20}
$$
where the last step follows from the Jensen's inequality and the fact that both  $\sqrt{z}$  and  $\log (z)$  are concave with respect to  $z$ . By plugging (20) into (15), we complete the proof.
B.2 A Generalization Bound for Learning with Unbounded Noise
We first recall our setup:  $y_{i} = h^{*}(\mathbf{x}_{i}) + \xi_{i}$  with  $\mathbf{x}_i \stackrel{iid}{\sim} P$  and  $\xi_i \stackrel{iid}{\sim} \mathcal{N}(0, \sigma^2)$ . We assume  $\sup_{\mathbf{x} \in \mathcal{X}} |h^{*}(\mathbf{x})| \leqslant A$  and denote by  $h_\theta$  our parametric model.
Truncation. Due to  $\sup_{\mathbf{x} \in \mathcal{X}} |h^{*}(\mathbf{x})| \leqslant A$ , we can consider the truncated model  $\pi_A \circ h_\theta$ , which does not lose the expressivity but can make the mathematical analysis much simpler. In addition, due to the noise, the labels are unbounded but the generalization bound in Corollary B.11 requires boundedness. To handle this challenge, we define a truncated loss  $\ell_B(y, y') := \frac{1}{2}(y - y')^2 \wedge \frac{1}{2}B^2$ . The value of  $B$  will be specified later. For the truncated loss, we have
$$
\left| \partial_ {y} \ell_ {B} \left(y, y ^ {\prime}\right) \right| \leqslant B, \quad \left| \ell_ {B} \left(y, y ^ {\prime}\right) \right| \leqslant \frac {B ^ {2}}{2}. \tag {21}
$$
To facilitate our statements, we define the following truncated risks
$$
L _ {A} (\theta) = \mathbb {E} _ {\mathbf {x}, y} [ \ell (\pi_ {A} \circ h _ {\theta} (\mathbf {x}), y) ]
$$
$$
L _ {A, B} (\theta) = \mathbb {E} _ {\mathbf {x}, y} \left[ \ell_ {B} \left(\pi_ {A} \circ h _ {\theta} (\mathbf {x}), y\right) \right]
$$
$$
\hat {L} _ {A, B} (\theta) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \ell_ {B} (\pi_ {A} \circ h _ {\theta} (\mathbf {x} _ {i}), y _ {i}) \right].
$$
$$
L _ {A} (\theta) = \frac {1}{2} \| \pi_ {A} \circ h _ {\theta} - h ^ {*} \| _ {L ^ {2} (P)} ^ {2} + \frac {1}{2} \sigma^ {2}. \tag {22}
$$
Assumption 2 (Model capacity). Let  $\| \cdot \|$  be a norm over the parameter space  $\Theta$  used to control the model capacity. Define  $\mathcal{H}_J = \{x \mapsto h_\theta(x) : \| \theta \| \leqslant J\}$ . Denote by  $p_{\mathcal{H}}$  the number of parameters. Let  $\| \cdot \|_U$  be a norm satisfying that there exists a constant  $\alpha_{\mathcal{H}}$  such that
$$
\left\| \theta \right\| \leqslant \alpha_ {\mathcal {H}} \left\| \theta \right\| _ {U}, \forall \theta \in \Theta . \tag {23}
$$
We make the following assumptions.
- Covering number. Suppose that there exists a  $\hat{\gamma}_n: \mathbb{R}_{\geqslant 0} \to \mathbb{R}_{\geqslant 0}$  such that
$$
\mathcal {N} \left(\mathcal {H} _ {J}, \hat {\rho} _ {n}, t\right) \leqslant \left(\frac {3 \hat {\gamma} _ {n} (J)}{t}\right) ^ {p _ {\mathcal {H}}} f o r t \leqslant \hat {\gamma} _ {n} (J). \tag {24}
$$
Note that  $\hat{\gamma}_n(\cdot)$  may depend on the training data  $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_n$  and we let  $\gamma(J) = \mathbb{E}[\hat{\gamma}_n(J)]$
- Approximation error. We assume that there is  $\theta^{*} \in \Theta$  such that
$$
\left\| \pi_ {A} \circ h _ {\theta^ {*}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \epsilon_ {*} \text {a n d} \| \theta^ {*} \| _ {U} \leqslant M _ {*} \tag {25}
$$
Thus,  $L_{A}(\theta^{*})\leqslant (\epsilon^{*} + \sigma^{2}) / 2$
The assumption (25) essentially means that  $h^*$  can be well approximated by our model  $h_\theta$  with the parameter norm  $\| \theta^* \|_U$  well controlled.
Consider the regularized estimator given by
$$
\hat {\theta} _ {n} \in \underset {\theta} {\operatorname {a r g m i n}} \left(\hat {L} _ {A, B} (\theta) + \lambda \| \theta \| _ {U}\right), \tag {26}
$$
where  $A, B$ , and  $\lambda$  are hyperparameters to determined later.
Remark B.12. Note that penalizing the upper bound  $\| \theta \| _U$  instead of  $\| \theta \|$  is very common in practice. First, if the constant  $\alpha_{\mathcal{H}}$  is not too big, then this choice does not hurt the performance too much. Second, the upper bound  $\| \theta \| _U$  is often more interpretable and easier to compute than  $\| \theta \|$
Lemma B.13 (Properties of  $\hat{\theta}_n$ ). For any  $\delta \in (0,1)$ , let  $U_{\lambda} = \frac{1}{2\lambda}\left(\epsilon_{*} + \sigma^{2} + B^{2}\sqrt{\frac{2\log(2 / \delta)}{n}}\right) + M_{*}$ . With probability at least  $1 - \delta$ , we have
$$
\begin{array}{l} \hat {L} _ {A, B} \left(\hat {\theta} _ {n}\right) \leqslant \frac {1}{2} \left(\epsilon_ {*} + \sigma^ {2}\right) + \frac {1}{2} B ^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}} + \lambda M _ {*} \tag {27} \\ \| \hat {\theta} _ {n} \| _ {U} \leqslant U _ {\lambda}. \\ \end{array}
$$
Proof. Noting that  $\theta^{*}$  is independent of the training data, we have for any  $\delta \in (0,1)$ , it holds w.p. at least  $1 - \delta$  that
$$
\begin{array}{l} \hat {L} _ {A, B} \left(\theta^ {*}\right) \stackrel {(a)} {\leqslant} L _ {A, B} \left(\theta^ {*}\right) + \frac {1}{2} B ^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}} \leqslant L _ {A} \left(\theta^ {*}\right) + \frac {1}{2} B ^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}} \\ \stackrel {(b)} {\leqslant} \frac {1}{2} (\epsilon_ {*} + \sigma^ {2}) + \frac {1}{2} B ^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}}, \\ \end{array}
$$
where  $(a)$  and  $(b)$  follow from the Hoeffding's inequality (Theorem B.1) and the approximation assumption (25), respectively.
Since  $\hat{\theta}_n$  is one solution of the minimization problem, we have
$$
\begin{array}{l} \hat {L} _ {A, B} \left(\hat {\theta} _ {n}\right) + \lambda \left\| \hat {\theta} _ {n} \right\| _ {U} \leqslant \hat {L} _ {A, B} \left(\theta^ {*}\right) + \lambda \left\| \theta^ {*} \right\| _ {U} \\ \leqslant \frac {1}{2} (\epsilon_ {*} + \sigma^ {2}) + \frac {1}{2} B ^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}} + \lambda M _ {*}. \\ \end{array}
$$
Then, the conclusions follow trivially from the above inequality.
Proposition B.14 (Excess risk). Denote by  $\hat{\theta}_n$  the estimator defined in (26). Suppose Assumption 2 holds and  $B > 2A$ . For any  $\delta \in (0,1/2)$ , we have w.p. at least  $1 - 2\delta$  that
$$
\begin{array}{l} \left\| \pi_ {A} \circ h _ {\hat {\theta} _ {n}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} - \epsilon_ {*} \lesssim \frac {\sigma^ {3} B}{(B - 2 A) ^ {2}} e ^ {- \frac {(B - 2 A) ^ {2}}{2 \sigma^ {2}}} + \lambda M _ {*} \\ + B ^ {2} \sqrt {\frac {p _ {\mathcal {H}} \log (B \gamma (U _ {\lambda} / \alpha_ {\mathcal {H}}) + 3)}{n}} + B ^ {2} \sqrt {\frac {\log (4 / \delta)}{n}}. \\ \end{array}
$$
Remark B.15. If  $\gamma(J) \sim J^{\beta_{\mathcal{H}}}$ , then by taking  $B = C(2A + \sigma \sqrt{\log n})$  with  $C \geqslant 1$  and  $\lambda \sim 1/\sqrt{n}$ , we have
$$
\left\| \pi_ {A} \circ h _ {\hat {\theta} _ {n}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} - \epsilon_ {*} \lesssim \frac {M _ {*} + \operatorname {p o l y} (\sigma , A , \log n , \log (4 / \delta) , \log M _ {*} , \log (1 / \alpha_ {\mathcal {H}})) \sqrt {p _ {\mathcal {H}} \beta_ {\mathcal {H}}}}{\sqrt {n}}. \tag {28}
$$
Thus, we show how to set the hyperparameters  $B$ ,  $\lambda$  to yield a standard  $\tilde{\mathcal{O}}(1/\sqrt{n})$  rate. Note that on the right hand side of (28),  $\alpha_{\mathcal{H}}$ ,  $\beta_{\mathcal{H}}$  and  $p_{\mathcal{H}}$  depend on the model  $h_\theta$  and we will specify them for CNNs and LCNs separately in this paper. However, it should be stressed that Proposition B.14 is generally applicable, which might be of independent interest.
Proof. We will prove this theorem by utilizing the following decomposition:
$$
\begin{array}{l} L _ {A} (\hat {\theta} _ {n}) = L _ {A} (\hat {\theta} _ {n}) - L _ {A, B} (\hat {\theta} _ {n}) \quad \tag {Step1} \\ + L _ {A, B} \left(\hat {\theta} _ {n}\right) - \hat {L} _ {A, B} \left(\hat {\theta} _ {n}\right) \quad (\text {S t e p 2}) \\ + \hat {L} _ {A, B} (\hat {\theta} _ {n}) \quad \left(\text {S t e p 3}\right). \\ \end{array}
$$
Step 1. We first need the following tail bound of  $\mathcal{N}(0,\sigma^2)$  (Vershynin, 2018, Proposition 2.1.2): For  $X\sim \mathcal{N}(0,\sigma^2)$ , it holds that
$$
\mathbb {P} \{X \geqslant t \} \leqslant \frac {\sigma}{t} \cdot \frac {1}{\sqrt {2 \pi}} e ^ {- t ^ {2} / (2 \sigma^ {2})} \quad \forall t > 0. \tag {29}
$$
For any  $\theta$ , denote  $Z(\mathbf{x}) = \pi_A \circ h_\theta(\mathbf{x}) - h^*(\mathbf{x}) + \xi$  where  $\xi \sim \mathcal{N}(0, \sigma^2)$ . Then, we have
$$
\begin{array}{l} \left| L _ {A} (\theta) - L _ {A, B} (\theta) \right| = \frac {1}{2} \mathbb {E} _ {Z} \left[ \left(Z ^ {2} - B ^ {2}\right) \mathbf {1} _ {| Z | \geqslant B} \right] \\ = \frac {1}{2} \int_ {0} ^ {+ \infty} \mathbb {P} (Z ^ {2} - B ^ {2} \geqslant t) \mathrm {d} t \\ \leqslant \frac {1}{2} \int_ {0} ^ {+ \infty} \mathbb {P} \left(| \xi | \geqslant \sqrt {B ^ {2} + t} - 2 A\right) \mathrm {d} t \\ \stackrel {(a)} {\leqslant} \frac {1}{\sqrt {2 \pi}} \int_ {0} ^ {+ \infty} \frac {\sigma}{\sqrt {B ^ {2} + t} - 2 A} e ^ {- \frac {(\sqrt {B ^ {2} + t} - 2 A) ^ {2}}{2 \sigma^ {2}}} \mathrm {d} t \\ = \int_ {B} ^ {+ \infty} \frac {2 \sigma}{\sqrt {2 \pi}} \frac {s}{s - 2 A} e ^ {- \frac {(s - 2 A) ^ {2}}{2 \sigma^ {2}}} d s \quad (s = \sqrt {B ^ {2} + t}) \\ \leqslant \frac {2 \sigma^ {2} B}{B - 2 A} \int_ {B} ^ {+ \infty} \frac {1}{\sigma \sqrt {2 \pi}} e ^ {- \frac {(s - 2 A) ^ {2}}{2 \sigma^ {2}}} d s \\ = \frac {2 \sigma^ {2} B}{B - 2 A} \mathbb {P} (\xi \geqslant B - 2 A) \\ \stackrel {(b)} {\leqslant} \frac {2 \sigma^ {3} B}{(B - 2 A) ^ {2} \sqrt {2 \pi}} e ^ {- \frac {(B - 2 A) ^ {2}}{2 \sigma^ {2}}}, \\ \end{array}
$$
where  $(a)$  and  $(b)$  follow from (29). By taking  $\theta = \hat{\theta}_n$ , we have
$$
L _ {A} \left(\hat {\theta} _ {n}\right) - L _ {A, B} \left(\hat {\theta} _ {n}\right) \leqslant \frac {2 \sigma^ {3} B}{(B - 2 A) ^ {2} \sqrt {2 \pi}} e ^ {- \frac {(B - 2 A) ^ {2}}{2 \sigma^ {2}}}. \tag {30}
$$
Step 2. Define  $\mathcal{F}_J = \{(x,y)\mapsto \ell_B(\pi_A\circ h_\theta (x),y):\| \theta \| \leqslant J\}$ . Due to
$$
\begin{array}{l} \left| \ell_ {B} \left(\pi_ {A} \circ h _ {\theta} (\mathbf {x}), y\right) - \ell_ {B} \left(\pi_ {A} \circ h _ {\theta^ {\prime}} (\mathbf {x}), y\right) \right| \leqslant B \left| \pi_ {A} \circ h _ {\theta} (\mathbf {x}) - \pi_ {A} \circ h _ {\theta^ {\prime}} (\mathbf {x}) \right| \\ \leqslant B \left| h _ {\theta} (\mathbf {x}) - h _ {\theta^ {\prime}} (\mathbf {x}) \right|, \\ \end{array}
$$
we have  $\mathcal{N}(\mathcal{F}_J,\hat{\rho}_n,t)\leqslant \mathcal{N}(\mathcal{H}_J,\hat{\rho}_n,t / B)\leqslant (3B\hat{\gamma}_n(J) / t)^{p_{\mathcal{H}}}$  for  $t\leqslant B\hat{\gamma}_n(J)$ , where the last inequality comes from Assumption 2. For any  $J > 0$ , by Corollary B.11 and noting  $\gamma (J) = \mathbb{E}[\hat{\gamma}_n(J)]$ , It is straightforward that the following holds w.p.  $1 - \delta$
$$
\sup  _ {\| \theta \| _ {\mathcal {P}} \leqslant J} \left| L _ {A, B} (\theta) - \hat {L} _ {A, B} (\theta) \right| \lesssim B ^ {2} \sqrt {\frac {p _ {\mathcal {H}} \log (B \gamma (J) + 3)}{n}} + B ^ {2} \sqrt {\frac {\log (4 / \delta)}{n}}.
$$
By Lemma B.13, it holds  $w.p. 1 - \delta$  that  $\| \hat{\theta}_n\| _U\leqslant U_\lambda$ , which leads to  $\| \hat{\theta}_n\| \leqslant U_\lambda /\alpha_{\mathcal{H}}$ . Plugging it into the above bound gives:  $w.p. 1 - 2\delta$  it holds that