SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The first line is again by applying the claim to  $\mathcal{F}_Q(R)$ . In the last line, we used two facts:
1.  $\sigma$  is ReLU, so  $|a|\sigma (b)\leq |ab|\leq \sqrt{Q}\cdot \sqrt{Q} R = QR$  and
2. the observation that
$$
\mathbb {E} _ {\epsilon} \left| \sum_ {i} ^ {n} \epsilon_ {i} \right| \leq \sqrt {\mathbb {E} _ {\epsilon} \left| \sum_ {i} ^ {n} \epsilon_ {i} \right| ^ {2}} = \sqrt {\sum_ {i , j = 1} ^ {n} \mathbb {E} _ {\varepsilon} [ \varepsilon_ {i} \varepsilon_ {j} ]} = \sqrt {\sum_ {i = 1} ^ {n} \mathbb {E} _ {\varepsilon} [ \varepsilon_ {i} ^ {2} ]} = \sqrt {n}
$$
since  $\mathbb{E}[\varepsilon_i] = 0$ ,  $\varepsilon_{i}$  and  $\varepsilon_{j}$  are independent if  $i\neq j$  and  $\varepsilon_i^2\equiv 1$ .
Step 3. In this step, we prove that
$$
\frac {1}{n} \mathbb {E} _ {\varepsilon} \left[ \sup _ {| a | = \| w \| \leq \sqrt {Q}, | b | \leq \sqrt {Q} R} \left| \sum_ {i = 1} ^ {n} \varepsilon_ {i} a \sigma (w \cdot x _ {i} + b) \right| \right] \leq \frac {3 \sqrt {2} Q R}{\sqrt {n}}
$$
To this end, we modify the data points as  $\tilde{x}_i = (x_i,R)$  and the parameters as  $\tilde{w} = (w^T,\frac{b}{R})$ . Then, observe that  $w\cdot x_{i} + b = \tilde{w}\cdot \tilde{x}_{i},\| \tilde{x}_{i}\| = \sqrt{\|x_{i}\|^{2} + R^{2}}\leq \sqrt{2} R$ , and  $a^2 +\| \tilde{w}\| ^2 = a^2 +\| w\| ^2 +(\frac{b}{R})^2\leq 3Q$ . Therefore, we can write the above by the following:
$$
\begin{array}{l} \frac {1}{n} \mathbb {E} _ {\varepsilon} \left[ \sup  _ {a ^ {2} = \| w \| ^ {2} \leq Q, | b | \leq Q} \left| \sum_ {i = 1} ^ {n} \varepsilon_ {i} a \sigma (w \cdot x _ {i} + b) \right| \right] \leq \frac {1}{n} \mathbb {E} _ {\varepsilon} \left[ \sup  _ {a ^ {2} + \| \tilde {w} \| ^ {2} \leq 3 Q} \left| \sum_ {i = 1} ^ {n} \varepsilon_ {i} a \sigma (\tilde {w} \cdot \tilde {x} _ {i}) \right| \right] \\ = \frac {1}{n} \mathbb {E} _ {\varepsilon} \left[ \sup  _ {a ^ {2} + \| \tilde {w} \| ^ {2} \leq 3 Q} | a | \| \tilde {w} \| \left| \sum_ {i = 1} ^ {n} \varepsilon_ {i} \sigma \left(\frac {\tilde {w}}{\| \tilde {w} \|} \cdot \tilde {x} _ {i}\right) \right| \right] \\ \leq \frac {1}{n} \mathbb {E} _ {\varepsilon} \left[ \sup  _ {a ^ {2} + \| \tilde {w} \| ^ {2} \leq 3 Q, \| u \| _ {2} \leq 1} \frac {a ^ {2} + \| \tilde {w} \| ^ {2}}{2} \left| \sum_ {i = 1} ^ {n} \varepsilon_ {i} \sigma (u \cdot \tilde {x} _ {i}) \right| \right] \\ \leq \frac {3 Q}{2 n} \mathbb {E} _ {\varepsilon} \left[ \sup  _ {\| u \| _ {2} \leq 1} \left| \sum_ {i = 1} ^ {n} \varepsilon_ {i} \sigma (u \cdot \tilde {x} _ {i}) \right| \right] \\ \leq \frac {3 Q}{n} \mathbb {E} _ {\varepsilon} \left[ \sup  _ {\| u \| _ {2} \leq 1} \sum_ {i = 1} ^ {n} \varepsilon_ {i} \sigma (u \cdot \tilde {x} _ {i}) \right] \\ = 3 Q \widehat {\operatorname {R a d}} (\sigma \circ \mathcal {H} _ {2}, \tilde {S} _ {n}) \leq 3 Q \widehat {\operatorname {R a d}} (\mathcal {H} _ {2}, \tilde {S} _ {n}) \leq \frac {3 Q}{\sqrt {n}} \max  _ {i} \| \tilde {x} _ {i} \| _ {2} \leq \frac {3 \sqrt {2} Q R}{\sqrt {n}} \\ \end{array}
$$
Here,  $\mathcal{H}_2\coloneqq \{u\in \mathbb{R}^d\mid \| u\| _2\leq 1\}$  and  $\tilde{S}_n = \{\tilde{x}_1,\dots ,\tilde{x}_n\}$ . When removing the absolute value, we used that the sum is always non-negative and that it is symmetric when replacing  $\varepsilon$  by  $-\varepsilon$ . When removing  $\sigma$ , we make use of the Contraction Lemma for Rademacher complexity [Shalev-Shwartz and Ben-David, 2014, Lemma 26.9]. Finally, for  $\mathrm{Rad}(\mathcal{H}_2,\tilde{S}_n)$  we used the expression for the Rademacher complexity of the class of linear functions on Hilbert space. [Shalev-Shwartz and Ben-David, 2014, Lemma 26.10]. This concludes Step 3.
Step 4. In this step, we finally consider sets  $S$  which are sampled from the product measure  $\mu^n$ , i.e. sets where  $x_1, \ldots, x_n$  are independent data samples with law  $\mu$ . From steps 1 - 3, we know that
$$
\operatorname {R a d} (\mathcal {F} _ {Q}) = \mathbb {E} _ {S _ {n} \sim \mu^ {n}} \left[ \widehat {\operatorname {R a d}} (\mathcal {F}, S _ {n}) \right] \leq \frac {(1 + 3 \sqrt {2}) Q}{\sqrt {n}} \mathbb {E} _ {(x _ {1}, \ldots , x _ {n}) \sim \mu^ {n}} \big [ \max  _ {1 \leq i \leq n} \| x _ {i} \| \big ].
$$
We bound  $\mathbb{E}_{(x_1,\ldots ,x_n)\sim \mu^n}\big[\max_{1\leq i\leq n}\| x_i\|\big]$  by Lemma H.1 to obtain
$$
\operatorname {R a d} (\mathcal {F} _ {Q}) \leq (1 + 3 \sqrt {2}) Q \left(\frac {\mathbb {E} _ {x \sim \mu} [ \| x \| ]}{\sqrt {n}} + \sigma \sqrt {2 \frac {\log n}{n}}\right)
$$
A similar result follows immediately for the more general function class
$$
\mathcal {F} _ {A, Q} := \{f \in \mathcal {B}: [ f ] _ {\mathcal {B}} \leq Q, | f (0) | \leq A \}. \tag {5}
$$
Corollary D.3. Under the same conditions as Lemma D.2, we have
$$
\operatorname {R a d} (\mathcal {F} _ {A, Q}) \leq (1 + 3 \sqrt {2}) Q \left(\frac {\mathbb {E} _ {x \sim \mu} [ \| x \| ]}{\sqrt {n}} + \sigma \sqrt {2 \frac {\log n}{n}}\right) + \frac {A}{\sqrt {n}}
$$
Proof. We note that  $f \in \mathcal{F}_{A,Q}$  if and only if  $f = \tilde{f} + \alpha$  with  $f \in \mathcal{F}_Q$  and  $|\alpha| \leq A$ . Hence, for any fixed dataset  $S$ , we have
$$
\begin{array}{l} n \widehat {\mathrm {R a d}} _ {n} (\mathcal {F} _ {A, Q}) = \mathbb {E} _ {\varepsilon} \left[ \sup _ {f \in \mathcal {F} _ {A, Q}} \sum_ {i = 1} ^ {n} \varepsilon_ {i} f (x _ {i}) \right] = \mathbb {E} _ {\varepsilon} \left[ \sup _ {f \in \mathcal {F} _ {Q}, | \alpha | \leq A} \sum_ {i = 1} ^ {n} \varepsilon_ {i} \big (\alpha + f (x _ {i}) \big) \right] \\ \leq \mathbb {E} _ {\varepsilon} \left[ \sup _ {f \in \mathcal {F} _ {Q}} \sum_ {i = 1} ^ {n} \varepsilon_ {i} f (x _ {i}) \right] + \mathbb {E} _ {\varepsilon} \left[ \sup _ {| \alpha | \leq A} \sum_ {i = 1} ^ {n} \varepsilon_ {i} \alpha \right] \leq \widehat {\mathrm {R a d}} _ {n} (\mathcal {F} _ {Q}) + \frac {A}{\sqrt {n}} \\ \end{array}
$$
by the argument of Step 2 in the proof of Lemma D.2.
A bound on the Rademacher complexity, together with the sub-Gaussian property of the distribution  $\mu$ , allows us to control the 'generalization gap' in homogeneous Barron spaces.
Corollary D.4. Assume that  $\mu$  is a  $\sigma^2$ -sub-Gaussian distribution on  $\mathbb{R}^d$ . Let  $(X_1, \ldots, X_n)$  be iid random variables with law  $\mu$  and  $f^*$  a  $\mu$ -measurable function such that
$$
\left| f ^ {*} (x) - f ^ {*} (0) \right| \leq B _ {1} + B _ {2} \| x \|
$$
$\mu$  -almost everywhere.Let
$$
\widehat {\mathcal {R}} _ {n} (f) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left| f (X _ {i}) - f ^ {*} (X _ {i}) \right| ^ {2}, \qquad \mathcal {R} (f) = \mathbb {E} _ {x \sim \mu} \left[ | f (x) - f ^ {*} (x) | ^ {2} \right].
$$
Then with probability at least  $1 - 2\delta$  over the random draw of  $X_{1},\ldots ,X_{n}$ , the bound
$$
\sup  _ {f - f ^ {*} (0) \in \mathcal {F} _ {A, Q}} \left(\mathcal {R} (f) - \widehat {\mathcal {R}} _ {n} (f)\right) \leq C ^ {*} \left(\left(Q + B _ {2}\right) \left(\mathbb {E} _ {x \sim \mu} \| x \| + \sigma^ {2} + 1\right) + A + B _ {1}\right) ^ {2} \frac {\log (n / \delta)}{\sqrt {n}}
$$
holds for a constant  $C^* > 0$  which does not depend on  $\delta, Q, d, \mu$  or  $n$ .
Proof. Step 1. From Lemma H.2, with probability at least  $1 - \delta$  we have
$$
\max _ {1 \leq i \leq n} \| X _ {i} \| \leq \mathbb {E} _ {x \sim \mu} \left[ \| x \| \right] + \sigma \sqrt {2 \log (n / \delta)}.
$$
We denote  $R_{n} \coloneqq \mathbb{E}_{x\sim \mu}\big[\| x\|\big] + \sigma \sqrt{2\log(n / \delta)}$  for simplicity.
Step 2. Consider the modified loss function
$$
\ell_ {\xi} (f) = \min  \left\{f ^ {2}, \xi^ {2} \right\},
$$
which is bounded by  $\xi^2$  and satisfies  $|\partial_f\ell_\xi |\leq 2R$ , i.e.  $\ell_{\xi}$  is  $2\xi$ -Lipschitz continuous. We thus observe that, with probability at least  $1 - \delta$  over the choice of random set  $S = \{x_{1},\ldots ,x_{n}\}$ , we have
$$
\mathbb {E} _ {x \sim \mu} \big [ \ell_ {\xi} \big (f (x) - f ^ {*} (x) \big) \big ] - \frac {1}{n} \sum_ {i = 1} ^ {n} \ell_ {\xi} \big (f (x _ {i}) - f ^ {*} (x _ {i}) \big) \leq 4 \xi \mathbb {E} \big [ \widehat {\mathrm {R a d}} (\mathcal {F} _ {Q}, S _ {n}) \big ] + \xi^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}}
$$
by [Shalev-Shwartz and Ben-David, 2014, Theorem 26.5] and the Contraction Lemma for Rademacher complexities, [Shalev-Shwartz and Ben-David, 2014, Lemma 26.9]. In particular
$$
\begin{array}{l} \mathbb {E} _ {x \sim \mu} \big [ \ell_ {\xi} \big (f (x) - f ^ {*} (x) \big) \big ] \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \ell_ {\xi} \big (f (x _ {i}) - f ^ {*} (x _ {i}) \big) + \xi^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}} \\ + 4 (1 + 3 \sqrt {2}) Q \xi \left(\frac {\mathbb {E} _ {x \sim \mu} \| x \|}{\sqrt {n}} + \sigma \sqrt {\frac {2 \log n}{n}}\right) + \frac {A \xi}{\sqrt {n}}. \\ \end{array}
$$
Step 3. By the union bound, with probability at least  $1 - 2\delta$ , both the norm bound of Step 1 and the generalization bound of Step 2 hold. In the following, we assume that both bounds hold. Note that
$$
\left| f (x) - f ^ {*} (x) \right| \leq \left| f (x) - f ^ {*} (0) \right| + \left| f ^ {*} (x) - f ^ {*} (0) \right| \leq \left(A + B _ {1}\right) + \left(Q + B _ {2}\right) \| x \|.
$$
In particular, if  $\xi \geq (A + B_1) + (Q + B_2)R$ , then  $|f(x) - f^*(x)| \leq \xi$  on  $B_R(0)$ , so  $\ell_{\xi}(f(x) - f^*(x)) \equiv \ell(f(x) - f^*(x))$ . Applying the generalization bound with  $R_n = \mathbb{E}_{x \sim \mu} \|x\| + \sigma \sqrt{2 \log(n/\delta)}$  and  $\xi_n = (Q + B_2)R_n + (A + B_1)$ , we find that  $\ell_{\xi_n}(f(x_i) - f^*(x_i)) = \ell(f(x_i) - f^*(x_i))$  for all  $i$  by assumption and thus, with probability at least  $1 - 2\delta$ , we have
$$
\begin{array}{l} \mathbb {E} _ {x \sim \mu} \big [ \ell_ {\xi} \big (f (x) - f ^ {*} (x) \big) \big ] \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \big (f (x _ {i}) - f ^ {*} (x _ {i}) \big) ^ {2} + \big ((Q + B _ {2}) R _ {n} + A + B _ {1} \big) ^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}} \\ + 4 (1 + 3 \sqrt {2}) Q \big ((Q + B _ {2}) R _ {n} + A + B _ {1} \big) \left(\frac {\mathbb {E} _ {x \sim \mu} \| x \|}{\sqrt {n}} + \sigma \sqrt {\frac {2 \log n}{n}}\right). \\ \end{array}
$$
Step 4. Finally, we bound the population risk with the true loss function rather than  $\ell_{\xi}$ . Thus we find that for  $B_R \coloneqq B_R(0)$  we have
$$
\begin{array}{l} \mathbb {E} _ {x \sim \mu} \left[ (f (x) - f ^ {*} (x)) ^ {2} \right] = \mathbb {E} _ {x \sim \mu} \left[ (f (x) - f ^ {*} (x)) ^ {2} 1 _ {B _ {R}} \right] + \mathbb {E} _ {x \sim \mu} \left[ (f (x) - f ^ {*} (x)) ^ {2} 1 _ {\mathbb {R} ^ {d} \setminus B _ {R}} \right] \\ \leq \mathbb {E} _ {x \sim \mu} \left[ \ell_ {\xi} \big (f (x) - f ^ {*} (x) \big) \right] + \mathbb {E} _ {x \sim \mu} \left[ (A + B _ {1} + (Q + B _ {2}) \| x \|) ^ {2} 1 _ {\mathbb {R} ^ {d} \setminus B _ {R}} \right] \\ \end{array}
$$
for  $R \geq 3$ . From Lemma H.3 with  $R_{n} = \mathbb{E}\|x\| + \sigma \sqrt{2\log(n/\delta)}$ , we have
$$
\mathbb {E} _ {x \sim \mu} \big [ \| x \| ^ {2} 1 _ {B _ {R _ {n}} (0) ^ {c}} (x) \big ] \leq \sqrt {2 \pi} \exp \left(- \frac {\log (n / \delta)}{2}\right) \left((\mathbb {E} \| x \|) ^ {2} + 2 \sigma^ {2}\right) = \sqrt {2 \pi} \frac {(\mathbb {E} \| x \|) ^ {2} + 2 \sigma^ {2}}{\sqrt {n / \delta}}.
$$
Remark D.5. In particular, Corollary D.4 applies if the target function  $f^{*}$  is Lipschitz-continuous with  $B_{1} = 0$  and  $B_{2} = [f^{*}]_{Lip} \leq [f^{*}]_{\mathcal{B}}$ . However, continuity is not necessary, and even noisy labels would be admissible. We do not pursue this generality here.
E Proofs of the convergence theorems
In this appendix, we present the proofs of Theorems 2.1 and 3.3. In these, we combine the upper bound of the Rademacher complexity of the unit ball in homogeneous Barron space in form of the generalization bound of Corollary D.4 with a  $\Gamma$ -convergence argument (to guarantee the convergence of minimizers to minimizers). The main ingredients in the proof of  $\Gamma$ -convergence are
- the compact embedding theorem for homogeneous Barron space (to guarantee that every subsequence of  $f_{n}$  has a convergent subsequence) and
- the direct approximation theorem for homogeneous Barron space (to obtain a bound on the lowest achievable energy using a neural network with  $m$  neurons).
We first present convergence proofs in  $L^p(\mu)$  in Section E.1, followed by proofs of  $\Gamma$ -convergence in Section E.2. We combine the arguments to prove the statements from the main body of the document in Section E.3.
E.1 Convergence in  $L^p(\mu)$
We start by establishing convergence in  $L^2(\mu)$  at an explicit convergence rate. Then, using this  $L^2(\mu)$  convergence we will extend this result to general  $L^p(\mu)$ .
We introduce one of our main theorem of this section, which gives us an explicit bound of  $L^2(\mu)$ -loss. For convenience, we denote  $\theta \coloneqq (a, W, b)$  for the rest of the section.
Theorem E.1 ( $L^2$ -convergence). Let  $\hat{\theta} \in \operatorname{argmin}_{\theta} \widehat{\mathcal{R}}_{n,m,\lambda}(\theta)$ . If  $\delta \geq e^{-n}$ , and  $f^{*} \in \mathcal{F}_{Q^{*}}$ , then with probability at least  $1 - 4\delta$  over the choice of random points  $x_{1}, \ldots, x_{n}$  we have
$$
\mathcal {R} (f _ {\hat {\theta}}) \leq C \left(\frac {(Q ^ {*}) ^ {2}}{m} \left(\mathbb {E} \big [ \| x \| ^ {2} \big ]\right) + \lambda Q ^ {*} + Q ^ {*} \left(\mathbb {E} \| x \| + \sigma^ {2} + [ f ^ {*} ] _ {\mathcal {B}}\right) \frac {\log (n / \delta)}{\sqrt {n}}\right)
$$
up to higher order terms in the small quantities  $(\lambda m)^{-1},m^{-1},n^{-1 / 2}\log n$
Proof. Outline. We use Theorem C.5 for  $L^2(\mu_n)$  with  $f = f^*$  to obtain a function for which  $\widehat{\mathcal{R}}_{n,m,\lambda}$  is low. The empirical risk minimizer (ERM) has even lower risk. The weight decay penalty additionally provides a norm-bound in homogeneous Barron space for the ERM, and we use Corollary D.4 to control the generalization gap.
Step 1. Due to Theorem C.5, there exists a  $\tilde{\theta} := (\tilde{a},\tilde{w},\tilde{b})\in \mathbb{R}^m\times \mathbb{R}^{m\times d}\times \mathbb{R}^{m + 1}$  such that
$$
R _ {W D} (\tilde {\theta}) \leq [ f ^ {*} ] _ {\mathcal {B}} \tag {6}
$$
$$
\widehat {\mathcal {R}} _ {n} (f _ {\widetilde {\theta}}) = [ f _ {\widetilde {\theta}} - f ^ {*} ] _ {L ^ {2} (\mu_ {n})} ^ {2} \leq \frac {4 [ f ^ {*} ] ^ {2}}{m} \sup _ {\| w \| = 1} \int_ {\mathbb {R} ^ {d}} | w ^ {T} x | ^ {2} d \mu_ {n} \leq \frac {4 [ f ^ {*} ] ^ {2}}{m} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \| x _ {i} \| ^ {2}\right).
$$
We will always consider  $\delta$  such that  $\log (1 / \delta) \leq n$ . In this regime, plugging-in the bound on the second moments of  $\mu_{n}$  from Lemma H.4 gives the corresponding bound
$$
\widehat {\mathcal {R}} _ {n} \left(f _ {\tilde {\theta}}\right) \leq \frac {4 \left[ f ^ {*} \right] ^ {2}}{m} \left(\mathbb {E} \left[ \| x \| ^ {2} \right] + 8 \sigma^ {2} \sqrt {\frac {\log (1 / \delta)}{n}}\right) \tag {7}
$$
with probability  $1 - \delta$ . We will assume that this estimate is valid for the remainder of the proof. In particular, since  $\hat{\theta}$  minimizes  $\hat{\mathcal{R}}_{n,m,\lambda}$ , we find that
$$
\widehat {\mathcal {R}} _ {n, m, \lambda} (\hat {\theta}) \leq \widehat {\mathcal {R}} _ {n, m, \lambda} (\tilde {\theta}) \leq \frac {4 [ f ^ {*} ] ^ {2}}{m} \left(\mathbb {E} [ \| x \| ^ {2} ] + 8 \sigma^ {2} \sqrt {\frac {\log (1 / \delta)}{n}}\right) + \lambda [ f ^ {*} ] _ {\mathcal {B}}. \tag {8}
$$
Step 2. Next, we bound  $[f_{\hat{\theta}}]_{\mathcal{B}}$  and  $|f_{\hat{\theta}}(0) - f^{*}(0)|$  ( $Q$  and  $A$  in Corollary D.4). We first bound the Barron semi-norm by
$$
[ f _ {\hat {\theta}} ] _ {\mathcal {B}} \leq \frac {1}{\lambda} \widehat {\mathcal {R}} _ {n, m, \lambda} (\hat {\theta}) \leq \frac {1}{\lambda} \widehat {\mathcal {R}} _ {n, m, \lambda} (\tilde {\theta}).
$$
Moving on to bounding  $A$ , we find from the empirical risk bound
$$
\min _ {1 \leq i \leq n} | f - f ^ {*} | (x _ {i}) = \sqrt {\min _ {1 \leq i \leq n} | f - f ^ {*} | ^ {2} (x _ {i})} \leq \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} | f - f ^ {*} | (x _ {i})} \leq \sqrt {\widehat {\mathcal {R}} _ {n} (f)}
$$
$$
\min _ {1 \leq i \leq n} | f _ {\tilde {\theta}} - f ^ {*} | (x _ {i}) \leq \sqrt {\widehat {\mathcal {R}} _ {n , m , \lambda} (\tilde {\theta})}.
$$
$$
\max _ {1 \leq i \leq n} \| x _ {i} \| \leq \mathbb {E} _ {x \sim \mu} \big [ \| x \| \big ] + \sigma \sqrt {2 \log (n / \delta)}.
$$
by Lemma H.2. Again, we assume that the estimate holds in the following. Hence, the index  $i$  for which the minimum is attained in (8) satisfies the bound
$$
\left\| x _ {i} \right\| \leq \mathbb {E} _ {x \sim \mu} \left[ \| x \| \right] + \sigma \sqrt {2 \log (n / \delta)}.
$$
Combining the bounds on  $|f_{\hat{\theta}}(x_i) - f^* (x_i)|$  and the Lipschitz constants of  $f_{\hat{\theta}},f^{*}$ , we find that
$$
\begin{array}{l} \left| f _ {\hat {\theta}} - f ^ {*} \right| (0) \leq \left| f _ {\hat {\theta}} - f ^ {*} \right| \left(x _ {i}\right) + \left(\left[ f _ {\hat {\theta}} \right] _ {\mathcal {B}} + \left[ f ^ {*} \right] _ {\mathcal {B}}\right) \| x _ {i} \| \\ \leq \sqrt {\widehat {\mathcal {R}} _ {n , m , \lambda} (\tilde {\theta})} + \left([ f ^ {*} ] _ {\mathcal {B}} + \frac {1}{\lambda} \widehat {\mathcal {R}} _ {n, m, \lambda} (\tilde {\theta})\right) \left(\mathbb {E} _ {x \sim \mu} \left[ \| x \| \right] + \sigma \sqrt {2 \log (n / \delta)}\right). \\ \end{array}
$$
Step 3. Comparing  $\hat{\theta}$  to  $\tilde{\theta}$ , we observe that
$$
\begin{array}{l} \mathcal {R} (f _ {\hat {\theta}}) = \widehat {\mathcal {R}} _ {n} (f _ {\hat {\theta}}) + \mathcal {R} (f _ {\hat {\theta}}) - \widehat {\mathcal {R}} _ {n} (f _ {\hat {\theta}}) \\ \leq \widehat {\mathcal {R}} _ {n, m, \lambda} (\widehat {\theta}) + \mathcal {R} (f _ {\widehat {\theta}}) - \widehat {\mathcal {R}} (f _ {\widehat {\theta}}) \\ \leq \widehat {\mathcal {R}} _ {n, m, \lambda} (\tilde {\theta}) + \mathcal {R} (f _ {\hat {\theta}}) - \widehat {\mathcal {R}} _ {n} (f _ {\hat {\theta}}) \\ \end{array}
$$
where we used the fact that  $\hat{\theta}$  is a minimizer of  $\widehat{\mathcal{R}}_{n,m,\lambda}(\theta)$  and (6). In the following, we use the bound on  $[f_{\hat{\theta}}]_{\mathcal{B}}$  to control the generalization gap.
Step 4. Recall that  $[f_{\hat{\theta}}]_{\mathcal{B}} \leq [f^{*}]_{\mathcal{B}} + \frac{1}{\lambda} \widehat{\mathcal{R}}_{n,m,\lambda}(f_{\hat{\theta}}) = [f^{*}]_{\mathcal{B}} + O((\lambda m)^{-1})$ . Thus, with probability at least  $1 - 2\delta$ , we obtain the bound
$$
\big (\mathcal {R} - \widehat {\mathcal {R}} _ {n} \big) (f _ {\hat {\theta}}) \leq \left([ f ^ {*} ] _ {\mathcal {B}} + \frac {1}{\lambda} \widehat {\mathcal {R}} _ {n, m, \lambda} (\hat {\theta})\right) \left(\mathbb {E} \| x \| + \sigma^ {2} + [ f ^ {*} ] _ {\mathcal {B}}\right) \frac {\log (n / \delta)}{\sqrt {n}}
$$
from Corollary D.4 for a slightly modified constant  $C > 0$  (with  $B_{1} = 0$ ) and up to higher order terms in  $(\lambda m)^{-1}$  and  $n$ .
Step 5. By the union bound, all probabilistic bounds hold simultaneously with probability at least  $1 - 4\delta$ . In this case
$$
\mathcal {R} (f _ {\hat {\theta}}) \leq C \left(\frac {Q ^ {2}}{m} \left(\mathbb {E} \big [ \| x \| ^ {2} \big ] + \sigma^ {2} \sqrt {\frac {\log (1 / \delta)}{n}}\right) + \lambda Q + [ f ^ {*} ] _ {\mathcal {B}} \left(\mathbb {E} \| x \| + \sigma^ {2} + [ f ^ {*} ] _ {\mathcal {B}}\right) \frac {\log (n / \delta)}{\sqrt {n}}\right)
$$
up to higher order terms in  $m^{-1}$ ,  $\log n / \sqrt{n}$ ,  $(\lambda m)^{-1}$  etc.
Since  $\mathcal{R}(\theta) = \| f_{\theta} - f^{*}\|_{L^2 (\mu)}^2$ , we can interpret Theorem E.1 as a convergence statement in  $L^2 (\mu)$  at a suitable rate. The statement generalizes to  $L^p$ -convergence at a rate.
Corollary E.2 ( $L^p$ -convergence). Let  $p \in [1, \infty]$  and  $\hat{\theta}$  as in Theorem E.1. Then there exists a constant  $\tilde{C} > 0$  depending on  $\mathbb{E}\|x\|$ ,  $\mathbb{E}[\|x\|^2]$ ,  $\sigma^2$  and  $p$  such that
$$
\left\| f _ {\hat {\theta}} - f ^ {*} \right\| _ {L ^ {p} (\mu)} \leq \tilde {C} \left(\widehat {\mathcal {R}} _ {n, m, \lambda} (\hat {\theta}) ^ {1 / 2} + [ f ^ {*} ] _ {\mathcal {B}}\right) ^ {1 - 1 / p} \left\| f _ {\hat {\theta}} - f ^ {*} \right\| _ {L ^ {2} (\mu)} ^ {1 / p}.
$$
Proof. Since  $\mu$  is sub-Gaussian, we note that all moments of  $\mu$  are finite:  $\mathbb{E}[(1 + \| x\|)^q ] < \infty$  for all  $q\in [1,\infty)$ . In particular, if  $g$  is a measurable function which satisfies  $|g(x)|\leq C_g(1 + \| x\|)$  for some  $C)g > 0$ , then
$$
\| g \| _ {L ^ {p} (\mu)} ^ {p} = \mathbb {E} [ g \cdot g ^ {p - 1} ] \leq \mathbb {E} [ g ^ {2} ] ^ {1 / 2} \mathbb {E} [ g ^ {2 (p - 1)} ] ^ {1 / 2} = \| g \| _ {L ^ {2}} \| g \| _ {L ^ {2 (p - 1)}} ^ {p - 1}.
$$
If  $g = f_{\hat{\theta}} - f^{*} \in \mathcal{B}$ , then by the continuous embedding  $\mathcal{B} \hookrightarrow L^q(\mu)$  we find that
$$
\left\| f _ {\hat {\theta}} - f ^ {*} \right\| _ {L ^ {2 (p - 1)} (\mu)} \leq C \left(| f _ {\hat {\theta}} - f ^ {*} | (0) + [ f _ {\hat {\theta}} - f ^ {*} ] _ {\mathcal {B}}\right).
$$
Recall that  $|f_{\hat{\theta}} - f^{*}|(0)\leq \widehat{\mathcal{R}}_{n,m,\lambda}(\hat{\theta})^{1 / 2} + C[f^{*}]_{\mathcal{B}}$
We note that Corollary E.2 is generally suboptimal. Indeed, for  $p \leq 2$ , the stronger bound
$$
\left\| f _ {\hat {\theta}} - f ^ {*} \right\| _ {L ^ {p} (\mu)} \leq \left\| f _ {\hat {\theta}} - f ^ {*} \right\| _ {L ^ {2} (\mu)} = O \left(\left(\frac {1}{m} + \lambda + \frac {\log n}{\sqrt {n}}\right) ^ {1 / 2}\right)
$$
holds as  $L^2 (\mu)$  embeds continuously into  $L^p (\mu)$ .
E.2 Gamma-expansion of regularized risk functionals
As before, we denote  $\theta_{n} = (a,W,b)_{n}\in \mathbb{R}^{m_{n}}\times \mathbb{R}^{m_{n}\times d}\times \mathbb{R}^{m_{n} + 1}$ . Since  $\mathcal{R}(f_{\theta}) = \| f_{\theta} - f^{*}\|_{L^2 (\mu)}^2$ , Theorem E.1 can be taken as a statement that  $f_{\hat{\theta}_n}\to f^*$  as  $n\to \infty$  in  $L^2 (\mu)$ . However, this does not tell us about the behavior of  $f_{\hat{\theta}_n}$  in a  $\mu$ -null set, i.e. where the distribution  $\mu$  provides us no information. This interpolation between known values can be deduced from our next result. We first present a simplified version, in which we assume that we have already taken the limits  $m,n\to \infty$  before taking  $\lambda \rightarrow 0$ . We couple the limits  $n,m_n,\lambda_n$  below.
We use the notion of  $\Gamma$ -convergence from the calculus of variations. For a brief introduction, see Appendix B.  $\Gamma$ -convergence depends on the underlying topology of the space, and we make the following convention: We say that  $f_{\lambda} \xrightarrow{good} f$  if  $f_{\lambda} \to f$  locally uniformly (uniformly on compact sets) and in  $L^2(\mu)$ . Other definitions are admissible and lead to the same general theory. Since Barron functions grow at most linearly at  $\infty$  due to Lipschitz-continuity, we note that this is notion of convergence is generated by a metric
$$
d (f, g) = \max _ {x \in \mathbb {R} ^ {d}} \frac {| f (x) - g (x) |}{1 + \| x \| ^ {2}}
$$
at least on bounded subsets of Barron space. This suffices for all applications below and spares us from considering  $\Gamma$ -convergence on more general topological spaces - which is also possible.
Theorem E.3. Let
$$
\mathcal {R} _ {\lambda}: \mathcal {B} \rightarrow [ 0, \infty), \quad \mathcal {R} _ {\lambda} (f) = \| f - f ^ {*} \| _ {L ^ {2} (\mu)} ^ {2} + \lambda [ f ] _ {\mathcal {B}}.
$$
$$
F _ {\lambda}: \mathcal {B} \to [ 0, \infty) F _ {\lambda} (f) = \frac {\mathcal {R} _ {\lambda} (f)}{\lambda} = \frac {\| f - f _ {d} ^ {*} \| _ {L ^ {2} (\mu)} ^ {2}}{\lambda} + [ f ] _ {\mathcal {B}}
$$
$$
F: \mathcal {B} \to [ 0, \infty ] \qquad F (f) = \left\{ \begin{array}{l l} {[ f ] _ {\mathcal {B}}} & {i f f = f ^ {*}   \mu \text {- a . e .}} \\ {+ \infty} & {e l s e} \end{array} \right..
$$
Then  $\Gamma -\lim_{\lambda \to 0}F_{\lambda} = F$  with respect to the notion of convergence  $\xrightarrow{good}$  defined above.