SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Despite its somewhat counterintuitive properties,  $\Gamma$ -convergence has proved invaluable in many areas of the calculus of variations. It has been applied to homogenization by Bach et al. [2021], dimension reduction for thin sheets and shells by Friesecke et al. [2002a, 2003, 2002b], Bhattacharya et al. [2016], Lewicka et al. [2010] and the study of phase boundaries by Modica and Mortola [1977], Modica [1987]. While the  $\Gamma$ -convergence of functionals does not imply the convergence of their gradient flows even in situations of practical significance (see e.g. the example of Dondl et al. [2019]), Serfaty [2011], Sandier and Serfaty [2004], Mugnai and Röger [2011], Ilmanen [1993], Alikakos et al. [1994] provide important examples of situations where this can be established in a suitable sense. Even more, Bronsard and Kohn [1990] use  $\Gamma$ -convergence to gain insight into PDE dynamics.
C Homogeneous Barron spaces
In this section, we introduce the abstract framework which is used to prove our main theoretical result, Corollary 3.3. A neural network with  $m$  neurons in a single hidden layer can be represented as
$$
f _ {m} (x) = b _ {0} + \sum_ {i = 1} ^ {m} a _ {i} \sigma \left(w _ {i} ^ {T} x + b _ {i}\right) \quad \text {o r} \quad f _ {m} (x) = b _ {0} + \frac {1}{m} \sum_ {i = 1} ^ {m} a _ {i} \sigma \left(w _ {i} ^ {T} x + b _ {i}\right). \tag {4}
$$
The network weights and biases are  $(a, W, b) \in \mathbb{R}^m \times \mathbb{R}^{m \times d} \times \mathbb{R}^{m + 1}$ . The normalization depends on personal preference, with the former being more common in practice and the latter more common in theoretic analyses. We define the weight decay regularizer by
$$
R _ {W D} (a, W, b) = \frac {\| a \| _ {\ell^ {2}} ^ {2} + \| W \| _ {F} ^ {2}}{2} = \frac {1}{2} \left(\sum_ {i = 1} ^ {m} a _ {i} ^ {2} + \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {d} w _ {i j} ^ {2}\right)
$$
or  $R_{WD}(a,W,b) = \frac{1}{2m}\sum_{i=1}^{m}\left(a_i^2 +\|w_i\|_{\ell^2}^2\right)$  respectively. Here  $\| \cdot \|_2$  denotes the Euclidean  $\ell^2$ -norm of a vector and  $\|W\|_F$  denotes the Frobenius norm of the matrix  $W$  whose rows are the vectors  $w_i^T$ . Note that we do not control the magnitude of the biases  $b_i$  in the regularizer. This is a common approach, as the bias does not influence the Lipschitz-constant of the function represented by the neural network, which is useful in studying the generalization of the neural network.
We study function classes corresponding to arbitrarily wide neural networks with a single hidden layer, where the norm corresponds to the weight decay regularizer. We dub these function spaces 'homogeneous Barron spaces' in analogy to the more classical Barron spaces studied by E et al. [2019c], Ma et al. [2020], E and Wojtowytsch [2020], which correspond to a weight decay regularizer which also controls the bias. For homogeneous Barron spaces, coordinate transformations by Euclidean motions induce an isometry of the function class, while the origin plays a special role in classical Barron spaces. This justifies our terminology, as the data space is treated as isotropic and homogeneous by this function class. Homogeneous Barron spaces have also been studied by Ongie et al. [2019], Parhi and Nowak [2021, 2022] under the name Radon-BV spaces. A closely related class of spaces has been considered as the 'variation spaces of the ReLU dictionary' by Siegel and Xu [2020, 2022, 2023].
Heuristically, (homogeneous) Barron spaces are a function class tailored to replacing the finite superposition of ReLU ridges in (4) by an arbitrary superposition while keeping the weight decay regularizer finite. Due to the lack of control over the bias term, we will see that a slightly awkward technical definition is needed. Let  $\pi$  be a probability distribution on the parameter space  $\mathbb{R} \times \mathbb{R}^d \times \mathbb{R}$ . We would like to define
$$
f _ {\pi , b _ {0}}: \mathbb {R} ^ {d} \to \mathbb {R}, \quad f _ {\pi , b _ {0}} (x) = b _ {0} + \mathbb {E} _ {(a, w, b) \sim \pi} [ a \sigma (w ^ {T} x + b) ] = \int_ {\mathbb {R} \times \mathbb {R} ^ {d} \times \mathbb {R}} a \sigma (w ^ {T} x + b) \mathrm {d} \pi_ {(a, w, b)}
$$
$$
R _ {W D} (\pi) = \frac {1}{2} \int_ {\mathbb {R} \times \mathbb {R} ^ {d} \times \mathbb {R}} | a | ^ {2} + \| w \| _ {2} ^ {2} d \pi_ {(a, w, b)}.
$$
$f_{\pi ,b_0}$  is an analogue of neural networks with a single hidden layer, but of arbitrary and possibly uncountably infinite width. Every finite neural network can be expressed in this fashion for the empirical measure  $\pi_{m} = \frac{1}{m}\sum_{i = 1}^{m}\delta_{(a_{i},w_{i},b_{i})}$ , in which case  $R_{WD}(\pi_m) = R_{WD}(a,W,b)$ .
Unfortunately, even if  $R_{WD}(\pi) < \infty$ , it is not clear that the integral defining  $f_{\pi}$  exists in a meaningful sense. We do however note that formally
$$
\begin{array}{l} \left| f _ {\pi} (x) - f _ {\pi} (y) \right| = \mathbb {E} \left[ a \left\{\sigma \left(w ^ {T} x + b\right) - \sigma \left(w ^ {T} y + b\right) \right\} \right] \leq \mathbb {E} \left[ | a | \left| \left| w ^ {T} x + b \right| - \left| w ^ {T} y + b \right| \right| \right] \\ \leq \| x - y \| \mathbb {E} \big [ | a | \| w \| \big ] \leq \frac {\| x - y \|}{2} \mathbb {E} \big [ | a | ^ {2} + \| w \| ^ {2} \big ] = \| x - y \| R _ {W D} (\pi). \\ \end{array}
$$
Thus, the integral defining  $f_{\pi}(x)$  exists for all  $x$  if and only if it exists for, say,  $x = 0$ . We exploit this in the following modified definition. Let  $\pi$  be a probability distribution on the parameter space  $\mathbb{R} \times \mathbb{R}^d \times \mathbb{R}$  and  $y \in \mathbb{R}$ . We denote
$$
f _ {\pi , y} (x) = y + \mathbb {E} _ {(a, w, b) \sim \pi} \left[ a \left(\sigma \left(w ^ {T} x + b\right) - \sigma (b)\right) \right]
$$
By the same argument as before, we observe that
1.  $f_{\pi ,y}(0) = y$  and
2.  $|f_{\pi, y}(x) - f_{\pi, y}(x')| \leq R_{WD}(\pi) \| x - x' \|$ .
The class of functions of the form  $f_{\pi, y}$  forms the homogeneous Barron space. Still, every finite neural network  $f_{m}$  can be represented in this fashion with  $b_{0} = y - \sum_{i=1}^{m} a_{i} \sigma(b_{i})$  or  $b_{0} = y - \frac{1}{m} \sum_{i=1}^{m} a_{i} \sigma(b_{i})$ .
Definition C.1 (Homogeneous Barron space). Let  $f: \mathbb{R}^d \to \mathbb{R}$  be a function. We define the semi-norms
$$
[ f ] _ {\mathcal {B}} = \inf  _ {f \equiv f _ {\pi , y}} R _ {W D} (\pi), \qquad [ f ] _ {0} = | f (0) |.
$$
The homogeneous Barron space is the function class  $\mathcal{B}(\mathbb{R}^d) = \{f:\mathbb{R}^d\to \mathbb{R}:[f]_{\mathcal{B}} < \infty \}$ . By the definition of a function  $[f]_0 < \infty$  is automatically true.
We note a few important properties. First, we consider two function classes:
$$
\begin{array}{l} \mathcal {F} _ {Q} = \overline {{\operatorname {c o n v} \left\{a \left(\sigma (w \cdot x + b) - \sigma (b)\right) : a ^ {2} + \| w \| ^ {2} \leq 2 Q \right\}}} \\ \mathcal {F} _ {Q} (R) = \overline {{\operatorname {c o n v} \left\{a \left(\sigma (w \cdot x + b) - \sigma (b)\right): a ^ {2} + \| w \| ^ {2} \leq 2 Q , | b | \leq \sqrt {Q} R \right\}}}. \\ \end{array}
$$
Note that  $\mathcal{F}_Q(R) \subseteq \mathcal{F}_Q \subseteq \{f \in \mathcal{B} : f(0) = 0, [f]_{\mathcal{B}} \leq Q\}$ . The closure is taken with respect to locally uniform convergence, i.e. pointwise convergence which is uniform on all compact sets. Due to the homogeneity of ReLU activation, we may prove the following.
Lemma C.2. The identity  $\mathcal{F}_Q = \{f\in \mathcal{B}:f(0) = 0,[f]_{\mathcal{B}}\leq Q\}$  holds.
While the claim is natural, its proof is surprisingly technical and postponed until the end of the section. The class  $\mathcal{F}_Q(R)$  will be used below for technical purposes. Of major importance below are the compact embedding theorem and the direct approximation theorem.
Theorem C.3 (Compact embedding). Let  $f_{n} \in \mathcal{B}$  be a sequence such that  $\lim \inf_{n\to \infty}[f_n]_0 + [f_n]_{\mathcal{B}} < +\infty$ . Then there exists  $f$  in  $\mathcal{B}$  such that
1.  $f_{n}\to f$  in  $C^0 (K)$  for all compact sets  $K\subseteq \mathbb{R}^{d}$
2.  $f_{n}\rightarrow f$  in  $L^p (\mu)$  for all measures  $\mu$  with finite  $p$ -th moments,  $p\in [1,\infty)$ .
3.  $[f]_{\mathcal{B}} \leq \lim \inf_{n \to \infty} [f_n]_{\mathcal{B}}$ .
Proof. It is sufficient to show that the set  $\tilde{\mathcal{F}}_Q = \{a\big(\sigma (w\cdot x + b) - \sigma (b)\big)\mid a^2 +|w|^2\leq 2Q\}$  is compact in  $C^0 (K)$  and  $L^2 (\mu)$  for all  $K$  and  $\mu$  as above, in which case also its closed convex hull is compact [Rudin, 1991, Theorem 3.20.]. To this end, observe that the map
$$
F: \mathbb {R} \times \mathbb {R} ^ {d} \times \mathbb {R} \rightarrow C ^ {0} (K), \qquad (a, w, b) \mapsto a \big (\sigma (w ^ {\cdot} + b) - \sigma (b) \big)
$$
is continuous for any compact set  $K$ . Since  $K$  is compact, we have  $K \subseteq B_R(0)$  for some  $R > 0$  and thus  $|w \cdot x| \leq \sqrt{2Q} R$  for all  $x \in K$ . In particular,
$$
\sigma (w \cdot x + b) - \sigma (b) = \left\{ \begin{array}{l l} w \cdot x & \text {i f} b > \sqrt {2 Q} R \\ 0 & \text {i f} b <   \sqrt {2 Q} R \end{array} \right..
$$
Hence  $\tilde{\mathcal{F}}_Q = F(\{(a,w,b):a^2 +|w|^2\leq 2Q,b\leq \sqrt{2Q} R\})$  is the continuous image of a compact set, hence compact. We have thus proved a compact embedding into  $C^0 (K)$  for any compact set  $K$ .
Exhausting  $\mathbb{R}^d$  by the sequence of compact sets  $\overline{B_m(0)}$ ,  $m \in \mathbb{N}$  and using a diagonal sequence argument, we see that under the conditions of Theorem C.3, there exists  $f \in \mathcal{B}$  such that  $f_n \to f$  pointwise everywhere on  $\mathbb{R}^d$  and uniformly on compact subsets. Additionally, we observe that  $f(0) = 0$  and there exists a uniform upper bound on the Lipschitz constants of the sequence  $f_n$ . We conclude that  $f_n \to f$  in  $L^p(\mu)$  from the Dominated Convergence Theorem using  $Q\|x\|$  as a dominating function.
As a consequence, we find the following.
Corollary C.4. 1.  $\mathcal{B}$  is a Banach space.
2.  $C_c^\infty (\mathbb{R}^d)\subseteq \mathcal{B}(\mathbb{R}^d)$
Proof. The first claim follows as in [Siegel and Xu, 2021, Lemma 1], where it is proved in a more general context for dictionaries which are compact in a Hilbert space - in our case, the dictionary  $x \mapsto a\{\sigma(w^T x + b) - \sigma(b)\}$ . The second claim follows from [Ongie et al., 2019, Corollary 1].  $\square$
We conclude with a theorem which establishes a rate of approximation for Barron functions in a weaker topology.
Theorem C.5 (Direct approximation). Let  $f \in \mathcal{B}$  and  $\mu$  a measure on  $\mathbb{R}^d$  with finite second moments. Then for any  $m \in \mathbb{N}$  there exist  $c \in \mathbb{R}$  and  $(a_i, w_i, b_i) \in \mathbb{R} \times \mathbb{R}^d \times \mathbb{R}$  such that
$$
\sum_ {i = 1} ^ {m} a _ {i} ^ {2} + \| w _ {i} \| ^ {2} \leq [ f ] _ {\mathcal {B}}, \quad \left\| f - c - \sum_ {i = 1} ^ {m} a _ {i} \sigma (w _ {i} ^ {T} x + b _ {i}) \right\| _ {L ^ {2} (\mu)} \leq \frac {2 [ f ] \mathcal {B}}{\sqrt {m}} \sup _ {\| w \| = 1} \sqrt {\int_ {\mathbb {R} ^ {d}} | w ^ {T} x | ^ {2} \mathrm {d} \mu_ {x}}.
$$
This notion of convergence is generated by a topology, but not a metric. Other notions of convergence can be considered and induce the same function class.
Proof. A proof of this result can be found in [Wojtowytsch, 2022, Appendix C] in the proof of Proposition 2.6.  $\square$
Proof of Lemma C.2. Step 1. Assume that  $f \in \mathcal{B}$  such that  $f(0) = 0$  and  $[f]_{\mathcal{B}} \leq Q$ . By the Direct Approximation Theorem (which is proved in [Wojtowytsch, 2022, Appendix C] without using Lemma C.2), we find that for every  $m \in \mathbb{N}$  and every measure  $\mu$  on  $\mathbb{R}^d$  with finite second moments, there exists
$$
f _ {m} (x) = \frac {1}{m} \sum_ {i = 1} ^ {m} a _ {i} \{\sigma (w _ {i} \cdot x + b _ {i}) - \sigma (b _ {i}) \} \quad \in \mathcal {F} _ {Q}
$$
such that  $\| f_m - f \|_{L^2(\mu)} \leq C_\mu \| f \|_{\mathcal{B}} m^{-1/2}$ . In particular,  $f$  is in the closed convex hull of  $\{a (\sigma(w^T x + b) - \sigma(b)) : a^2 + |w|^2 \leq 2Q\}$  if the closure is taken with respect to the  $L^2(\mu)$  topology. Additionally, the sequence  $f_m$  has a uniformly bounded Lipschitz constant and is therefore compact in  $C^0(K)$  for all compact  $K$  by a corollary to the Arzela-Ascoli theorem [Dobrowolski, 2010, Satz 2.42]. In particular,  $f_m \to f$  uniformly and thus  $f \in \mathcal{F}_Q$ .
Step 2. Denote  $f_{(a,w,b)}(x) = a\{\sigma (w^T x + b) - \sigma (b)\}$ . Since  $f_{(a,w,b)}(0) = a\{\sigma (b) - \sigma (b)\} = 0$  for all  $a,w,b$ , we conclude that  $f(0) = 0$  for all  $f\in \mathcal{F}_Q$ . If  $f\in \mathcal{F}_Q$ , then there exists a sequence
$$
f _ {n} (x) = \sum_ {i = 1} ^ {N _ {n}} \lambda_ {i, n} a _ {i, n} \left\{\sigma \left(w _ {i, n} \cdot x + b _ {i, n}\right) - \sigma \left(b _ {i, n}\right) \right\}
$$
such that  $f_{n} \to f$  locally uniformly. If the biases remain uniformly bounded, the sequence of empirical distributions
$$
\pi_ {n} = \frac {1}{N _ {n}} \sum_ {i = 1} ^ {N _ {n}} \lambda_ {i, n} \delta_ {\left(a _ {i, n}, w _ {i, n}, b _ {i, n}\right)}
$$
has a convergent subsequence by Prokhorov's Theorem [Klenke, 2006, Satz 13.29]. We denote the limiting distribution as  $\pi$ . The convergence of Radon measures implies the convergence of  $f_{n}$  to  $f_{\pi}(x) = \mathbb{E}_{(a,w,b)\sim \pi}\left[a\left\{\sigma (w^T x + b) - \sigma (b)\right\} \right]$  by definition. Since  $f_{n}$  converges locally uniformly by assumption,  $f = f_{\pi}\in \mathcal{B}$  and  $[f]_{\mathcal{B}}\leq Q$
If the biases do not remain bounded, we note that for every compact set  $K \subseteq \mathbb{R}^d$  we can extract a convergent subsequence of the measures by the same argument used to prove Theorem C.3, effectively making the sequence of biases bounded. We can extend the argument to the entire space exploiting that
$$
\lim  _ {b \rightarrow \infty} \left(\sigma (w \cdot x + b) - \sigma (b)\right)\rightarrow \sigma (b / | b |) w \cdot x
$$
locally uniformly.
D Rademacher complexity of homogeneous Barron space
Following a classical strategy implemented e.g. by E et al. [2019a] in a similar context, we estimate the Rademacher complexity of homogeneous Barron space and use it to bound the generalization gap (i.e. the discrepancy between empirical risk and population risk). In our setting, we face additional technical obstacles:
1. We deal with general sub-Gaussian data distributions  $\mu$  rather than data distributions with compact support.
2. We do not control the magnitude of the bias variables.
3. We consider  $\ell^2$ -loss, which is neither globally Lipschitz-continuous nor bounded.
In combination, these complications require a refined technical analysis similar to Appendix C. Let us summarize several notations which will be needed below.
- Rad - the empirical Rademacher complexity of a function class over a given dataset.
-  $\mathrm{Rad}_n$  - the expected Rademacher complexity of a function class over a data set composed of  $n$  iid samples from the data distribution  $\mu$ .
-  $\mathcal{R}$  - the population risk  $\mathcal{R}(f) = \| f - f^{*}\|_{L^2 (\mu)}^2 = \mathbb{E}_{x\sim \mu}[|f(x) - f^{*}(x)|^2 ]$  . We generally take this to operate on the level of functions, parametrized or not. By an abuse of notation, we identify  $\mathcal{R}(a,W,b)\coloneqq \mathcal{R}(f_{(a,W,b)})$
-  $\widehat{\mathcal{R}}_n$  - the empirical risk  $\widehat{\mathcal{R}}_n(f) = \| f - f^*\|_{L^2 (\mu_n)}^2 = \frac{1}{n}\sum_{i = 1}^n |f(x_i) - f^* (x_i)|^2$  over a data set  $\{x_{1},\ldots ,x_{n}\}$  where  $\mu_{n} = \frac{1}{n}\sum_{i = 1}^{n}\delta_{x_{i}}$  is the empirical measure. Equally,  $\widehat{\mathcal{R}}_n$  can be considered for functions or parameters with the natural identification.
-  $\widehat{\mathcal{R}}_{n,m,\lambda}$ . The regularized empirical risk
$$
\widehat {\mathcal {R}} _ {n, m, \lambda} (a, W, b) = \widehat {\mathcal {R}} _ {n} (a, W, b) + \frac {\lambda}{2} \big (\| a \| ^ {2} + \| W \| ^ {2} \big).
$$
We only consider this quantity on the parameter level, where it is computable. While the weight decay regularizer is an upper bound for  $[f_{(a,W,b)}]_B$ , the two are generally not the same since the parameter-to-function map of a neural network is generally not injective.
-  $R_{WD}$  - the weight decay regularizer.
-  $\mathcal{F}_Q$  - the set of functions for which  $[f]_{\mathcal{B}} \leq Q$  and  $f(0) = 0$ .
-  $\mathcal{F}_{A,Q}$  - the set of functions for which  $[f]_{\mathcal{B}} \leq Q$  and  $|f(0)| \leq A$ .
As is common in the mathematics community,  $C$  will generally denote a constant which does not depend on quantities (unless specified otherwise) and which may change value from line to line. Some facts about sub-Gaussian distributions, which we believe to be well-known to the experts, are collected in Appendix H.
Definition D.1 (Rademacher Complexity). Let  $S = \{x_{1},\ldots ,x_{n}\}$  be a set of points in  $\mathbb{R}^d$  (a data sample) and  $\mathcal{F}$  a real-valued function class. We define the empirical Rademacher complexity of  $\mathcal{F}$  on the data sample as
$$
\widehat {\operatorname {R a d}} (\mathcal {F}; S) = \mathbb {E} _ {\varepsilon} \left[ \sup  _ {f \in \mathcal {F}} \frac {1}{n} \sum_ {i = 1} ^ {n} \varepsilon_ {i} f (x _ {i}) \right]
$$
where  $\varepsilon_{i}$  are iid random variables which take the values  $\pm 1$  with equal probability  $\frac{1}{2}$ . The population Rademacher complexity is defined as
$$
\operatorname {R a d} _ {n} (\mathcal {F}) = \mathbb {E} _ {S \sim \mu^ {n}} \left[ \widehat {\operatorname {R a d}} (\mathcal {F}; S) \right],
$$
i.e. as the expected empirical Rademacher complexity over a set of  $n$  iid data points.
In this section, we will find a upper bound of Rademacher Complexity of  $\mathcal{B}$ . We will denote by  $S_{n}$  the set of  $n$  samples, and  $\widehat{\mathrm{Rad}}(\mathcal{F}, S_{n})$  the sample Rademacher Complexity of  $\mathcal{F}$  given the samples  $S_{n}$ . We furthermore denote  $R := \max\{\|x_{1}\|, \ldots, \|x_{n}\|\}$  and consider the function classes  $\mathcal{F}_{Q}$  and  $\mathcal{F}_{Q}(R)$  as in Appendix C:
$$
\begin{array}{l} \mathcal {F} _ {Q} = \overline {{\operatorname {c o n v} \left\{a \left(\sigma (w \cdot x + b) - \sigma (b)\right): a ^ {2} + \| w \| ^ {2} \leq 2 Q \right\}}} \\ \mathcal {F} _ {Q} (R) = \overline {{\operatorname {c o n v} \left\{a \left(\sigma (w \cdot x + b) - \sigma (b)\right): a ^ {2} + \| w \| ^ {2} \leq 2 Q , | b | \leq \sqrt {Q} R \right\}}}. \\ \end{array}
$$
Lemma D.2. Let  $S_{n} = \{x_{1},\ldots ,x_{n}\}$  be a data set in  $\mathbb{R}^d$  . Then
$$
\widehat {\operatorname {R a d}} \left(\mathcal {F} _ {Q}, S _ {n}\right) \leq \frac {(1 + 3 \sqrt {2}) Q}{\sqrt {n}} \max  _ {1 \leq i \leq n} \| x _ {i} \|.
$$
Assume  $\mu$  is a  $\sigma^2$  sub-Gaussian distribution in  $\mathbb{R}^d$ . Then
$$
\operatorname {R a d} \left(\mathcal {F} _ {Q}\right) \leq (1 + 3 \sqrt {2}) Q \left(\frac {\mathbb {E} _ {x \sim \mu} [ \| x \| ]}{\sqrt {n}} + \sigma \sqrt {2 \frac {\log n}{n}}\right)
$$
for all  $n\geq 2$
Proof. Initially, we fix a set  $S = \{x_2, \ldots, x_n\}$  of  $n$  points. We will later take the expectation over  $S$ , using the sub-Gaussian property of  $\mu$  for an explicit norm bound. Define  $R := \max_{1 \leq i \leq n} \| x_i \|$ . To this end, we first prove the following claim, which enables us to focus on only single neuron functions instead of entire  $\mathcal{F}_Q$ :
Claim: Let  $\varepsilon_1, \ldots, \varepsilon_n \in \mathbb{R}$ . Then
$$
\sup _ {\mathcal {F} _ {Q}} \sum_ {i} \epsilon_ {i} f (x _ {i}) = \sup _ {a ^ {2} + \| w \| ^ {2} \leq 2 Q} \sum_ {i} \epsilon_ {i} a \big \{\sigma (w ^ {T} x _ {i} + b) - \sigma (b) \big \}
$$
Proof of Claim. Note that  $\mathcal{F}_Q$  is the closed convex hull of single neuron ridge functions, i.e. single neuron ridge functions are the extreme points of the closed convex set  $\mathcal{F}_Q$ .
To verify the claim, first note that  $f \mapsto \sum_{i=1}^{n} \varepsilon_i f(x_i)$  is a continuous linear functional on  $C^0(K)$  for any compact  $K \subseteq \mathbb{R}^d$  containing the finite set  $S$ . It is well known that  $C^0(K)$  is a Banach Space. Therefore, if  $\{a(\sigma(w \cdot x + b) - \sigma(b)) | a^2 + |w|^2 \leq 2Q\}$  is compact in  $C^0(K)$ , then [Rudin, 1991, Theorem 3.20.] implies that  $\mathcal{F}_Q$ , a closed convex hull of the compact set, is also compact. Then, from the compactness of  $\mathcal{F}_Q$ , we can use Bauer [1958]'s maximum principle and see that the supremum is attained at an extreme point. Compactness follows from the compact embedding Theorem, see Theorem C.3 above.
Over the next steps, we will bound  $\widehat{\operatorname{Rad}}(\mathcal{F}_Q, S_n)$ .
Step 1. In this step, we prove that
$$
\widehat {\operatorname {R a d}} \left(\mathcal {F} _ {Q}; S _ {n}\right) = \widehat {\operatorname {R a d}} \left(\mathcal {F} _ {Q} (R); S _ {n}\right).
$$
To show this, we first observe that if  $|b| \geq \| w\| R$ , then  $\sigma (w\cdot x + b) - \sigma (b) = \sigma \bigl (sgn(b)\bigr)w\cdot x$  since  $|w^{T}x_{i}|\leq \| w\| \| x_{i}\| \leq |b|R$  for all  $1\leq i\leq n$ . This means for  $\forall |b|\geq \| w\| R$ , the precise value of  $b$  does not change the value of  $\sigma (w\cdot x + b) - \sigma (b)$ .
Now, we compute the  $\widehat{\mathrm{Rad}} (\mathcal{F}_Q,S_n)$
$$
\begin{array}{l} \widehat {n \operatorname {R a d}} (\mathcal {F} _ {Q}, S _ {n}) = \mathbb {E} _ {\epsilon} \left[ \sup  _ {\mathcal {F} _ {Q}} \sum_ {i} \epsilon_ {i} f (x _ {i}) \right] \\ = \mathbb {E} _ {\epsilon} \left[ \sup  _ {a ^ {2} + \| w \| ^ {2} \leq 2 Q} \sum_ {i} \epsilon_ {i} a \left\{\sigma (w \cdot x _ {i} + b) - \sigma (b) \right\} \right] \\ = \mathbb {E} _ {\epsilon} \left[ \sup  _ {a ^ {2} + \| w \| ^ {2} \leq 2 Q, | b | \leq \| w \| R} \sum_ {i} \epsilon_ {i} a \left\{\sigma \left(w \cdot x _ {i} + b\right) - \sigma (b) \right\} \right] \\ = n \widehat {\operatorname {R a d}} \big (\mathcal {F} _ {Q} (R), S _ {n} \big) \\ \end{array}
$$
For the first line, we used the claim.
Step 2. Using the uniform bound on the magnitude of the bias from the previous step, in this step we bound the Rademacher complexity by
$$
\widehat {\operatorname {R a d}} (\mathcal {F} _ {Q}; S) = \widehat {\operatorname {R a d}} \big (\mathcal {F} _ {Q} (R); S \big) \leq \mathbb {E} _ {\varepsilon} \left[ \sup  _ {| w | \leq Q, | b | \leq Q R} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \varepsilon_ {i} \sigma (w \cdot x _ {i} + b) \right| \right] + \frac {Q R}{\sqrt {n}}
$$
We verify this from the definition of Rademacher Complexity of  $\mathcal{F}_Q(R)$ . Note that  $a\sigma (wx + b) = (\lambda a)\sigma ((w / \lambda)x + b / \lambda)$ . In particular, we may assume without loss of generality that  $|a|^2 = \| w\|^2\leq Q$  for optimal balance which makes  $a^2 +\| w\|^2$  minimal without changing the neuron output.
$$
n \operatorname {R a d} \big (\mathcal {F} _ {Q} (R), S _ {n} \big) = \mathbb {E} _ {\epsilon} \left[ \sup _ {\mathcal {F} _ {Q}} \epsilon_ {i} f (x _ {i}) \right]
$$
$$
\begin{array}{l} = \mathbb {E} _ {\epsilon} \left[ \sup  _ {a ^ {2} + \| w \| ^ {2} \leq Q, | b | \leq \| w \| R} \sum_ {i = 1} ^ {n} \epsilon_ {i} a \left(\sigma (w \cdot x _ {i} + b) - \sigma (b)\right) \right] \\ \leq \mathbb {E} _ {\epsilon} \left[ \sup  _ {| a | = \| w \| \leq \sqrt {Q}, | b | \leq \sqrt {Q} R} \left(\left| \sum_ {i} \epsilon_ {i} a \sigma (w \cdot x _ {i} + b) \right| + \left| \sum_ {i} \epsilon_ {i} a \sigma (b) \right|\right) \right]. \\ \end{array}
$$
In this step, we only consider the first term.:
$$
\begin{array}{l} \mathbb {E} _ {\varepsilon} \left[ \sup  _ {| a | = \| w \| \leq \sqrt {Q}, | b | \leq \sqrt {Q} R} \left| \sum_ {i} \epsilon_ {i} a \sigma (b) \right| \right] \leq \mathbb {E} _ {\epsilon} \left[ \sup  _ {| a | \leq \sqrt {Q}, | b | \leq \sqrt {Q} R} \left| \sum_ {i} \epsilon_ {i} a \sigma (b) \right| \right] \\ \leq \sup  _ {| a | \leq \sqrt {Q}, | b | \leq \sqrt {Q} R} | a | \sigma (b)   \mathbb {E} _ {\epsilon} \left| \sum_ {i} \epsilon_ {i} \right| \leq Q R   \sqrt {n}. \\ \end{array}
$$