SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 2. For any  $L, B \geq 1$  and  $\epsilon \in (0,1]$  s.t.  $\frac{LB}{\epsilon} \geq 1$ , let  $\Psi_{L,a,n}$  be the class of all  $L$ -Lipschitz functions from  $\{\mathbf{x} \in \mathbb{R}^n : ||\mathbf{x}|| \leq B\}$  to  $\mathbb{R}$ , which equal some fixed  $a \in \mathbb{R}$  at  $\mathbf{0}$ . Let  $\mathcal{W}_{B,n}$  be the class of linear functions from  $\mathbb{R}^d$  to  $\mathbb{R}^n$  over the domain  $\{\mathbf{x} \in \mathbb{R}^d : ||\mathbf{x}|| \leq 1\}$  with Frobenius norm at most  $B$ , namely
$$
\mathcal {W} _ {B, n} := \left\{\mathbf {x} \rightarrow W \mathbf {x}: W \in \mathbb {R} ^ {n \times d}, \| W \| _ {F} \leq B \right\}.
$$
Then the Rademacher complexity of  $\Psi_{a,L,n} \circ \mathcal{W}_{B,n} \coloneqq \{\psi \circ g : \psi \in \Psi_{L,a,n}, g \in \mathcal{W}_{B,n}\}$  on  $m$  inputs from  $\{\mathbf{x} \in \mathbb{R}^d : \| \mathbf{x} \| \leq 1\}$  is at most  $\epsilon$ , if  $m \geq \left(\frac{LB}{\epsilon}\right)^{\frac{\epsilon L^2 B^2}{\epsilon^2}}$  for some universal constant  $c > 0$ .
Since  $\mathcal{F}_{B,n,d}^{f,W_0 = 0}\subseteq \Psi_{L,a,n}\circ \mathcal{W}_{B,n}$  for any fixed  $f$ , the Rademacher complexity of the latter upper bounds the Rademacher complexity of the former. Thus, we get the following corollary:
Corollary 1. Let  $f: \mathbb{R}^n \to \mathbb{R}$  be a fixed  $L$ -Lipschitz function. Then the Rademacher complexity of  $\mathcal{F}_{B,n,d}^{f,W_0 = 0}$  on  $m$  inputs from  $\{\mathbf{x} \in \mathbb{R}^d : \| \mathbf{x} \| \leq 1\}$  is at most  $\epsilon$ , if  $m \geq \left(\frac{LB}{\epsilon}\right)^{\frac{cL^2B^2}{\epsilon^2}}$  for some universal constant  $c > 0$ .
Comparing the corollary and Theorem 1, we see that the sample complexity of Lipschitz functions composed with matrix linear ones is  $\exp\left(\tilde{\Theta}(L^2 B^2 / \epsilon^2)\right)$  (regardless of whether the Lipschitz function is fixed in advance or not). On the one hand, it implies that the complexity of this class is very large (exponential) as a function of  $L, B$  and  $\epsilon$ . On the other hand, it implies that for any fixed  $L, B, \epsilon$ , it is finite completely independent of the number of parameters. The exponential dependence on the problem parameters is rather unusual, and in sharp contrast to the case of scalar-valued predictors (that is, functions of the form  $\mathbf{x} \mapsto f(\mathbf{w}^\top \mathbf{x})$  where  $||\mathbf{w} - \mathbf{w}_0|| \leq B$  and  $f$  is  $L$ -Lipschitz), for which the sample complexity is just  $O(L^2 B^2 / \epsilon^2)$ .
The key ideas in the proof of Theorem 2 can be roughly sketched as follows: First, we show that due to the Frobenius norm constraints, every function  $\mathbf{x} \mapsto f(W\mathbf{x})$  in our class can be approximated (up to some  $\epsilon$ ) by a function of the form  $\mathbf{x} \mapsto f(\tilde{W}_{\epsilon}\mathbf{x})$ , where the rank of  $\tilde{W}_{\epsilon}$  is at most  $B^{2}/\epsilon^{2}$ . In other words, this approximating function can be written as  $f(UV\mathbf{x})$ , where  $V$  maps to  $\mathbb{R}^{B^{2}/\epsilon^{2}}$ . Equivalently, this can be written as  $g(V\mathbf{x})$ , where  $g(z) = f(Uz)$  over  $\mathbb{R}^{B^{2}/\epsilon^{2}}$ . This reduces the problem to bounding the complexity of the function class which is the composition of all linear functions to  $\mathbb{R}^{B^{2}/\epsilon^{2}}$ , and all Lipschitz functions over  $\mathbb{R}^{B^{2}/\epsilon^{2}}$ , which we perform through covering numbers and the following technical result:
Lemma 1. Let  $\mathcal{F}$  be a class of functions from Euclidean space to  $\{\mathbf{x} \in \mathbb{R}^r : \| \mathbf{x} \| \leq B\}$ . Let  $\Psi_{L,a}$  be the class of all  $L$ -Lipschitz functions from  $\{\mathbf{x} \in \mathbb{R}^r : \| \mathbf{x} \| \leq B\}$  to  $\mathbb{R}$ , which equal some fixed  $a \in \mathbb{R}$  at  $\mathbf{0}$ . Letting  $\Psi_{L,a} \circ \mathcal{F} := \{\psi \circ f : \psi \in \Psi_{L,a}, f \in \mathcal{F}\}$ , its covering number satisfies
$$
\log N (\Psi_ {L, a} \circ \mathcal {F}, d _ {m}, \epsilon) \leq \left(1 + \frac {8 B L}{\epsilon}\right) ^ {r} \cdot \log \frac {8 B L}{\epsilon} + \log N (\mathcal {F}, d _ {m}, \frac {\epsilon}{4 L}).
$$
The proof for this Lemma is an extension of Theorem 4 from Golowich et al. 2018, which considered the case  $r = 1$ . We emphasize that the exponential dependence on  $r$  arises from the covering of the class  $\Psi_{L,a}$ , which is achieved by covering its domain  $\{x \in \mathbb{R}^r : \| x \| \leq B\}$  by a set of points  $N_x$  of size  $(1 + BL / \epsilon)^r$  and covering its range  $[a - LB, a + LB]$  by a set  $N_y$  of size  $2LB / \epsilon$ . As we will show, this implies that the set of all functions from  $N_x$  to  $N_y$  is a cover for  $\Psi_{L,a}$ .
Application to Deep Neural Networks. Theorem 2 which we established above can also be used to study other types of predictor classes. In what follows, we show an application to deep neural networks, establishing a size/dimension-independent sample complexity bound that depends only on the Frobenius norm of the first layer, and the spectral norms of the other layers (albeit exponentially). This is surprising, since all previous bounds of this type we are aware of strongly depend on various norms of all layers, which can be arbitrarily larger than the spectral norm in a size-independent setting
(such as the Frobenius norm), or made strong assumptions on the activation function (e.g., Neyshabur et al. [2015, 2017], Bartlett et al. [2017], Golowich et al. [2018], Du and Lee [2018], Daniely and Granot [2019], Vardi et al. [2022]).
Formally, we consider scalar-valued depth- $k$  "neural networks" of the form
$$
\mathcal {F} _ {k} := \left\{x \rightarrow \mathbf {w} _ {k} f _ {k - 1} \left(W _ {k - 1} f _ {k - 2} (\dots f _ {1} (W _ {1} \mathbf {x}))\right): | | \mathbf {w} _ {k} | | \leq S _ {k},   \forall j   | | W _ {j} | | \leq S _ {j},   | | W _ {1} | | _ {F} \leq B \right\}
$$
where each  $W_{j}$  is a parameter matrix of some arbitrary dimensions,  $\mathbf{w}_{k}$  is a vector and each  $f_{j}$  is some fixed 1-Lipschitz function satisfying  $f_{j}(\mathbf{0}) = 0$ . This is a rather relaxed definition for neural networks, as we do not assume anything about the activation functions  $f_{j}$ , except that it is Lipschitz. To analyze this function class, we consider  $\mathcal{F}_k$  as a subset of the class
$$
\left\{x \rightarrow f (W \mathbf {x}): \| W \| _ {F} \leq B, f: \mathbb {R} ^ {n} \rightarrow \mathbb {R} \text {i s} L \text {- L i p s c h i t z} \right\},
$$
where  $L = \prod_{j=2}^{k} S_{j}$  (as this clearly upper bounds the Lipschitz constant of  $z \mapsto \mathbf{w}_{k} f_{k-1}(W_{k-1} f_{k-2}(\ldots f_{1}(z) \ldots))$ ). By applying Theorem 2 (with the same conditions) we have
Corollary 2. For any  $B, L \geq 1$  and  $\epsilon \in (0,1]$  s.t.  $\frac{B}{\epsilon} \geq 1$ , we have that the Rademacher complexity of  $\mathcal{F}_k$  on  $m$  inputs from  $\{\mathbf{x} \in \mathbb{R}^d : \| \mathbf{x} \| \leq 1\}$  is at most  $\epsilon$ , if
$$
m \geq \left(\frac {L B}{\epsilon}\right) ^ {\frac {c L ^ {2} B ^ {2}}{\epsilon^ {2}}}
$$
where  $L \coloneqq \prod_{j=2}^{k} S_{j}$  and  $c > 0$  is some universal constant.
Of course, the bound has a bad dependence on the norm of the weights, the Lipschitz parameter and  $\epsilon$ . On the other hand, it is finite for any fixed choice of these parameters, fully independent of the network depth, width, nor on any matrix norm other than the spectral norms, and the Frobenius norm of the first layer only. We note that in the size-independent setting, controlling the product of the spectral norms is both necessary and not sufficient for finite sample complexity bounds (see discussion in Vardi et al. [2022]). The bound above is achieved only by controlling in addition the Frobenius norm of the first layer.
3.2 No Finite Sample Complexity with  $W_0 \neq 0$
In subsection 3.1, we showed size-independent sample complexity bounds when the initialization/reference matrix  $W_{0}$  is zero. Therefore, it is natural to ask if it is possible to achieve similar size-independent bounds with non-zero  $W_{0}$ . In this subsection we show that perhaps surprisingly, the answer is negative: Even for very small non-zero  $W_{0}$ , it is impossible to control the sample complexity of  $\mathcal{F}_{B,n,d}^{f,W_0}$  independent of the size/dimension parameters  $d,n$ . Formally, we have the following theorem:
Theorem 3. For any  $m \in \mathbb{N}$  and  $\epsilon \in (0, \frac{1}{4}]$ , there exists  $d = m + 1, n = 2m$ ,  $W_0 \in \mathbb{R}^{n \times d}$  with  $\| W_0 \| = 2\sqrt{2} \cdot \epsilon$  and a function  $f: \mathbb{R}^n \to \mathbb{R}$  which is 1-Lipschitz, for which  $\mathcal{F}_{B=1,n,d}^{f,W_0}$  can shatter  $m$  points from  $\{\mathbf{x} \in \mathbb{R}^d : \| \mathbf{x} \| \leq 1\}$  with margin  $\epsilon$ .
The theorem strengthens the lower bound of Daniely and Granot [2022] and the previous subsection, which only considered the  $W_0 = 0$  case. We emphasize that the result holds already when  $||W_0||$  is very small (equal to  $2\sqrt{2} \cdot \epsilon$ ). Moreover, the proof technique can be used to show a similar result even if we allow for functions Lipschitz w.r.t. the infinity norm (and not just the Euclidean norm as we have done so far), at the cost of a higher required value of  $n$ . This is of interest, since non-element-wise activations used in practice (such as variants of the max function) are Lipschitz with respect to that norm, and some previous work utilized such stronger Lipschitz constraints to obtain sample complexity guarantees (e.g., Daniely and Granot [2019]).
Interestingly, the proof of the theorem is simpler than the  $W_0 = 0$  case, and involves a direct non-probabilistic construction. It can be intuitively described as follows: We choose a fixed set of
2This is without loss of generality, since if  $f_{j}$  is  $L_{j}$ -Lipschitz, we can rescale it by  $1 / L_{j}$  and multiply  $S_{j + 1}$  by  $L_{j}$ .
vectors  $\mathbf{x}_1, \ldots, \mathbf{x}_m$  and a matrix  $W_0$  (essentially the identity matrix with some padding) so that  $W_0\mathbf{x}_i$  encodes the index  $i$ . For any choice of target values  $\mathbf{y} \in \{\pm \epsilon\}^m$ , we define a matrix  $W_{\mathbf{y}}'$  (which is all zeros except the values of a half column that are located in a strategic location), so that  $W_{\mathbf{y}}'\mathbf{x}_i$  encodes the entire vector  $\mathbf{y}$ . Letting  $W_{\mathbf{y}} = W_{\mathbf{y}}' + W_0$ , we get a matrix of bounded distance to  $W_0$ , so that  $W_{\mathbf{y}}\mathbf{x}_i$  encodes both  $i$  and  $\mathbf{y}$ . Thus, we just need  $f$  to be the fixed function that given an encoding for  $\mathbf{y}$  and  $i$ , returns  $y_i$ , hence  $\mathbf{x} \mapsto f(W_{\mathbf{y}}\mathbf{x})$  shatters the set of points.
3.3 Vector-valued Linear Predictors are Learnable without Uniform Convergence
The class  $\mathcal{F}_{B,n,d}^{f,W_0}$ , which we considered in the previous subsection, is closely related to the natural class of matrix-valued linear predictors  $(\mathbf{x}\mapsto W\mathbf{x})$  with bounded Frobenius distance from initialization, composed with some Lipschitz loss function  $\ell$ . We can formally define this class as
$$
\mathcal {G} _ {B, n, d} ^ {\ell , W _ {0}} := \left\{\left(\mathbf {x}, y\right)\rightarrow \ell \left(W \mathbf {x}; y\right): W \in \mathbb {R} ^ {n \times d}, \| W - W _ {0} \| _ {F} \leq B \right\}.
$$
For example, standard multiclass linear predictors fall into this form. Note that when  $y$  is fixed, this is nothing more than the class  $\mathcal{F}_{B,n,d}^{\ell_y,W_0}$  where  $\ell_y(z) = \ell (z;y)$ . The question of learnability here boils down to the question of whether, given an i.i.d. sample  $\{\mathbf{x}_i,y_i\}_{i = 1}^m$  from an unknown distribution, we can approximately minimize  $\mathbb{E}_{(\mathbf{x},y)}[\ell (W\mathbf{x},y)]$  arbitrarily well over all  $W:\| W - W_0\| \leq B$ , provided that  $m$  is large enough.
For multiclass linear predictors, it is natural to consider the case where the loss  $\ell$  is also convex in its first argument. In this case, we can easily establish that the class  $\mathcal{G}_{B,n,d}^{\ell,W_0}$  is learnable with respect to inputs of bounded Euclidean norm, regardless of the size/dimension parameters  $n,d$ . This is because for each  $(\mathbf{x},y)$ , the function  $W \mapsto \ell(W\mathbf{x};y)$  is convex and Lipschitz in  $W$ , and the domain  $\{W:||W-W_0||_F \leq B\}$  is bounded. Therefore, we can approximately minimize  $\mathbb{E}_{(\mathbf{x},y)}[\ell(W\mathbf{x},y)]$  by applying stochastic gradient descent (SGD) over the sequence of examples  $\{\mathbf{x}_i,y_i\}_{i=1}^m$ . This is a consequence of well-known results (see for example Shalev-Shwartz and Ben-David [2014]), and is formalized as follows:
Theorem 4. Suppose that for any  $y$ , the function  $\ell(., y)$  is convex and  $L$ -Lipschitz. For any  $B > 0$  and fixed matrix  $W_0$ , there exists a randomized algorithm (namely stochastic gradient descent) with the following property: For any distribution over  $(\mathbf{x}, y)$  such that  $||\mathbf{x}|| \leq 1$  with probability 1, given an i.i.d. sample  $\{(\mathbf{x}_i, y_i)\}_{i=1}^m$ , the algorithm returns a matrix  $\hat{W}$  such that  $||\hat{W} - W_0||_F \leq B$  and
$$
\mathbb {E} _ {\hat {W}} \left[ \mathbb {E} _ {(\mathbf {x}, y)} [ \ell (\hat {W} \mathbf {x}; y) ] - \min _ {W: \| W - W _ {0} \| _ {F} \leq B} \mathbb {E} _ {(\mathbf {x}, y)} [ \ell (W \mathbf {x}; y) ] \right] \leq \frac {B L}{\sqrt {m}}.
$$
Thus, the number of samples  $m$  required to make the above less than  $\epsilon$  is at most  $\frac{B^2L^2}{\epsilon^2}$ .
Perhaps unexpectedly, we now turn to show that this positive learnability result is not due to uniform convergence: Namely, we can learn this class, but not because the empirical average and expected value of  $\ell(W\mathbf{x};y)$  are close uniformly over all  $W: \| W - W_0\| \leq B$ . Indeed, that would have required that a uniform convergence measure such as the fat-shattering dimension of our class would be bounded. However, this turns out to be false: The class  $\mathcal{G}_{B,n,d}^{\ell,W_0}$  can shatter arbitrarily many points of norm  $\leq 1$ , and at any scale  $\epsilon \leq 1$ , for some small  $W_0$  and provided that  $n,d$  are large enough<sup>3</sup>. In the previous section, we already showed such a result for the class  $\mathcal{F}_{B,n,d}^{f,W_0}$ , which equals  $\mathcal{G}_{B,n,d}^{\ell,W_0}$  when  $y$  is fixed and  $f(W\mathbf{x}) = \ell(W\mathbf{x};y)$ . Thus, it is enough to prove that the same impossibility result (Theorem 3) holds even if  $f$  is a convex function. This is indeed true using a slightly different construction:
Theorem 5. For any  $m \in \mathbb{N}$  and  $\epsilon \in (0, \frac{1}{4}]$ , there exists large enough  $d = \Theta(m)$ ,  $n = \Theta(\exp(m))$ ,  $W_0 \in \mathbb{R}^{n \times d}$  with  $\|W_0\| = 4\sqrt{2} \cdot \epsilon$  and a convex function  $f: \mathbb{R}^n \to \mathbb{R}$  which is 1-Lipschitz with
This precludes uniform convergence, since it implies that for any  $m$ , we can find a set of  $2m$  points  $\{\mathbf{x}_i, y_i\}_{i=1}^{2m}$ , such that if we sample  $m$  points with replacement from a uniform distribution over this set, then there is always some  $W$  in the class so that the average value of  $\ell(W\mathbf{x};y)$  over the sample and in expectation differs by a constant independent of  $m$ . The fact that the fat-shattering dimension is unbounded does not contradict learnability here, since our goal is to minimize the expectation of  $\ell(W\mathbf{x};y)$ , rather than view it as predicted values which are then composed with some other loss.
respect to the infinity norm (and hence also with respect to the Euclidean norm), for which  $\mathcal{F}_{B = 1,n,d}^{f,W_0}$  can shatter  $m$  points from  $\{\mathbf{x}\in \mathbb{R}^d:||\mathbf{x}||\leq 1\}$  with margin  $\epsilon$ .
Overall, we see that the problem of learning vector-valued linear predictors, composed with some convex Lipschitz loss (as defined above), is possible using a certain algorithm, but without having uniform convergence. This connects to a line of work establishing learning problems which are provably learnable without uniform convergence (such as Shalev-Shwartz et al. [2010], Feldman [2016]). However, whether these papers considered synthetic constructions, we consider an arguably more natural class of linear predictors of bounded Frobenius norm, composed with a convex Lipschitz loss over stochastic inputs of bounded Euclidean norm. In any case, this provides another example for when learnability can be achieved without uniform convergence.
4 Neural Networks with Element-Wise Lipschitz Activation
In section 3 we studied the complexity of functions of the form  $\mathbf{x} \mapsto f(W\mathbf{x})$  (or possibly deeper neural networks) where nothing is assumed about  $f$  besides Lipschitz continuity. In this section, we consider more specifically functions which are applied element-wise, as is common in the neural networks literature. Specifically, we will consider the following hypothesis class of scalar-valued, one-hidden-layer neural networks of width  $n$  on inputs in  $\{\mathbf{x} \in \mathbb{R}^d : ||\mathbf{x}|| \leq b_x\}$ , where  $\sigma(\cdot)$  is a Lipschitz function on  $\mathbb{R}$  applied element-wise, and where we only bound the norms as follows:
$$
\mathcal {F} _ {b, B, n, d} ^ {\sigma , W _ {0}} := \left\{\mathbf {x} \rightarrow \mathbf {u} ^ {\top} \sigma (W \mathbf {x}): \mathbf {u} \in \mathbb {R} ^ {n}, W \in \mathbb {R} ^ {n \times d}, | | \mathbf {u} | | \leq b, | | W - W _ {0} | | _ {F} \leq B \right\},
$$
where  $\mathbf{u}^{\top}\sigma (W\mathbf{x}) = \sum_{j}u_{j}\sigma (\mathbf{w}_{j}^{\top}\mathbf{x})$ . We note that we could have also considered a more general version, where  $\mathbf{u}$  is also initialization-dependent: Namely, where the constraint  $||\mathbf{u}||\leq b$  is replaced by  $||\mathbf{u} - \mathbf{u}_0||\leq b$  for some fixed  $\mathbf{u}_0$ . However, this extension is rather trivial, since for vectors  $\mathbf{u}$  there is no distinction between the Frobenius and spectral norms. Thus, to consider  $\mathbf{u}$  in some ball of radius  $b$  around some  $\mathbf{u}_0$ , we might as well consider the function class displayed above with the looser constraint  $||\mathbf{u}||\leq b + ||\mathbf{u}_0||$ . This does not lose much tightness, since such a dependence on  $||\mathbf{u}_0||$  is also necessary (see remark 2 below).
The sample complexity of  $\mathcal{F}_{b,B,n,d}^{\sigma,W_0}$  was first studied in the case of  $W_0 = 0$ , with works such as Neyshabur et al. [2015, 2017], Du and Lee [2018], Golowich et al. [2018], Daniely and Granot [2019] proving bounds for specific families of the activation  $\sigma(\cdot)$  (e.g., homogeneous or quadratic). For general Lipschitz  $\sigma(\cdot)$  and  $W_0 = 0$ , Vardi et al. [2022] proved that the Rademacher complexity of  $\mathcal{F}_{b,B,n,d}^{\sigma,W_0=0}$  for any  $L$ -Lipschitz  $\sigma(\cdot)$  is at most  $\epsilon$ , if the number of samples is  $\tilde{O}\left(\left(\frac{bBb_xL}{\epsilon}\right)^2\right)$ . They left the case of  $W_0 \neq 0$  as an open question. In a recent preprint, Daniely and Granot [2022] used an innovative technique to prove a bound in this case, but not a fully size-independent one (there remains a logarithmic dependence on the network width  $n$  and the input dimension  $d$ ). In what follows, we prove a bound which handles the  $W_0 \neq 0$  case and is fully size-independent, under the assumption that  $\sigma(\cdot)$  is smooth. The proof (which is somewhat involved) involves techniques different from both previous papers, and may be of independent interest.
Theorem 6. Suppose  $\sigma(\cdot)$  (as function on  $\mathbb{R}$ ) is  $L$ -Lipschitz,  $\mu$ -smooth (i.e.,  $\sigma'(\cdot)$  is  $\mu$ -Lipschitz) and  $\sigma(0) = 0$ . Then for any  $b, B, n, D, \epsilon > 0$  such that  $B b_x \geq 2$ , and any  $W_0$  such that  $||W_0|| \leq B_0$ , the Rademacher complexity of  $\mathcal{F}_{b,B,n,d}^{\sigma,W_0}$  on  $m$  inputs from  $\{\mathbf{x} \in \mathbb{R}^d : ||\mathbf{x}|| \leq b_x\}$  is at most  $\epsilon$ , if  $m \geq \frac{1}{\epsilon^2} \cdot \tilde{O}\left((1 + b b_x(LB_0 + (\mu + L)B(1 + B_0 b_x))^2\right)$ .
Thus, we get a sample complexity bounds that depend on the norm parameters  $b, b_x, B_0$ , the Lipschitz parameter  $L$ , and the smoothness parameter  $\mu$ , but is fully independent of the size parameters  $n, d$ . Note that for simplicity, the bound as written above hides some factors logarithmic in  $m, B, L, b_x -$  see the proof in the appendix for the precise expression.
We note that if  $W_0 = 0$ , we can take  $B_0 = 0$  in the bound above, in which case the sample complexity scales as  $\tilde{O} (((\mu + L)bb_xB)^2 /\epsilon^2)$ . This is the same as in Vardi et al. [2022] (see above) up to the dependence on the smoothness parameter  $\mu$ .
Remark 2. The upper bound on Theorem 6 depends quadratically on the spectral norm of  $W_0$  (i.e.,  $B_0$ ). This dependence is necessary in general. Indeed, even by taking the activation function  $\sigma(\cdot)$  to
be the identity,  $B = 0$  and  $b = 1$  we get that our function class contains the class of scalar-valued linear predictors  $\{\mathbf{x} \to \mathbf{v}^\top \mathbf{x} : \mathbf{x}, \mathbf{v} \in \mathbb{R}^d, ||\mathbf{v}|| \leq B_0\}$ . For this class, it is well known that the number of samples should be  $\Theta\left(\frac{B_0^2}{\epsilon^2}\right)$ , to ensure that the Rademacher complexity of that class is at most  $\epsilon$ .
4.1 Bounds for Deep Networks with Lipschitz Activations
As a final contribution, we consider the case of possibly deep neural networks, when  $W_0 = 0$  and the activations are Lipschitz and element-wise. Specifically, given the domain  $\{\mathbf{x} \in \mathbb{R}^d : ||\mathbf{x}|| \leq b_x\}$  in Euclidean space, we consider the class of scalar-valued neural networks of the form
$$
\mathbf {x} \rightarrow \mathbf {w} _ {k} ^ {\top} \sigma_ {k - 1} \left(W _ {k - 1} \sigma_ {k - 2} (\dots \sigma_ {1} ((W _ {1} \mathbf {x}))\right)
$$
where  $\mathbf{w}_k$  is a vector (i.e. the output of the function is in  $\mathbb{R}$ ) with  $||\mathbf{w}_k|| \leq b$ , each  $W_j$  is a parameter matrix s.t.  $||W_j||_F \leq B_j$ ,  $||W_j|| \leq S_j$  and each  $\sigma_j(\cdot)$  (as a function on  $\mathbb{R}$ ) is an  $L$ -Lipschitz function applied element-wise, satisfying  $\sigma_j(0) = 0$ . Let  $\mathcal{F}_{k,\{S_j\},\{B_j\}}^{\{\sigma_j\}}$  be the class of neural networks as above. Vardi et al. [2022] proved a sample complexity guarantee for  $k = 2$  (one-hidden-layer neural networks), and left the case of higher depths as an open problem. The theorem below addresses this problem, using a combination of their technique and a "peeling" argument to reduce the complexity bound of networks of depth  $k$  to those of depth  $(k - 1)$ . The resulting bound is fully independent of the network width (although strongly depends on the network depth), and is the first of this type (to the best of our knowledge) that handles general Lipschitz activations under Frobenius norm constraints.
Theorem 7. For any  $\epsilon, b > 0$ ,  $\{B_j\}_{j=1}^{k-1}, \{S_j\}_{j=1}^{k-1}, L$  with  $S_1, \ldots, S_{k-1}, L \geq 1$ , the Rademacher complexity of  $\mathcal{F}_{k,\{S_j\},\{B_j\}}^{\{\sigma_j\}}$  on  $m$  inputs from  $\{\mathbf{x} \in \mathbb{R}^d : ||\mathbf{x}|| \leq b_x\}$  is at most  $\epsilon$ , if
$$
m \geq \frac {c \left(k L ^ {k - 1} b R _ {k - 2} \log^ {\frac {3 (k - 1)}{2}} (m) \cdot \prod_ {i = 1} ^ {k - 1} B _ {i}\right) ^ {2}}{\epsilon^ {2}},
$$
where  $R_{k-2} = b_x L^{k-2} \prod_{i=1}^{k-2} S_i$ ,  $R_0 = b_x$  and  $c > 0$  is a universal constant.
We note that for  $k = 2$ , this reduces to the bound of Vardi et al. [2022] for one-hidden-layer neural networks.
5 Discussion and Open Problems