SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this paper, we provided several new results on the sample complexity of vector-valued linear predictors and feed-forward neural networks, focusing on size-independent bounds and constraining the distance from some reference matrix. The paper leaves open quite a few avenues for future research. For example, in Section 3, we studied the sample complexity of  $\mathcal{F}_{B,n,d}^{f,W_0}$  when  $n,d$  are unrestricted. Can we get a full picture of the sample complexity when  $n,d$  are also controlled? Even more specifically, can the lower bounds in the section be obtained for any smaller values of  $d,n$ ? As to the results in Section 4, is our Rademacher complexity bound for  $\mathcal{F}_{b,B,n,d}^{\sigma ,W_0}$  (one-hidden-layer networks and smooth activations) the tightest possible, or can it be improved? Also, can we generalize the result to arbitrary Lipschitz activations? In addition, what is the sample complexity of such networks when  $n,d$  are also controlled? In a different direction, it would be very interesting to extend the results of this section to deeper networks and non-zero  $W_0$ .
This research is supported in part by European Research Council (ERC) grant 754705, by the Israeli Council for Higher Education (CHE) via the Weizmann Data Science Research Center, and by a research grant from the Estate of Louise Yasgour.
Martin Anthony and Peter L. Bartlett. Vapnik-Chervonenkis Dimension Bounds for Neural Networks, page 108-130. Cambridge University Press, 1999. doi: 10.1017/CBO9780511624216.009.
A.1 Proof of Observation 1
The first part follows immediately from the definition. For the second part, let  $k \coloneqq N(\mathcal{F}, d, \frac{\epsilon}{2})$  and  $N = \{g_1, \dots, g_k\}$  be an  $\frac{\epsilon}{2}$ -cover for  $\mathcal{F}$ . For every  $g_i \in N$ , let  $f_i \in \mathcal{F}$  be some function such that  $d(f_i, g_i) \leq \frac{\epsilon}{2}$ : there must exist  $f_i$  with this property, otherwise we can remove  $g_i$  from  $N$  and still get an  $\frac{\epsilon}{2}$ -cover for  $\mathcal{F}$ , contradicting the minimality property of the covering number. For every  $f \in \mathcal{F}$ , let  $g_i \in N$  be the function with  $d(g_i, f) \leq \frac{\epsilon}{2}$ , then by the triangle inequality  $d(f_i, f) \leq d(f_i, g_i) + d(g_i, f) \leq \epsilon$ . Therefore  $\{f_1, \dots, f_k\}$  is a proper  $\epsilon$ -cover for  $\mathcal{F}$ .
We use the following lemmas from Daniely and Granot [2022]:
Lemma 2. For  $d \geq 20$ , there exists a set of vectors  $\mathbf{x}_1, \ldots, \mathbf{x}_m \in \mathbb{R}^d$  and a set of matrices  $W_1, \ldots, W_{2^m} \in \mathbb{R}^{n \times d}$  with  $n = \Theta(\exp(d))$  that have the following properties:
1.  $||\mathbf{x}_i|| = 1$  , for each  $i\in [m]$
2.  $m = \frac{n}{10}$
3.  $||W_s||_F^2 \leq 2d$ , for each  $s \in [2^m]$
4.  $||W_{s}\mathbf{x}_{i} - W_{t}\mathbf{x}_{j}||\geq \frac{1}{4},$  for each  $i,j\in [m]$  and  $s,t\in [2^m ]$  s.t.  $(s,i)\neq (t,j)$
Lemma 3. Let  $\mathbf{x}_1, \ldots, \mathbf{x}_m$  be a finite set of different points in some metric space  $(\mathcal{X}, d)$ , such that for each  $i \neq j \in [m]$ ,  $d(\mathbf{x}_i, \mathbf{x}_j) \geq \alpha$ . Let further be  $p_1, \ldots, p_m \in \mathbb{R}$  any set of points. Then there exists an  $L$ -Lipschitz function,  $f: \mathcal{X} \to \mathbb{R}$ , where
$$
L = \frac{2}{\alpha}\min_{C\geq 0}\max_{i\in [m]}|p_{i} - C|,
$$
such that for each  $i\in [m]$ $f(\mathbf{x}_i) = p_i$
Proof of Theorem 1. Let  $B \geq 1$ ,  $\epsilon \leq 1$ , and  $d$  that will be chosen later. Let  $\mathbf{x}_1, \ldots, \mathbf{x}_m \in \mathbb{R}^d$  and  $W_1', \ldots, W_{2^m}' \in \mathbb{R}^{n \times d}$  be the vectors and metrics that are defined in Lemma 2. Note that  $n = \Theta(e^d)$  and  $m = \Theta(n)$ .
Let  $W_{s} = \frac{8\epsilon}{L} \cdot W_{s}^{\prime}$  for each  $s \in [2^{m}]$ . By property 3 from that Lemma we have that  $||W_s||_F^2 \leq 128d\epsilon^2 / L^2$ , for each  $s \in [2^{m}]$ . Thus, by setting  $d := \lfloor \frac{L^2B^2}{128\epsilon^2} \rfloor$ , we get that  $||W_s||_F \leq B$ . Moreover, we have that  $m = \Theta(n) = 2^{\Theta(L^2B^2/\epsilon^2)}$ . By property 4 from that lemma, we have that the set  $Q = \{W_s\mathbf{x}_i : i \in [m], s \in [2^m]\}$  contains  $m2^m$  different elements such that for each pair  $W_s\mathbf{x}_i \neq W_t\mathbf{x}_j$  we have
$$
| | W _ {s} \mathbf {x} _ {i} - W _ {t} \mathbf {x} _ {j} | | = \frac {8 \epsilon}{L} \cdot | | W _ {s} ^ {\prime} \mathbf {x} _ {i} - W _ {t} ^ {\prime} \mathbf {x} _ {j} | | \geq \frac {2 \epsilon}{L}.
$$
Order the elements of the set  $2^{[m]}$  as  $S_{1},\ldots ,S_{2^{m}}$  in some arbitrary order, and define the function  $g:[m]\times [2^m ]\to \{\pm \epsilon \}$  as:
$$
g (k, i) = \left\{ \begin{array}{l l} \epsilon & i \in S _ {k} \\ - \epsilon & i \notin S _ {k} \end{array} \right..
$$
Apply Lemma 3 with the Euclidean metric space, to get an  $L$ -Lipschitz function,  $f: \mathbb{R}^n \to \mathbb{R}$ , such that for all  $i \in [m]$ ,  $s \in [2^m]$ ,
$$
f \left(W _ {s} \mathbf {x} _ {i}\right) = g (k, i).
$$
Moreover, we have that  $\mathbf{x} \to f(W_s\mathbf{x})$  belongs to  $\mathcal{F}_{B,n,d}^{f,W_0 = 0}$  for each  $s \in [2^m]$ . Therefore,  $\mathcal{F}_{B,n,d}^{f,W_0 = 0}$  can shatter  $m = \Theta(n) = \exp\left(\Theta(L^2B^2/\epsilon^2)\right)$  points from  $\{\mathbf{x} \in \mathbb{R}^d : ||\mathbf{x}|| \leq 1\}$  with margin  $\epsilon$ .
Define the  $L_{\infty}$  distance
$$
d _ {\infty} (g, g ^ {\prime}) = \sup  _ {z \in \mathcal {Z}} \| f (z) - f ^ {\prime} (z) \|,
$$
where  $\mathcal{Z}$  is the domain of  $g$  and  $g'$ . Let  $\mathcal{U}_B = \{\mathbf{x} \in \mathbb{R}^r : ||\mathbf{x}|| \leq B\}$  and fix some  $\epsilon > 0$ . By Observation 2 there exists a set  $N_x \subseteq \mathbb{R}^r$  with  $|N_x| = (1 + \frac{8BL}{\epsilon})^r$  s.t. for every  $\mathbf{x} \in \mathcal{U}_B$ , there exists  $\mathbf{x}' \in N_x$  with
$$
\left| \left| \mathbf {x} - \mathbf {x} ^ {\prime} \right| \right| \leq \frac {\epsilon}{4 L}.
$$
Note that  $\psi (\mathbf{x})\in [a - BL,a + BL]$  for any  $\psi \in \Psi_{L,a},\mathbf{x}\in \mathcal{U}_B$  . For simplicity, we assume that  $a = 0$  (the proof for any other  $a$  is essentially the same). Let  $N_{y}\coloneqq \{0,\pm \frac{\epsilon}{4},\pm \frac{\epsilon}{2},\ldots ,\pm \lfloor BL / 4\epsilon \rfloor 4\epsilon \}$  be  $\frac{\epsilon}{4}$  -cover for the closed interval  $[-BL,BL]$  . Given any  $\psi \in \Psi_{L,a = 0}$  and  $\mathbf{x}\in \mathcal{U}_B$  , construct  $\psi^{\prime}:N_{x}\to N_{y}$  as follows: let  $\mathbf{x}^{\prime}$  be the point in  $N_{x}$  that is nearest to  $\mathbf{x}$  , and let  $\psi^{\prime}(\mathbf{x})$  be the point in  $N_{y}$  that is nearest to  $\psi (\mathbf{x}^{\prime})$  . We get that
$$
\begin{array}{l} \left| \psi (\mathbf {x}) - \psi^ {\prime} (\mathbf {x}) \right| \leq \left| \psi (\mathbf {x}) - \psi \left(\mathbf {x} ^ {\prime}\right) \right| + \left| \psi \left(\mathbf {x} ^ {\prime}\right) - \psi^ {\prime} \left(\mathbf {x} ^ {\prime}\right) \right| + \left| \psi^ {\prime} \left(\mathbf {x} ^ {\prime}\right) - \psi^ {\prime} (\mathbf {x}) \right| \\ \leq L \cdot | | \mathbf {x} - \mathbf {x} ^ {\prime} | | + \frac {\epsilon}{4} + 0 \leq \frac {\epsilon}{2}. \\ \end{array}
$$
The number of such functions is at most
$$
\left| N _ {y} \right| ^ {\left| N _ {x} \right|},
$$
$$
\log N \left(\Psi_ {L, a = 0}, d _ {\infty}, \frac {\epsilon}{2}\right) = | N _ {x} | \log | N _ {y} | = \left(1 + \frac {8 B L}{\epsilon}\right) ^ {r} \cdot \log \frac {8 B L}{\epsilon}.
$$
Next, we argue that
$$
\log N (\Psi_ {L, a = 0} \circ \mathcal {F}, d _ {m}, \epsilon) \leq \log N (\Psi_ {L, a = 0}, d _ {\infty}, \frac {\epsilon}{2}) + \log N (\mathcal {F}, d _ {m}, \frac {\epsilon}{4 L}),
$$
from which the result will follow. To see this, pick any  $\psi \in \Psi_{L,a = 0}$  and  $g\in \mathcal{F}$ . By observation 1 there exists a proper  $\frac{\epsilon}{2L}$ -cover for  $\mathcal{F}$  of size  $N(\mathcal{F},d_m,\frac{\epsilon}{4L})$ . Let  $\psi^{\prime},g^{\prime}$  be the respective closest functions in the cover of  $\Psi_{L,a = 0}$  and the proper cover of  $\mathcal{F}$  (at scale  $\frac{\epsilon}{2}$  and  $\frac{\epsilon}{2L}$  respectively). Since  $g^{\prime}$  belongs to proper cover e.g.  $g^{\prime}\in \mathcal{F}$ , its range is  $\mathcal{U}_B$ . By the triangle inequality and since  $\psi$  is  $L$ -Lipschitz, we have
$$
\begin{array}{l} d _ {m} (\psi g, \psi^ {\prime} g ^ {\prime}) \leq d _ {m} (\psi g, \psi g ^ {\prime}) + d _ {m} (\psi g ^ {\prime}, \psi^ {\prime} g ^ {\prime}) \leq d _ {m} (\psi g, \psi g ^ {\prime}) + d _ {\infty} (\psi g ^ {\prime}, \psi^ {\prime} g ^ {\prime}) \leq \\ L \cdot d _ {m} (g, g ^ {\prime}) + d _ {\infty} (\psi g ^ {\prime}, \psi^ {\prime} g ^ {\prime}) \leq \frac {\epsilon}{2} + \frac {\epsilon}{2} = \epsilon \\ \end{array}
$$
A.4 Proof of Theorem 2
Observation 2. Let  $\mathcal{U}_B\coloneqq \{\mathbf{x}\in \mathbb{R}^r:||\mathbf{x}||\leq B\}$  and  $\epsilon >0$  . There is a set  $N\subseteq \mathcal{U}_B$  with  $(B / \epsilon)^r\leq |N|\leq \left(1 + \frac{2B}{\epsilon}\right)^r$  such that:
1. For every  $\mathbf{x} \in \mathcal{U}_B$ , there exists a  $\mathbf{x}' \in N$  with  $||\mathbf{x} - \mathbf{x}'|| \leq \epsilon$  ( $N$  is an  $\epsilon$ -cover for  $\mathcal{U}_B$ ).
2. For every  $\mathbf{x},\mathbf{y}\in N$  , we have that  $\| \mathbf{x} - \mathbf{y}\| \geq \epsilon$  ( $N$  is an  $\epsilon$  -packing for  $\mathcal{U}_B$ ).
Proof. This is a well-known volume argument. Let  $\mathcal{U}_B \coloneqq \{\mathbf{x} \in \mathbb{R}^r : ||\mathbf{x}|| \leq B\}$  and fix some  $\epsilon > 0$ . Choose  $N$  to be a maximal  $\epsilon$ -packing ( $\epsilon$ -separated) subset of  $\mathcal{U}_B$ . In other words,  $N \subseteq \mathcal{U}_B$  is such that  $||\mathbf{x} - \mathbf{y}|| \geq \epsilon$  for all  $\mathbf{x}, \mathbf{y} \in N$ ,  $\mathbf{x} \neq \mathbf{y}$  and no subset of  $\mathcal{U}_B$  has this property. The maximality property implies that for every  $\mathbf{x} \in \mathcal{U}_B$ , there exists  $\mathbf{x}' \in N$  with  $||\mathbf{x} - \mathbf{x}'|| \leq \epsilon$  (i.e.  $N$  is an  $\epsilon$ -cover for  $\mathcal{U}_B$ ). Otherwise there would exist  $\mathbf{x} \in \mathcal{U}_B$  that is at least  $\epsilon$ -far from all points in  $N$ . Thus  $N \cup \{\mathbf{x}\}$  would still be an  $\epsilon$ -separated set, contradicting the maximality property. Moreover, the separation property implies via the triangle inequality that the balls of radius  $\frac{\epsilon}{2}$  centered at the points in  $N$  are disjoint(up to null set). On the other hand, all such balls lie in  $(B + \epsilon)B_2^r$  where  $B_2^r$  denotes the unit Euclidean
ball centered at the origin. Comparing the volume gives  $vol\left(\frac{\epsilon}{2} B_2^r\right) \cdot |N| \leq vol\left((B + \frac{\epsilon}{2}) B_2^r\right)$  and  $|N| \cdot vol(\epsilon B_2^r) \geq vol(B \cdot B_2^r)$ . Since  $vol(cB_2^r) = c^r vol(B_2^r)$  for all  $c \geq 0$  we have
$$
| N | \leq \frac {v o l \left(\left(B + \frac {\epsilon}{2}\right) B _ {2} ^ {r}\right)}{v o l \left(\frac {\epsilon}{2} B _ {2} ^ {r}\right)} = \left(\frac {B + \frac {\epsilon}{2}}{\frac {\epsilon}{2}}\right) ^ {r},
$$
$$
| N | \geq \frac {v o l (B \cdot B _ {2} ^ {r})}{v o l (\epsilon \cdot B _ {2} ^ {r})} = \left(\frac {B}{\epsilon}\right) ^ {r}.
$$
We conclude that  $\left(\frac{B}{\epsilon}\right)^r \leq |N| \leq (1 + \frac{2B}{\epsilon})^r$ , as required.
The next lemma is shown in Corollary 9 in Kakade et al..
Lemma 4. Let  $\mathcal{W} = \{\mathbf{x}\to \langle \mathbf{w},\mathbf{x}\rangle :\| \mathbf{w}\| \leq B\}$  be the class of norm-bounded linear predictors over inputs from  $\{\mathbf{x}:\| \mathbf{x}\| \leq 1\}$ . Then for every  $\epsilon >0$
$$
\log N (\mathcal {W}, d _ {m}, \epsilon) \leq \frac {c B ^ {2}}{\epsilon^ {2}},
$$
for some universal constant  $c > 0$ .
Lemma 5. Let
$$
\mathcal {F} = \left\{\mathbf {x} \rightarrow W \mathbf {x}: W \in \mathbb {R} ^ {r \times d}, | | W | | _ {F} \leq B \right\}
$$
be class of functions over inputs from  $\{\mathbf{x}:\| \mathbf{x}\| \leq 1\}$ . Then for every  $\epsilon >0$
$$
\log N (\mathcal {F}, d _ {m}, \epsilon) \leq \frac {c r ^ {2} B ^ {2}}{\epsilon^ {2}},
$$
Proof. Applying Lemma 4 with  $\epsilon' = \frac{\epsilon}{\sqrt{r}}$ , we get a class of functions  $N_w$  with  $|N_w| = 2^{crB^2/\epsilon^2}$  such that for every function  $f_w \in \mathcal{W} := \{\mathbf{x} \to \mathbf{w}^\top \mathbf{x} : \| \mathbf{w} \| \leq B\}$ , there exists  $f_w' \in N_w$  with
$$
d _ {m} (f _ {w}, f _ {w} ^ {\prime}) \leq \frac {\epsilon}{\sqrt {r}}.
$$
Now we are ready to define a cover for  $\mathcal{F}$ . Let
$$
N = \left\{\left(\mathbf {x} _ {1}, \dots , \mathbf {x} _ {r}\right)\rightarrow \left(f _ {1} ^ {\prime} \left(\mathbf {x} _ {1}\right), \dots , f _ {r} ^ {\prime} \left(\mathbf {x} _ {r}\right)\right): f _ {j} ^ {\prime} \in N _ {w} \right\}.
$$
Let  $f\in \mathcal{F}$  be  $f(\mathbf{x}) = W\mathbf{x}$ . Then
$$
f (\mathbf {x}) = \left(\mathbf {w} _ {1} ^ {\top} \mathbf {x}, \dots , \mathbf {w} _ {r} ^ {\top} \mathbf {x}\right),
$$
where  $\mathbf{w}_j$  is the  $j$ -th row of  $W$ . Define  $f_j(\mathbf{x}) \coloneqq \mathbf{w}_j^\top \mathbf{x}$ . Since  $||W||_F \leq B$ , we have that  $||\mathbf{w}_j|| \leq B$ , in particular  $f_j \in \mathcal{W}$ . Remember that  $N_w$  is cover for  $\mathcal{W}$ , so there exists  $f_j' \in N_w$  s.t.
$$
d _ {m} \left(f _ {j}, f _ {j} ^ {\prime}\right) \leq \frac {\epsilon}{\sqrt {r}}.
$$
Let  $f' \in N$  be the function
$$
f ^ {\prime} (\mathbf {x}) := \left(f _ {1} ^ {\prime} (\mathbf {x}),..., f _ {r} ^ {\prime} (\mathbf {x})\right).
$$
$$
\begin{array}{l} d _ {m} (f, f ^ {\prime}) ^ {2} = \frac {1}{m} \sum_ {i = 1} ^ {m} | | f (\mathbf {x} _ {i}) - f ^ {\prime} (\mathbf {x} _ {i}) | | ^ {2} = \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {r} | | f _ {j} (\mathbf {x} _ {i}) - f _ {j} ^ {\prime} (\mathbf {x} _ {i}) | | ^ {2} = \sum_ {j = 1} ^ {r} \sum_ {i = 1} ^ {m} | | f _ {j} (\mathbf {x} _ {i}) - f _ {j} ^ {\prime} (\mathbf {x} _ {i}) | | ^ {2} \\ = \sum_ {j = 1} ^ {r} d _ {m} \left(f _ {j}, f _ {j} ^ {\prime}\right) ^ {2} \leq \epsilon^ {2}. \\ \end{array}
$$
Therefore,  $N$  is an  $\epsilon$ -cover for  $\mathcal{F}$  with
$$
| N | = | N _ {w} | ^ {r} = \left(2 ^ {\frac {c r B ^ {2}}{\epsilon^ {2}}}\right) ^ {r} = 2 ^ {\frac {c r ^ {2} B ^ {2}}{\epsilon^ {2}}}
$$
Proof of Theorem 2. Assume for now that  $L = 1$  and let  $W \in \mathbb{R}^{n \times d}$  be a matrix such that  $||W||_F \leq B$ . The singular value decomposition of  $W$  is a factorization of the form  $W = USV^\top$ , where  $U \in \mathbb{R}^{n \times n}$  and  $V \in \mathbb{R}^{d \times d}$  are orthogonal matrices and  $S \in \mathbb{R}^{n \times d}$  is a diagonal matrix with non-negative real numbers on the diagonal. We also can assume W.L.O.G that the upper-left submatrix of  $S$  is of the form  $diag(\sigma_1, \dots, \sigma_{\min\{d,n\}})$  and that  $\sigma_1 \geq \dots \geq \sigma_{\min\{d,n\}}$ . Let  $\tilde{W}_\epsilon := U\tilde{S}_\epsilon V^\top$  be a low-rank approximation to  $W$ . Where,  $\tilde{S}_\epsilon$  is defined by setting all the elements in  $S$  that are  $\leq \epsilon$  to zero. Formally, let  $r \in [\min\{n,d\}]$  be the largest number such that  $\sigma_r > \epsilon$  (or 0 if no such number exists) and partition  $U, S$  and  $V$  as follows:
$$
U = \left[ \begin{array}{c c} U _ {1} & U _ {2} \end{array} \right], S = \left[ \begin{array}{c c} S _ {1} & 0 \\ 0 & S _ {2} \end{array} \right], V = \left[ \begin{array}{c c} V _ {1} & V _ {2} \end{array} \right]
$$
where  $U_{1}\in \mathbb{R}^{n\times r},S_{1} = diag(\sigma_{1},\dots,\sigma_{r})\in \mathbb{R}^{r\times r}$  and  $V_{1}\in \mathbb{R}^{r\times d}$ . We define
$$
\tilde {S} _ {\epsilon} := \left[ \begin{array}{c c} S _ {1} & 0 \\ 0 & 0 \end{array} \right],
$$
which means that  $\tilde{W}_{\epsilon} = U_{1}\tilde{S}_{\epsilon}V_{1}^{\top}$ . The proof strategy consists of two parts: First, we argue that  $\tilde{W}_{\epsilon}$  approximates  $W$  in spectral norm. Second, we argue that  $\tilde{W}_{\epsilon}$  has a low rank, so it is enough to find a small cover for the class of functions  $\Psi_{L=1,a,r} \circ \mathcal{W}_{B,r}$  where  $r := \text{rank}(\tilde{W}_{\epsilon})$ , to get a small cover for  $\Psi_{L,a,n} \circ \mathcal{W}_{B,n}$ . Details follow. Let  $\mathbf{x} \in \mathbb{R}^{d}$  such that  $||\mathbf{x}|| \leq 1$ . Since  $U$  is an orthogonal matrix, we have that  $||U\mathbf{x}|| = ||\mathbf{x}|| \leq 1$ . Moreover, since the spectral norm of a matrix is equal to the largest singular value
$$
\left| \left| S - \tilde {S} _ {\epsilon} \right| \right| = \left| \left| S _ {2} \right| \right| = \sigma_ {r + 1} \leq \epsilon .
$$
Altogether we have
$$
\left\| W \mathbf {x} - \tilde {W} _ {\epsilon} \mathbf {x} \right\| = \left\| U S V ^ {\top} \mathbf {x} - U \tilde {S} _ {\epsilon} V ^ {\top} \mathbf {x} \right\| = \left\| S V ^ {\top} \mathbf {x} - \tilde {S} _ {\epsilon} V ^ {\top} \mathbf {x} \right\| \leq \left\| S - \tilde {S} _ {\epsilon} \right\| \cdot \left\| V ^ {\top} \mathbf {x} \right\| \leq \epsilon
$$
which means that  $\tilde{W}_{\epsilon}$  indeed approximates  $W$ , in the sense of the spectral norm. Moreover,
$$
B ^ {2} \geq | | W | | _ {F} ^ {2} = \sum_ {i = 1} ^ {\min  \{d, n \}} \sigma_ {i} ^ {2} \geq \sum_ {i = 1} ^ {r} \sigma_ {i} ^ {2},
$$
and since  $\sigma_i^2 \geq \epsilon^2$  for each  $i \leq r$ , we have that  $r \leq \frac{B^2}{\epsilon^2}$ . Thus, we have that  $\operatorname{Rank}(\tilde{W}_{\epsilon}) = r \leq \frac{B^2}{\epsilon^2}$ . If we approximate  $W$  up to  $\epsilon$ , then we can argue that for each  $f \in \Psi_{L=1,a,n}$
$$
\left| f (W \mathbf {x}) - f (\tilde {W} _ {\epsilon} \mathbf {x}) \right| \leq \left| \left| W \mathbf {x} - \tilde {W} _ {\epsilon} \mathbf {x} \right| \right| \leq \epsilon .
$$
$$
\tilde {\mathcal {W}} _ {\epsilon} := \left\{\tilde {W} _ {\epsilon}: | | W | | _ {F} \leq B, W \in \mathbb {R} ^ {n \times d} \right\},
$$
then we have by the triangle inequality that
$$
\log N \left(\Psi_ {L = 1, a, n} \circ \mathcal {W}, d _ {m}, 2 \epsilon\right) \leq \log N \left(\Psi_ {L = 1, a, n} \circ \tilde {\mathcal {W}} _ {\epsilon}, d _ {m}, \epsilon\right). \tag {2}
$$
Therefore, we now turn to find a small cover for
$$
\Psi_ {L = 1, a, n} \circ \tilde {\mathcal {W}} _ {\epsilon} \subseteq
$$
$$
\left\{\mathbf {x} \rightarrow f (W \mathbf {x}): \mathbf {x} \in \mathbb {R} ^ {d}, W \in \mathbb {R} ^ {n \times d}, | | W | | _ {F} \leq B, R a n k (W) \leq r, r = \frac {B ^ {2}}{\epsilon^ {2}}, f \in \Psi_ {L = 1, a, n} \right\}.
$$
Remember that by the singular value decomposition, if  $\text{Rank}(W) = r$ , then  $W = USV^{\top}$  where  $U \in \mathbb{R}^{n \times r}$ ,  $S = \text{diag}(\sigma_1, \dots, \sigma_r) \in \mathbb{R}^{r \times r}$  and  $V \in \mathbb{R}^{r \times d}$ . Also,  $U$  and  $V$  are orthogonal matrices, and  $||S||_F \leq B$ . Thus, the class from above is equal to
$$
\left\{\mathbf {x} \rightarrow f (U S V \mathbf {x}): U, V \text {a r e o r t h o g o n a l}, S = d i a g \left(\sigma_ {1}, \dots , \sigma_ {r}\right), \| S \| _ {F} \leq B, r = \frac {B ^ {2}}{\epsilon^ {2}}, f \in \Psi_ {L = 1, a, n} \right\}. \tag {3}
$$
Now we want to get rid of  $U$ . Observe that
$$
\left\{U \mathbf {x} \rightarrow f (U \mathbf {x}): U \text {i s o r t h o g o n a l ,} f \in \Psi_ {L = 1, a, n} \right\} \subseteq \Psi_ {L = 1, a, r},
$$
where we remind that  $\Psi_{L = 1,a,r}$  is the class of all 1-Lipschitz functions from  $\{\mathbf{x} \in \mathbb{R}^r : ||\mathbf{x}|| \leq B\}$  to  $\mathbb{R}$ , such that  $f(\mathbf{0}) = a$  for some fixed  $a \in \mathbb{R}$ . Moreover,
$$
\begin{array}{l} \left\{\mathbf {x} \rightarrow S V ^ {\top} \mathbf {x}: V \text {i s o r t h o g o n a l}, S = d i a g \left(\sigma_ {1}, \dots , \sigma_ {r}\right), | | S | | _ {F} \leq B, r = \frac {B ^ {2}}{\epsilon^ {2}} \right\} \\ \subseteq \left\{\mathbf {x} \to (\sigma_ {1} ^ {\top} \mathbf {v} _ {1} ^ {\top} \mathbf {x}, \dots , \sigma_ {r} \mathbf {v} _ {r} ^ {\top} \mathbf {x}): \mathbf {v} _ {i} \in \mathbb {R} ^ {d}, \sigma_ {i} \in \mathbb {R}, | | \mathbf {v} _ {i} | | = 1, \sum_ {i = 1} ^ {r} \sigma_ {i} ^ {2} = B ^ {2}, r = \frac {B ^ {2}}{\epsilon^ {2}} \right\}, \\ \end{array}
$$
where  $\mathbf{v}_i$  is the  $i$ -th column of  $V$ . Combining these observations, we have that the class of functions defined in Equation 3 is a subset of
$$
\left\{\mathbf {x} \rightarrow f \left(\mathbf {w} _ {1} ^ {\top} \mathbf {x}, \dots , \mathbf {w} _ {r} ^ {\top} \mathbf {x}\right): f \in \Psi_ {L = 1, a, r}, \sum_ {i = 1} ^ {r} \left\| \mathbf {w} _ {i} \right\| ^ {2} = B ^ {2}, r = \frac {B ^ {2}}{\epsilon^ {2}} \right\} = \Psi_ {L = 1, a, r} \circ \mathcal {W} _ {B, r}. \tag {4}
$$
This class is a composition of all linear functions from  $\mathbb{R}^d$  to  $\mathbb{R}^r$  of Frobenius norm at most  $B$ , and all 1-Lipschitz functions. The covering number of such linear functions analyzed in Lemma 5, and the covering number of such composed classes analyzed in Lemma 1. Altogether, we have that the covering number of the class in Eq. 4 is upper bound by
$$
\begin{array}{l} \log N (\Psi_ {L = 1, a, r} \circ \mathcal {W} _ {B, r}, d _ {m}, \epsilon) \leq \left(1 + \frac {8 B}{\epsilon}\right) ^ {r} \cdot \log \left(\frac {8 B}{\epsilon}\right) + \log N (\mathcal {W} _ {B, r}, m, \frac {\epsilon}{4}) \\ = \left(1 + \frac {8 B}{\epsilon}\right) ^ {r} \cdot \log \left(\frac {8 B}{\epsilon}\right) + \frac {c r ^ {2} B ^ {2}}{\epsilon^ {2}}, \\ \end{array}
$$
for some universal constant  $c > 0$  and  $r = \frac{B^2}{\epsilon^2}$ . From this point,  $c > 0$  represents some universal constant that may change from line to line. Combining with Eq. 2 and the assumption that  $\frac{B}{\epsilon} \geq 1$ , we have
$$
\log N \big (\Psi_ {L = 1, a, n} \circ \mathcal {W}, d _ {m}, \epsilon \big) \leq \left(\frac {B}{\epsilon}\right) ^ {\frac {c B ^ {2}}{\epsilon^ {2}}}.
$$
Observe that  $c \cdot \Psi_{L = 1,a,n} = \Psi_{L = c,ca,n}$  for  $c > 0$ , and hence it is easy to verify that  $N(\Psi_{L = 1,a,n} \circ \mathcal{W}, d_m, \epsilon) = N(\Psi_{L = c,a,n} \circ \mathcal{W}, d_m, \epsilon / c)$ . Therefore, for general  $L$ -Lipschitz functions we have
$$
\log N (\Psi_ {L, a, n} \circ \mathcal {W}, d _ {m}, \epsilon) \leq \left(\frac {L B}{\epsilon}\right) ^ {\frac {c L ^ {2} B ^ {2}}{\epsilon^ {2}}}.
$$
To convert the upper bound on the covering number to an upper bound on the Rademacher complexity, we turn to use the Dudley integral covering number bound (see Srebro and Sridharan). In particular, since  $g(\mathbf{x}) \leq LB$  for each  $g \in \Psi_{L,a,n} \circ \mathcal{W}_{B,n}$  and  $\mathbf{x} \in \mathbb{R}^d$  with  $\| \mathbf{x} \| \leq 1$ , we have
$$
R _ {m} \left(\Psi_ {L, a, n} \circ \mathcal {W} _ {B, n}\right) \leq \inf  _ {\epsilon \geq 0} \left\{4 \epsilon + \frac {1 2}{\sqrt {m}} \int_ {\epsilon} ^ {L B} \sqrt {\log N \left(\Psi_ {L , a , n} \circ \mathcal {W} _ {B , n} , d _ {m} , \tau\right)} d \tau \right\} \leq
$$
$$
\inf  _ {\epsilon \geq 0} \left\{4 \epsilon + \frac {1 2 L B}{\sqrt {m}} \sqrt {\log N \left(\Psi_ {L , a , n} \circ \mathcal {W} _ {B , n} , d _ {m} , \epsilon\right)} \right\} \leq \inf  _ {\epsilon \geq 0} \left\{\frac {\epsilon}{2} + \frac {1 2 L B}{\sqrt {m}} \sqrt {\log N \left(\Psi_ {L , a , n} \circ \mathcal {W} _ {B , n} , d _ {m} , \frac {\epsilon}{8}\right)} \right\} \leq
$$
$$
\inf  _ {\epsilon \in [ 0, 1 ]} \left\{\frac {\epsilon}{2} + \frac {1 2 L B}{\sqrt {m}} \sqrt {\left(\frac {L B}{\epsilon}\right) ^ {\frac {c L ^ {2} B ^ {2}}{\epsilon^ {2}}}} \right\}. \tag {5}
$$
Moreover, there exists a universal constant  $c' > 0$ , that for any  $\epsilon \in [0,1]$ , if  $m \geq \left(\frac{LB}{\epsilon}\right)^{\frac{c'L^2B^2}{\epsilon^2}}$ , then
$$
\frac {1 2 L B}{\sqrt {m}} \sqrt {\left(\frac {L B}{\epsilon}\right) ^ {\frac {c L ^ {2} B ^ {2}}{\epsilon^ {2}}}} \leq \frac {\epsilon}{2}.
$$