SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Combining with Eq. 5, the Rademacher complexity of  $\Psi_{L,a,n} \circ \mathcal{W}_{B,n}$  on  $m$  inputs from  $\{\mathbf{x} \in \mathbb{R}^d : \| \mathbf{x} \| \leq 1\}$  is at most  $\epsilon$ , if  $m \geq \left(\frac{LB}{\epsilon}\right)^{\frac{c'L^2B^2}{\epsilon^2}}$ .
A.5 Proof of Theorem 3
Let  $\mathbf{e}_i^d$  denote the indicator vector in  $\mathbb{R}^d$  with value one in the  $i$ -th coordinate, and value zero in the other coordinates. If  $d$  is clear from the context, we just use  $\mathbf{e}_i$  for simplicity. Let  $d = m + 1$  and  $n = 2m$ . Let  $\mathbf{x}_1, \ldots, \mathbf{x}_m \in \{0,1\}^d$  be defined by
$$
\mathbf {x} _ {i} = \frac {1}{\sqrt {2}} \cdot \left(\mathbf {e} _ {i} ^ {d} + \mathbf {e} _ {m + 1} ^ {d}\right).
$$
Let  $\epsilon \in (0,\frac{1}{4} ]$  and define
$$
W _ {0} = 2 \sqrt {2} \cdot \epsilon \cdot \left[ \begin{array}{c c} I _ {m \times m} & 0 _ {m \times 1} \\ 0 _ {m \times (m + 1)} \end{array} \right] \in \mathbb {R} ^ {n \times d}.
$$
By observation 2 (part 2, with  $B = 1, \epsilon = 1/2$ ), there exists  $\mathbf{z}_1, \dots, \mathbf{z}_{2^m} \in \mathbb{R}^m$  such that  $\| z_i \| \leq 1$ ,  $\| z_i - z_j \| \geq 1/2$  for any  $i, j \in [2^m]$ ,  $i \neq j$ . For any  $\mathbf{y} \in \{\pm \epsilon\}^m$ , we associate a different number from  $[2^m]$  and we denote this number by  $y$ . For any  $y \in [2^m]$  define
$$
W _ {y} ^ {\prime} = \left[ \begin{array}{c c} 0 _ {n \times m} & 0 _ {m \times 1} \\ & \mathbf {z} _ {y} \end{array} \right] \in \mathbb {R} ^ {n \times d},
$$
where  $\mathbf{z}_y\in \{0,1\} ^m$  is a column vector. Note that
$$
W _ {0} \mathbf {x} _ {i} = 2 \epsilon \cdot \left[ \begin{array}{c} e _ {i} ^ {d} \\ 0 _ {m \times 1} \end{array} \right] \in R ^ {n}, W _ {y} ^ {\prime} \mathbf {x} _ {i} = \frac {1}{\sqrt {2}} \cdot \left[ \begin{array}{c} 0 _ {m \times 1} \\ \mathbf {z} _ {y} \end{array} \right] \in R ^ {n}.
$$
Let  $W_{y} = W_{0} + W_{y}^{\prime}$  for each  $y\in [2^{m}]$ , then
$$
W _ {y} \mathbf {x} _ {i} = \left[ \begin{array}{c} 2 \boldsymbol {\epsilon} \cdot \boldsymbol {e} _ {i} ^ {d} \\ (1 / \sqrt {2}) \mathbf {z} _ {y} \end{array} \right]
$$
for all  $i\in [m]$  . Thus,
$$
| | W _ {y} \mathbf {x} _ {i} - W _ {t} \mathbf {x} _ {j} | | = \left\| \left[ \begin{array}{c} 2 \epsilon (e _ {i} ^ {d} - e _ {j} ^ {d}) \\ (1 / \sqrt {2}) (\mathbf {z} _ {y} - \mathbf {z} _ {t}) \end{array} \right] \right\| \geq 2 \epsilon
$$
for each  $i, j \in [m]$  and  $y, t \in [2^m]$  s.t.  $(y, i) \neq (t, j)$ . Note that the last inequality holds since  $\epsilon < 1/4$ . Apply Lemma 3 with the Euclidean metric space, on the set  $Q = \{W_y \mathbf{x}_i : i \in [m], y \in [2^m]\}$  that contains  $m2^m$  different elements, to get a 1-Lipschitz function  $f: \mathbb{R}^n \to \mathbb{R}$  such that
$$
f \left(W _ {y} \mathbf {x} _ {i}\right) = y _ {i} \quad \forall i \in [ m ], y \in [ 2 ^ {m} ], \mathbf {y} \in \{\pm \epsilon \} ^ {m}
$$
Moreover, note that since  $||W_y'||_F \leq 1$ , we have
$$
\left| \left| W _ {y} - W _ {0} \right| \right| _ {F} = \left| \left| W _ {y} ^ {\prime} \right| \right| _ {F} \leq 1, \quad \left| \left| W _ {0} \right| \right| = 2 \sqrt {2} \cdot \epsilon
$$
Namely, we have that the function  $\mathbf{x} \to f(W_y\mathbf{x})$  belongs to  $\mathcal{F}_{B = 1,n,d}^{f,W_0}$  with  $\| W_0\| = 2\sqrt{2} \cdot \epsilon$  for each  $y \in [2^m]$ . Therefore,  $\mathcal{F}_{B = 1,n,d}^{f,W_0}$  can shatter  $m$  points from  $\{\mathbf{x} \in \mathbb{R}^d : \| \mathbf{x}\| \leq 1\}$  with margin  $\epsilon$ .
A.6 Proof of Theorem 4
This theorem is an adaptation of Corollary 14.12 (page 197) in Shalev-Shwartz and Ben-David [2014]. The stochastic gradient descent (SGD) algorithm, with projection step and initialization at  $W_{0}$ , is described as Algorithm 1 above. Observe that we just describe plain SGD on a convex Lipschitz stochastic optimization problem (over matrices, which is a vector space). The only nonstandard thing is the initialization at  $W_{0}$  and the projection around  $W_{0}$  instead of  $\mathbf{0}$ . We state the key technical result as Lemma 6 (in turn an adaptation of Lemma 14.1 from Shalev-Shwartz and Ben-David [2014]), and sketch its proof for completeness.
Algorithm 1 Stochastic Gradient Descent (SGD), with projection step and initialization at  $W_0$
parameters: Scalar  $\eta >0$  , integer  $T > 0$  , vector  $W_{0}$    
initialize:  $W^{(1)} = 0$    
for  $t = 1,2,\ldots ,T$  do Sample  $(x_i,y_i)$  pick a subgradient  $V_{t}$  of  $\ell (W_{t}x_{i},y_{i})$  w.r.t  $W_{t}$  update  $W^{(t + \frac{1}{2})} = W^{(t)} - \eta V_t$  update  $W^{(t + 1)} = \arg \min_{W:\| w - w_0\| _F\leq B}\left\| W - W^{(t + \frac{1}{2})}\right\| _F$  Projection step   
end for   
output:  $\hat{W} = \frac{1}{T}\sum_{t = 1}^{\top}W^{(t)}$
Lemma 6. Let  $V_{1}, \ldots, V_{T}$  be an arbitrary sequence of matrices. Any algorithm with an initialization  $W_{1} = W_{0}$  and an update rule of the form
-  $W^{(t + \frac{1}{2})} = W^{(t)} - \eta V_t$
$W^{(t + 1)} = \arg \min_{W:\| W - W_0\| _F\leq B}\left\| W - W^{(t + \frac{1}{2})}\right\| _F$
$$
\sum_ {t = 1} ^ {\top} \langle W ^ {(t)} - W ^ {*}, V _ {t} \rangle \leq \frac {\| W ^ {*} - W _ {0} \| _ {F} ^ {2}}{2 \eta} + \frac {\eta}{2} \sum_ {t = 1} ^ {\top} \| V _ {t} \| _ {F} ^ {2},
$$
where  $\langle U,V\rangle = \sum_{i,j}U_{i,j}V_{i,j}$ . In particular, for every  $B,L > 0$ , if for all  $t$  we have that  $\| V_t\| _F\leq L$  and if we set  $\eta = \sqrt{\frac{B^2}{L^2T}}$ , then for every  $W^{*}$  with  $\| W^{*} - W_{0}\|_{F}\leq B$  we have
$$
\sum_ {t = 1} ^ {\top} \left\langle W ^ {(t)} - W ^ {*}, V _ {t} \right\rangle \leq \frac {B L}{\sqrt {T}}
$$
Proof sketch of Lemma 6: Lemma 6 is different than Lemma 14.1 in Shalev-Shwartz and Ben-David [2014] in two aspects: first, we add a projection step, but still Lemma 14.1 holds (see Subsection 14.4.1 from Shalev-Shwartz and Ben-David [2014]). Second, the initialization is at  $W_{0}$  and not 0, but this is also not a problem since that at the end of the proof of Lemma 14.1, Shalev-Shwartz and Ben-David [2014] showed that
$$
\sum_ {t = 1} ^ {\top} \langle W ^ {(t)} - W ^ {*}, V _ {t} \rangle \leq \frac {1}{2 \eta} | | W ^ {(1)} - W ^ {*} | | _ {F} ^ {2} + \frac {\eta}{2} \sum_ {t = 1} ^ {\top} | | V _ {t} | | _ {F} ^ {2}.
$$
In our case  $W^{(1)} = W_0$ , this proves the first part of the Lemma. The second part follows by upper bounding  $\| W_0 - W^* \|_F$  by B,  $\| V_t \|_F$  by  $L$  which is true since f is  $L$ -Lipschitz, dividing by T, and plugging in the value of  $\eta$ .
With Lemma 6 in hand, the rest of the analysis follows directly as in Corollary 14.12 Shalev-Shwartz and Ben-David [2014], only instead of Lemma 14.1 from that book, we use Lemma 6.
A.7 Proof of Theorem 5
Observation 3. Let  $f_1, \ldots, f_k$  be L-Lipschitz functions from  $\mathbb{R}^d$  to  $\mathbb{R}$  with respect to the  $L_p$  norm (with  $p \in [1, \infty]$ ), then
$$
f(\mathbf{x}):= \max_{1\leq i\leq k}f_{i}(\mathbf{x})
$$
is also an  $L$ -Lipschitz function with respect to the  $L_{p}$  norm.
Proof. First, we show a known property about  $L_{\infty}$ . If  $\mathbf{x}, \mathbf{y} \in \mathbb{R}^k$ , then
$$
\left| \max  _ {1 \leq i \leq k} x _ {i} - \max  _ {1 \leq i \leq k} y _ {i} \right| \leq \max  _ {1 \leq i \leq k} | x _ {i} - y _ {i} | \tag {6}
$$
Indeed, assume that  $\max_i x_i > \max_i y_i$ . In this case, we have
$$
\left| \max  _ {i} x _ {i} - \max  _ {i} y _ {i} \right| = \max  _ {i} x _ {i} - \max  _ {i} y _ {i}
$$
let us denote by  $i_0$  the index  $i_0 \in [k]$  such that  $x_{i_0} = \max_i x_i$ . Then we have
$$
\max  _ {i} x _ {i} - \max  _ {i} y _ {i} = x _ {i _ {0}} - \max  _ {i} y _ {i} \leq x _ {i _ {0}} - y _ {i _ {0}} \leq \max  _ {i} (x _ {i} - y _ {i}) \leq \max  _ {i} | x _ {i} - y _ {i} |.
$$
The case  $\max_i y_i \geq \max_i x_i$  is symmetric. By Eq. 6 and the assumption that  $f_i$  is  $L$ -Lipschitz for each  $i \in [m]$  we get
$$
| f (\mathbf {x}) - f (\mathbf {y}) | = \left| \max  _ {i} f _ {i} (\mathbf {x}) - \max  _ {i} f _ {i} (\mathbf {y}) \right| \leq \max  _ {i} | f _ {i} (\mathbf {x}) - f _ {i} (\mathbf {y}) | \leq L \| \mathbf {x} - \mathbf {y} \|,
$$
from which the result follows.
Proof of Theorem 5. Let  $\mathbf{e}_i^d$  denote the indicator vector in  $\mathbb{R}^d$  with value one in the  $i$ -th coordinate, and value zero in the other coordinates. If  $d$  is clear from the context, we just use  $\mathbf{e}_i$  for simplicity. Let  $d = m + 1$  and  $n = 2^m + m$ . Let  $\mathbf{x}_1, \ldots, \mathbf{x}_m \in \{0, 1\}^d$  be defined by  $\mathbf{x}_i = \frac{1}{\sqrt{2}} (\mathbf{e}_i^d + \mathbf{e}_{m + 1}^d)$ . Define
$$
W _ {0} = 4 \sqrt {2} \cdot \epsilon \cdot \left[ \begin{array}{c c} I _ {m \times m} & 0 _ {m \times 1} \\ 0 _ {2 ^ {m} \times (m + 1)} \end{array} \right] \in \mathbb {R} ^ {n \times d}
$$
For any  $\mathbf{y} \in \{\pm \epsilon\}^m$ , we associate a different number from  $[2^m]$ , and we denote this number by  $y$ . For any  $y \in [2^m]$  define
$$
W _ {y} ^ {\prime} = \left[ \begin{array}{c c} 0 _ {n \times m} & 0 _ {m \times 1} \\ & \mathbf {e} _ {y} \end{array} \right]
$$
where  $\mathbf{e}_y\in \{0,1\}^{2^m}$  is a column vector. Note that  $W_{0}\mathbf{x}_{i} = 4\epsilon \cdot \mathbf{e}_{i}^{n}$  and  $W_{y}^{\prime}\mathbf{x}_{i} = \frac{1}{\sqrt{2}}\mathbf{e}_{y + m}^{n}$ . Letting  $W_{y} = W_{0} + W_{y}^{\prime}$ , we have that
$$
W _ {y} \mathbf {x} _ {i} = 4 \epsilon \mathbf {e} _ {i} ^ {n} + \frac {1}{\sqrt {2}} \mathbf {e} _ {m + y} ^ {n}.
$$
Now we are ready to define  $f:\mathbb{R}^n\to \mathbb{R}$
$$
f(\mathbf{x}):= \max_{j\in [m],z\in [2^{m} ]}\text{s.t.}\mathbf{z}_{j} = \epsilon \left[(0.5\cdot \mathbf{e}_{j}^{n} + 0.5\cdot \mathbf{e}_{m + z}^{n})^{\top}\mathbf{x},\frac{1}{\sqrt{8}}\right] - (\frac{1}{\sqrt{8}} +\epsilon),
$$
for any  $j \in [m]$ ,  $z \in [2^m]$  and  $\mathbf{x} \in \mathbb{R}^n$ . We emphasize that  $\mathbf{z} \in \{\pm \epsilon\}^m$  is the vector that is associated with the number  $z \in [2^m]$ . By Hölder's inequality, we have that
$$
\left| \left(0. 5 \cdot \mathbf {e} _ {j} ^ {n} + 0. 5 \cdot \mathbf {e} _ {m + z} ^ {n}\right) ^ {\top} (\mathbf {x} - \mathbf {y}) \right| \leq \left| \left| 0. 5 \cdot \mathbf {e} _ {j} ^ {n} + 0. 5 \cdot \mathbf {e} _ {m + z} ^ {n} \right| \right| _ {1} \cdot \left| \left| \mathbf {x} - \mathbf {y} \right| \right| _ {\infty} \leq \left| \left| \mathbf {x} - \mathbf {y} \right| \right| _ {\infty},
$$
for any  $\mathbf{x},\mathbf{y}\in \mathbb{R}^n$ . Therefore, the function  $\mathbf{x}\rightarrow (0.5\cdot \mathbf{e}_j^n +0.5\cdot \mathbf{e}_{m + z}^n)^\top \mathbf{x}$  is 1-Lipschitz with respect to the infinity norm for each  $j\in [m],z\in [2^m ]$ . This implies by Observation 3 that  $f$  is also a 1-Lipschitz function with respect to the infinity norm. Since the composition of an affine map, nonnegative weighted sums, maximum, and adding a constant are all operations that preserve convexity, we have that  $f$  is a convex function. Finally, for any  $\mathbf{y}\in \{\pm \epsilon \} ^m$  and  $\mathbf{x}_i$  we have
- If  $y_{i} = \epsilon$ , then
$$
\begin{array}{l} f (W _ {y} \mathbf {x} _ {i}) = f (4 \epsilon \mathbf {e} _ {i} ^ {n} + (1 / \sqrt {2}) \mathbf {e} _ {m + y} ^ {n}) \\ = \max _ {j \in [ m ], z \in [ 2 ^ {m} ] \mathrm {s . t .} \mathbf {z} _ {j} = \epsilon} (0. 5 \mathbf {e} _ {j} ^ {n} + 0. 5 \mathbf {e} _ {m + z} ^ {n}) ^ {\top} (4 \epsilon \mathbf {e} _ {i} ^ {n} + \mathbf {e} _ {y + m} ^ {n}) - (1 / \sqrt {8} + \epsilon) \\ = (0. 5 \mathbf {e} _ {i} ^ {n} + 0. 5 \mathbf {e} _ {m + y} ^ {n}) ^ {\top} (4 \epsilon \mathbf {e} _ {i} ^ {n} + (1 / \sqrt {2}) \mathbf {e} _ {m + y} ^ {n}) - (1 / \sqrt {8} + \epsilon) = \epsilon . \\ \end{array}
$$
- If  $y_{i} = -\epsilon$  and exists  $k \in [m]$  with  $y_{k} = \epsilon$ , then
$$
\begin{array}{l} f \left(W _ {y} \mathbf {x} _ {i}\right) = f \left(4 \epsilon \mathbf {e} _ {i} ^ {n} + (1 / \sqrt {2}) \mathbf {e} _ {m + y} ^ {n}\right) \\ = \max  _ {j \in [ m ], z \in [ 2 ^ {m} ] \mathrm {s . t .} \mathbf {z} _ {j} = \epsilon} (0. 5 \mathbf {e} _ {j} ^ {n} + 0. 5 \mathbf {e} _ {m + z} ^ {n}) ^ {\top} (4 \epsilon \mathbf {e} _ {i} ^ {n} + (1 / \sqrt {2}) \mathbf {e} _ {m + y} ^ {n}) - (1 / \sqrt {8} + \epsilon) \\ = \left(0. 5 \mathbf {e} _ {k} ^ {n} + 0. 5 \mathbf {e} _ {m + y} ^ {n}\right) ^ {\top} \left(4 \epsilon \mathbf {e} _ {i} ^ {n} + (1 / \sqrt {2}) \mathbf {e} _ {m + y} ^ {n}\right) - (1 / \sqrt {8} + \epsilon) = - \epsilon , \\ \end{array}
$$
where in this case, the max is obtained for some  $k\in [m]$  with  $\mathbf{y}_k = \epsilon$
- Otherwise, for every  $k$  we have  $y_{k} = -\epsilon$ , then
$$
f (W _ {y} \mathbf {x} _ {i}) = f (4 \epsilon \mathbf {e} _ {i} ^ {n} + (1 / \sqrt {2}) \mathbf {e} _ {m + y} ^ {n}) = 1 / \sqrt {8} - (1 / \sqrt {8} + \epsilon) = - \epsilon .
$$
In all cases, we get that
$$
f \left(W _ {y} \mathbf {x} _ {i}\right) = y _ {i}.
$$
Therefore,  $\mathcal{F}_{B = 1,n,d}^{f,W_0}$  with  $\| W_0\| = \sqrt{2}\cdot 4\epsilon$  can shatter  $m$  points from  $\{\mathbf{x}\in \mathbb{R}^d:||\mathbf{x}||\leq 1\}$  with margin  $\epsilon$ .
A.8 Proof of Theorem 6
Lemma 7. Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be  $L$ -Lipschitz and  $\mu$ -smooth. Let  $\psi : I \to R$  (when  $I := (I_1, I_2, I_3) \subseteq \mathbb{R}^3$ ) be the function  $(x, y, v) \to \frac{\sigma(vx + y) - \sigma(y)}{v}$ . If  $I_1 = [-b_x, b_x]$  and  $I_3 = (0, \infty)$ , then  $\psi$  is  $O(\mu b_x^2 + L)$ -Lipschitz.
Proof. We show that  $\psi$  is Lipschitz in each coordinate, which implies that  $\psi$  is Lipschitz. To that end, it is enough to upper bound the norm of the gradient:
$$
\| \nabla \psi \| = \sqrt {\left(\frac {d \psi}{d x}\right) ^ {2} + \left(\frac {d \psi}{d y}\right) ^ {2} + \left(\frac {d \psi}{d v}\right) ^ {2}} \leq O \left(\left| \frac {d \psi}{d x} \right| + \left| \frac {d \psi}{d x} \right| + \left| \frac {d \psi}{d x} \right|\right).
$$
$$
\left| \frac {d \psi}{d x} \right| = \left| \frac {v \sigma^ {\prime} (v x + y)}{v} \right| \leq L.
$$
Since  $\sigma (\cdot)$  is  $\mu$  -smooth, namely  $\sigma^{\prime}(\cdot)$  is  $\mu$  -Lipschitz we have
$$
\left| \frac {d \psi}{d y} \right| = \left| \frac {\sigma^ {\prime} (v x + y) - \sigma^ {\prime} (y)}{v} \right| \leq \left| \mu \frac {v x}{v} \right| = \mu | x | \leq \mu b _ {x}.
$$
$$
\left| \frac {d \psi}{d v} \right| = \left| \frac {v x \sigma^ {\prime} (v x + y) - (\sigma (v x + y) - \sigma (y))}{v ^ {2}} \right| = \left| \frac {\sigma (y) - (\sigma (v x + y) + \sigma^ {\prime} (v x + y) (- v x))}{v ^ {2}} \right|.
$$
Note that  $\sigma(y) - (\sigma(vx + y) + \sigma'(vx + y)(-vx)$  is exactly the reminder between  $\sigma(y)$  and the first-order Taylor expansion of  $\sigma(y)$  at  $vx + y$ . In Observation 4 we analyzed such a reminder of a smooth function and thus we can upper bound the above equation by
$$
\frac {\mu}{2} \frac {(v x) ^ {2}}{v ^ {2}} \leq \frac {\mu}{2} b _ {x} ^ {2},
$$
where  $c$  is a middle point between  $y$  and  $vx + y$ . Therefore,  $\psi$  is a  $O(\mu b_x^2 + L)$ -Lipchitz function, as required.
Observation 4. Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be a continuously differentiable function. For all  $x, y \in \mathbb{R}$ , the first-order Taylor expansion of  $\sigma(y)$  at  $x$  is defined by  $\sigma(x) + \sigma'(x)(y - x)$ , and the reminder  $R_x(y)$  is defined by
$$
\sigma (y) = \sigma (x) + \sigma^ {\prime} (x) (y - x) + R _ {x} (y).
$$
If  $\sigma (\cdot)$  is a  $\mu$  -smooth i.e.  $\sigma^{\prime}(\cdot)$  is an  $L$  -Lipschitz function, then
$$
\left| R _ {x} (y) \right| \leq \frac {\mu}{2} | y - x | ^ {2}
$$
Proof. Since  $\sigma'(\cdot)$  is continuous, we have that
$$
\int_ {0} ^ {1} \left(\sigma^ {\prime} (x + t (y - x)) - \sigma^ {\prime} (x)\right) (y - x) d t = \sigma (y) - \sigma (x) - \sigma^ {\prime} (x) (y - x),
$$
then we can write
$$
\sigma (y) = \sigma (x) + \sigma^ {\prime} (x) (y - x) + \int_ {0} ^ {1} \left(\sigma^ {\prime} (x + t (y - x)) - \sigma^ {\prime} (x)\right) (y - x) d t.
$$
Since  $\sigma (\cdot)$  is a  $\mu$  -smooth
$$
\begin{array}{l} \left| R _ {x} (y) \right| = \left| \int_ {0} ^ {1} \left(\sigma^ {\prime} (x + t (y - x)) - \sigma^ {\prime} (x)\right) (y - x) d t \right| \leq \\ \int_ {0} ^ {1} \left| \left(\sigma^ {\prime} (x + t (y - x)) - \sigma^ {\prime} (x)\right) (y - x) \right| d t \leq \mu | y - x | ^ {2} \int_ {0} ^ {1} t d t = \frac {\mu}{2} | y - x | ^ {2} \\ \end{array}
$$
Lemma 8. Let  $\psi : \mathbb{R}^k \to R$  be an  $L$ -Lipschitz function. Namely for all  $\alpha, \beta \in \mathbb{R}^k$  we have  $|\psi(\alpha) - \psi(\beta)| \leq L||\alpha - \beta||$ . For  $f_1, \ldots, f_k$  functions from  $\mathbb{R}^d$  to  $\mathbb{R}$  and  $\mathbf{x} \in \mathbb{R}^d$ , let  $\psi \circ (f_1, \ldots, f_k)(\mathbf{x}) := \psi(f_1(\mathbf{x}), \ldots, f_k(\mathbf{x}))$ . For  $\mathcal{F}_1, \ldots, \mathcal{F}_k$  class of functions from  $\mathbb{R}^d$  to  $\mathbb{R}$ , let
$$
\psi \circ (F _ {1}, \dots , F _ {k}) = \left\{\mathbf {x} \rightarrow \psi \circ (f _ {1}, \dots , f _ {k}) (\mathbf {x}): f _ {1} \in \mathcal {F} _ {1} \wedge \dots \wedge f _ {k} \in \mathcal {F} _ {k} \right\}.
$$
$$
\log N (\psi \circ (F _ {1}, \dots , F _ {k}), d _ {m}, \sqrt {k} L r) \leq \log N (F _ {1}, d _ {m}, r) + \dots + \log N (F _ {k}, d _ {m}, r)
$$
Proof. Define  $B = \psi \circ (\mathcal{F}_1, \dots, \mathcal{F}_k)$ . Let  $\mathcal{F}_1', \dots, \mathcal{F}_k'$  be an  $r$ -covers of  $\mathcal{F}_1, \dots, \mathcal{F}_k$  respectively. Define  $B' = \psi \circ (\mathcal{F}_1', \dots, \mathcal{F}_m')$ . For all  $f_1 \in \mathcal{F}_1, \dots, f_k \in \mathcal{F}_k$ , there exists  $f_1' \in \mathcal{F}_1', \dots, f_k' \in \mathcal{F}_k'$  s.t.
$$
d _ {m} (f _ {i}, f _ {i} ^ {\prime}) \leq r
$$
For  $i = 1,\dots,k$ . Therefore,
$$
\begin{array}{l} d _ {m} (\psi (f _ {1}, \dots , f _ {k}), \psi (f _ {1} ^ {\prime}, \dots , f _ {k} ^ {\prime})) = \frac {1}{m} \sum_ {i = 1} ^ {m} \left(\psi \circ (f _ {1}, \dots , f _ {k}) (\mathbf {x} _ {i}) - \psi \circ (f _ {1} ^ {\prime}, \dots , f _ {k} ^ {\prime}) (\mathbf {x} _ {i})\right) ^ {2} = \\ \leq \frac {L ^ {2}}{m} \sum_ {i = 1} ^ {m} \left\| \left(f _ {1} (\mathbf {x} _ {i}), \dots , f _ {k} (\mathbf {x} _ {i})\right) ^ {\top} - \left(f _ {1} ^ {\prime} (\mathbf {x} _ {i}), \dots , f _ {k} ^ {\prime} (\mathbf {x} _ {i})\right) ^ {\top} \right\| ^ {2} \\ = \frac {L ^ {2}}{m} \sum_ {i = 1} ^ {m} \left(f _ {1} (\mathbf {x} _ {i}) - f _ {1} ^ {\prime} (\mathbf {x} _ {i})\right) ^ {2} + \dots + \frac {L ^ {2}}{m} \sum_ {i = 1} ^ {m} \left(f _ {k} (\mathbf {x} _ {i}) - f _ {k} ^ {\prime} (\mathbf {x} _ {i})\right) ^ {2} \\ = L ^ {2} d _ {m} \left(f _ {1}, f _ {1} ^ {\prime}\right) ^ {2} + \dots + L ^ {2} d _ {m} \left(f _ {k}, f _ {k} ^ {\prime}\right) ^ {2} \leq k L ^ {2} r ^ {2}. \\ \end{array}
$$
Hence,  $B'$  is an  $(\sqrt{k} L r) - \text{cover}$  for  $B$ . Moreover, since  $|B'| \leq |\mathcal{F}_1'| \cdot \ldots \cdot |\mathcal{F}_k'|$ , we have
$$
N (\psi \circ (\mathcal {F} _ {1}, \dots , \mathcal {F} _ {k}), d _ {m}, \sqrt {k} L r) \leq N (\mathcal {F} _ {1}, d _ {m}, r) \cdot \dots \cdot N (\mathcal {F} _ {k}, d _ {m}, r)
$$
Lemma 9. Let  $\mathcal{F} = \{\mathbf{x}\to \langle \mathbf{w},\mathbf{x}\rangle :||\mathbf{w}||\leq B\}$  be the class of Euclidean norm-bounded linear predictors. Let  $L > 0$  and  $k > 0$  be some parameters, then for every  $\epsilon \in (0,L]$ ,
1.  $\sqrt{\log N(\mathcal{F},d_m,\epsilon)}\leq \frac{cBb_{\mathbf{x}}}{\epsilon}$
2.  $\int_{\epsilon}^{L}\sqrt{\log N(\mathcal{F},d_m,k\tau)} d\tau \leq \frac{cBb_x}{k}\left(\log (L) - \log (\epsilon)\right)$
For some universal constant  $c > 0$
Proof. The first part of the lemma is shown in Corollary 9 in Kakade et al.. For the second part,
$$
\int_ {\epsilon} ^ {L} \sqrt {N (\mathcal {F} , d _ {m} , k \tau)} d \tau \leq \int_ {\epsilon} ^ {L} \frac {c B b _ {x}}{k \tau} d \tau = \frac {c B b _ {x}}{k} (\log (L) - \log (\epsilon))
$$
Lemma 10. Let  $B \geq 2$  and let  $F = \{ \mathbf{x} \to v : v \in (0, B] \}$  be the class of constant functions. Let  $L > 0$  and  $k > 0$  be some parameters, then for every  $\epsilon \in (0, L]$ ,
1.  $\log N(\mathcal{F},d_m,\epsilon)\leq \frac{2\log_2(B)}{\epsilon}$
2.  $\int_{\epsilon}^{L}\sqrt{\log N(\mathcal{F},d_m,k\tau)} d\tau \leq \frac{2\log_2(B)}{k} (\log (L) - \log (\epsilon))$
Proof. Using  $n$  numbers we can represent every number in  $[0, B]$  with an accuracy of  $\epsilon = \frac{B}{n}$ . Thus,  $N(\mathcal{F}, d_m, \frac{B}{n}) \leq n$ , namely  $\log N(\mathcal{F}, d_m, \epsilon) \leq \log (\lceil \frac{B}{\epsilon} \rceil) \leq \frac{2 \log(B)}{\epsilon}$  for each  $0 < \epsilon \leq 1$  and  $B \geq 2$ , which proves the first part of the lemma. Moreover,
$$
\int_ {\epsilon} ^ {L} \sqrt {\log N (\mathcal {F} , d _ {m} , k \tau)} d \tau \leq \int_ {\epsilon} ^ {L} \frac {2 \log (B)}{k \tau} d \tau = \frac {2 \log (B)}{k} (\log (L) - \log (\epsilon))
$$
Proof of Theorem 6. Fix some set of inputs  $\mathbf{x}_1, \ldots, \mathbf{x}_m$  with norm at most  $b_x$ . The Rademacher complexity equals
$$
\begin{array}{l} \mathbb {E} \sup  _ {| | W | | _ {F} \leq B} \frac {1}{m} \sum_ {i = 1} ^ {m} \epsilon_ {i} v ^ {\top} \sigma \left(\left(W + w _ {0}\right) \mathbf {x} _ {i}\right) \leq \frac {b}{m} \mathbb {E} \sup  _ {| | W | | _ {F} \leq B} \left\| \sum_ {i = 1} ^ {m} \epsilon_ {i} \sigma \left(W \mathbf {x} _ {i} + W _ {0} \mathbf {x} _ {i}\right) \right\| \\ \leq \frac {b}{m} \mathbb {E} _ {W} \left\| \sum_ {i = 1} ^ {m} \epsilon_ {i} \sigma \left(W \mathbf {x} _ {i} + W _ {0} \mathbf {x} _ {i}\right) - \sum_ {i = 1} ^ {m} \epsilon_ {i} \sigma \left(W _ {0} \mathbf {x} _ {i}\right) \right\| + \frac {b}{m} \mathbb {E} \left\| \sum_ {i = 1} ^ {m} \epsilon_ {i} \sigma \left(W _ {0} \mathbf {x} _ {i}\right) \right\|. \tag {7} \\ \end{array}
$$
Let's start by upper bound the right-hand side of Equation 7, namely
$$
\frac {b}{m} \mathbb {E} \left\| \sum_ {i = 1} ^ {m} \epsilon_ {i} \sigma \left(W _ {0} \mathbf {x} _ {i}\right) \right\|.
$$
By definition of the spectral norm, we have that  $\| W_0\mathbf{x}_i\| \leq B_0b_x$ . Since  $\sigma (\cdot)$  is  $L$ -Lipschitz and  $\sigma (0) = 0$  we have that  $\| \sigma (W_0\mathbf{x}_i)\| \leq LB_0b_x$ . Let  $y_{i}\coloneqq \sigma (W_{0}\mathbf{x}_{i})$  where  $\| y_i\| \leq LB_0b_x$ . Then the expression above equals
$$
\frac {b}{m} \mathbb {E} \left\| \sum_ {i = 1} ^ {m} \epsilon_ {i} y _ {i} \right\| = \frac {b}{m} \mathbb {E} \sqrt {\left\| \sum_ {i = 1} ^ {m} \epsilon_ {i} y _ {i} \right\| ^ {2}} \leq \frac {b}{m} \sqrt {\mathbb {E} \left\| \sum_ {i = 1} ^ {m} \epsilon_ {i} y _ {i} \right\| ^ {2}} = \sqrt {\sum_ {j = 1} ^ {n} \mathbb {E} \left(\sum_ {i = 1} ^ {m} \epsilon_ {i} y _ {i , j}\right) ^ {2}}
$$
$$
\stackrel {\mathbb {E} \left[ \epsilon_ {i} \epsilon_ {i ^ {\prime}} \right] = 0} {=} \frac {b}{m} \sqrt {\sum_ {i , j} y _ {i , j} ^ {2}} = \frac {b}{m} \sqrt {\sum_ {i = 1} ^ {m} \| y _ {i} \| ^ {2}} \leq \frac {L B _ {0} b b _ {x}}{\sqrt {m}}. \tag {8}
$$
Moving back to the left-hand side of Equation 7, let  $\bar{\mathbf{x}} := \mathbf{x} / ||\mathbf{x}||$  for any non-zero  $\mathbf{x}$  (or  $0$  for  $\mathbf{x} = \mathbf{0}$ ). We have
$$
\begin{array}{l} \frac {b}{m} \mathbb {E} \sup _ {| | W | | _ {F} \leq B} \left\| \sum_ {i = 1} ^ {m} \epsilon_ {i} \sigma (W \mathbf {x} _ {i} + W _ {0} \mathbf {x} _ {i}) - \sum_ {i = 1} ^ {m} \epsilon_ {i} \sigma (W _ {0} \mathbf {x} _ {i}) \right\| \\ \leq \frac {b}{m} \mathbb {E} \sup  _ {W} \sqrt {\sum_ {j = 1} ^ {n} \left(\sum_ {i = 1} ^ {m} \epsilon_ {i} \left(\sigma \left(b _ {x} \mathbf {w} _ {j} ^ {\top} \bar {\mathbf {x}} _ {i} + b _ {x} \mathbf {w} _ {0 , j} ^ {\top} \bar {\mathbf {x}} _ {i}\right) - \sigma \left(b _ {x} \mathbf {w} _ {0 , j} ^ {\top} \bar {\mathbf {x}} _ {i}\right)\right)\right) ^ {2}}, \\ \end{array}
$$
where  $\mathbf{w}_{0,j}$  is the j row of  $W_{0}$ .