SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Each matrix in the set  $\{W: \| W \|_F \leq B\}$  is composed of rows, whose sum of squared norms is at most  $(Bb_x)^2$ . Thus, the set can be equivalently defined as the set of  $d \times n$  matrices, where each row  $j$  equals  $v_j \mathbf{w}_j$  for some  $v_j > 0$ ,  $\| w_j \| \leq 1$  and  $\| \mathbf{v} \|^2 \leq (Bb_x)^2$ . Noting that each  $v_j$  is positive, we can upper bound the expression in the displayed equation above as follows:
$$
\begin{array}{l} \frac {b}{m} \mathbb {E} _ {| | \mathbf {v} | | \leq B b _ {x}, | | \mathbf {w} _ {j} | | \leq 1} \sqrt {\sum_ {j = 1} ^ {n} \left(\sum_ {i = 1} ^ {m} \epsilon_ {i} \left(\sigma \left(v _ {j} \mathbf {w} _ {j} ^ {\top} \bar {\mathbf {x}} _ {i} + b _ {x} \mathbf {w} _ {0 , j} ^ {\top} \bar {\mathbf {x}} _ {i}\right) - \sigma \left(b _ {x} \mathbf {w} _ {0 , j} ^ {\top} \bar {\mathbf {x}} _ {i}\right)\right)\right) ^ {2}} \\ = \frac {b}{m} \mathbb {E} _ {| | \mathbf {v} | | \leq B b _ {x}, | | \mathbf {w} _ {j} | | \leq 1} \sqrt {\sum_ {j = 1} ^ {n} v _ {j} ^ {2} \left(\sum_ {i = 1} ^ {m} \frac {\epsilon_ {i}}{v _ {j}} \left(\sigma \left(v _ {j} \mathbf {w} _ {j} ^ {\top} \bar {\mathbf {x}} _ {i} + b _ {x} \mathbf {w} _ {0 , j} ^ {\top} \bar {\mathbf {x}} _ {i}\right) - \sigma \left(b _ {x} \mathbf {w} _ {0 , j} ^ {\top} \bar {\mathbf {x}} _ {i}\right)\right)\right) ^ {2}} \\ \leq \frac{b}{m}\mathbb{E}_{||\mathbf{v}^{\prime}||\leq B,||\mathbf{v}||\leq Bb_{x},||\mathbf{w}_{j}||\leq 1}\sqrt{\sum_{j = 1}^{n}v_{j}^{\prime 2}\left(\sum_{i = 1}^{m}\frac{\epsilon_{i}}{v_{j}}\left(\sigma(v_{j}\mathbf{w}_{j}^{\top}\bar{\mathbf{x}}_{i} + b_{x}\mathbf{w}_{0,j}^{\top}\bar{\mathbf{x}}_{i}) - \sigma(b_{x}\mathbf{w}_{0,j}^{\top}\bar{\mathbf{x}}_{i})\right)\right)^{2}}. \\ \end{array}
$$
For any choice of  $\epsilon$ ,  $\mathbf{v}$  and  $\mathbf{w}_1, \ldots, \mathbf{w}_n$ , the expression inside the expectation above can be written as
$$
\sup_{\| \mathbf{v}^{\prime}\| \leq Bb_{x}}\sqrt{\sum_{j = 1}^{n}v^{\prime}_{j}^{2}a_{j}^{2}} = \sup_{v^{\prime}_{j}:\sum_{j}v^{\prime}_{j}^{\prime} \leq (Bb_{x})^{2}}\sqrt{\sum_{j = 1}^{n}v^{\prime}_{j}^{2}a_{j}^{2}},
$$
for some numbers  $a_1, \ldots, a_n$ . Clearly, this is maximized by letting  $v_{j^*}' = Bb_x$  for some  $j^* \in \arg \max_j a_j^2$ , and  $v_{j}' = 0$  for all  $j \neq j^*$ . Plugging this observation back into the above expression, we can upper-bound the displayed equation by
$$
\frac{bBb_{x}}{m}\mathbb{E}_{||\mathbf{v}||\leq Bb_{x},||\mathbf{w}_{j}||\leq 1}\max_{j}\left|\sum_{i = 1}^{m}\frac{\epsilon_{i}}{v_{j}} \left(\sigma (v_{j}\mathbf{w}_{j}^{\top}\bar{\mathbf{x}}_{i} + b_{x}\mathbf{w}_{0,j}^{\top}\bar{\mathbf{x}}_{i}) - \sigma (b_{x}\mathbf{w}_{0,j}^{\top}\bar{\mathbf{x}}_{i})\right)\right|.
$$
Since the spectral norm upper bounds the norm of each row in a matrix, we can upper bound the above equation by
$$
\leq \frac {b B b _ {x}}{m} \mathbb {E} _ {v \in (0, B b _ {x} ], | | \mathbf {w} | | \leq 1, | | w _ {0} | | \leq B _ {0}} \left| \sum_ {i = 1} ^ {m} \frac {\epsilon_ {i}}{v} \left(\sigma (v \mathbf {w} ^ {\top} \bar {\mathbf {x}} _ {i} + b _ {x} \mathbf {w} _ {0} ^ {\top} \bar {\mathbf {x}} _ {i}) - \sigma (b _ {x} \mathbf {w} _ {0} ^ {\top} \bar {x} _ {i})\right) \right|.
$$
Let  $\psi : I \to \mathbb{R}$  be
$$
(\alpha , y, v) \rightarrow \frac {\sigma (v \alpha + y) - \sigma (y)}{v},
$$
where  $I = [-1,1] \times [-B_0b_x,B_0b_x] \times [-Bb_x,Bb_x]$ . Note that  $|\mathbf{w}^{\top}\bar{\mathbf{x}}_i| \leq ||\mathbf{w}^{\top}||||\bar{\mathbf{x}}_i|| \leq 1$  and  $|b_{x}w_{0}^{\top}\bar{\mathbf{x}}_{i}| \leq ||\mathbf{w}_{0}\top ||||\bar{\mathbf{x}}_{i}|| \leq B_{0}b_{x}$ . Therefore we can upper-bound the above expression by
$$
\frac {b B b _ {x}}{m} \mathbb {E} _ {v \in (0, B b _ {x} ], \| \mathbf {w} \| \leq 1, \| \mathbf {w} _ {0} \| \leq B _ {0}} \left| \sum_ {i = 1} ^ {m} \epsilon_ {i} \psi \left(\mathbf {w} ^ {\top} \bar {\mathbf {x}} _ {i}, \mathbf {w} _ {0} \top \mathbf {x} _ {i}, v\right) \right|. \tag {9}
$$
By Lemma 7 we have that  $\psi$  is  $c(\mu + L)$ -Lipschitz for some constant  $c$ . By a Taylor expansion and the fact that  $\sigma(\cdot)$  is smooth we have that  $\sigma(\beta) = \sigma(v\alpha + \beta) + \sigma'(\gamma)(-v\alpha)$ , where  $\gamma$  is some middle point between  $\beta$  and  $v\alpha + \beta$ . Therefore,
$$
\psi (\alpha , \beta , v) = \frac {\sigma (v \alpha + \beta) - \sigma (\beta)}{v} = \frac {\sigma^ {\prime} (\gamma) (- v \alpha)}{v} = - \alpha \sigma^ {\prime} (\gamma).
$$
since  $|\alpha| \leq 1$  and  $|\sigma'(\gamma)| \leq L$ , we have that  $\psi$  is bounded on  $I$  by  $L$ . Let
$$
\mathcal {F} = \left\{\mathbf {x} \rightarrow \left(\mathbf {w} ^ {\top} \bar {\mathbf {x}}, \mathbf {w} _ {0} \top \mathbf {x}, v\right): \| \mathbf {w} \| \leq 1, | | \mathbf {w} _ {0} | | \leq B _ {0}, | | \mathbf {v} | | \leq B b _ {x} \right\},
$$
$$
\mathcal {F} _ {1} = \left\{\mathbf {x} \rightarrow \mathbf {w} ^ {\top} \bar {\mathbf {x}}: \| \mathbf {w} \| \leq 1 \right\},
$$
$$
\mathcal {F} _ {2} = \left\{\mathbf {x} \rightarrow \mathbf {w} _ {0} \top \mathbf {x}: | | \mathbf {w} _ {0} | | \leq B _ {0} \right\},
$$
$$
\mathcal {F} _ {3} = \left\{\mathbf {x} \rightarrow v: v \in (0, B b _ {x} ] \right\}.
$$
Considering Equation 9, this is  $bBb_{x}$  times the Rademacher complexity of the function class  $\psi \circ \mathcal{F}$ . For every  $\epsilon > 0$ , using the Dudley Integral (see Srebro and Sridharan) we can upper bound the Rademacher complexity of  $\psi \circ \mathcal{F}$  by
$$
4 \epsilon + 1 2 \int_ {\epsilon} ^ {L} \sqrt {\frac {\log N (\psi \circ F , d _ {m} , \tau)}{m}} d \tau .
$$
From this point,  $c > 0$  represents some universal constant that may change from line to line. By Lemma 8 (with  $k = 3$ ) we can upper bound the above expression by
$$
4 \epsilon + 1 2 \int_ {\epsilon} ^ {L} \sqrt {\frac {\log N (F _ {1} , d _ {m} , \frac {c \tau}{\mu + L})}{m}} d \tau + 1 2 \int_ {\epsilon} ^ {L} \sqrt {\frac {\log N (F _ {2} , d _ {m} , \frac {c \tau}{\mu + L})}{m}} d \tau + 1 2 \int_ {\epsilon} ^ {L} \sqrt {\frac {\log N (F _ {3} , d _ {m} , \frac {c \tau}{\mu + L})}{m}} d \tau .
$$
By Lemma 9 and Lemma 10, for any  $\epsilon \geq 0$ , we can upper bound the above expression by
$$
4 \epsilon + \frac {c (\mu + L)}{\sqrt {m}} \Bigl (\log (L) - \log (\epsilon) + B _ {0} b _ {x} \log (L) - B _ {0} b _ {x} \log (\epsilon) + \log (B b _ {x}) \log (L) - \log (B b _ {x}) \log (\epsilon) \Bigr).
$$
By choosing  $\epsilon = \frac{1}{\sqrt{m}}$  we can upper bound Eq. 9 by
$$
\frac {4 + c (\mu + L) \left(\log (L) + \log (m) + B _ {0} b _ {x} \log (L) + B _ {0} b _ {x} \log (m) + \log (B b _ {x}) \log (L) + \log (B b _ {x}) \log (m)\right)}{\sqrt {m}}.
$$
Combining with Eq. 8, we get an upper bound on the Rademacher Complexity of  $\mathcal{F}_{b,B,n,d}^{g,W_0}$  of the form
$$
\begin{array}{l} \frac {4 + c (\mu + L) b B b _ {x} \left(\log (L) + \log (m) + B _ {0} b _ {x} \log (L) + B _ {0} b _ {x} \log (m)\right)}{\sqrt {m}} \\ + \frac {c (\mu + L) b B b _ {x} \left(\log (B b _ {x}) \log (L) + \log (B b _ {x}) \log (m)\right)}{\sqrt {m}} + \frac {L B _ {0} b b _ {x}}{\sqrt {m}}. \\ \end{array}
$$
Upper bounding this by  $\epsilon$  and solving for  $m$ , the result follows.
A.9 Proof of Theorem 7
To simplify notation, we rewrite  $\mathcal{F}_{k,\{S_j\},\{B_j\}}^{\{\sigma_j\}}$  as simply  $\mathcal{F}_k$ . For convenience, for any  $1 \leq l \leq k - 1$ , we define the class  $\mathcal{F}_l$  slightly differently. Each function in  $\mathcal{F}_l$  has the form
$$
\mathbf {x} \rightarrow \sigma_ {l} \left(W _ {l} \sigma_ {l - 1} (\dots \sigma_ {1} ((W _ {1} \mathbf {x})))\right), \tag {10}
$$
with the same constraints on the weights and the activation functions. We also define  $\mathcal{F}_0$  to be the class that contains just the identity function. The next Claim captures the "peeling" argument.
Lemma 11. For any integer  $1 \leq l \leq k - 1$ ,
$$
\mathbb {E} \sup  _ {f \in F _ {l}} \left| \left| \frac {1}{m} \sum_ {i = 1} ^ {m} \epsilon_ {i} f (\mathbf {x} _ {i}) \right| \right| \leq 2 c B _ {l} L \left(\frac {R _ {l - 1}}{\sqrt {m}} + \log^ {\frac {3}{2}} (m) \mathbb {E} \sup  _ {f \in F _ {l - 1}} \left| \left| \frac {1}{m} \sum_ {i = 1} ^ {m} \epsilon_ {i} f (\mathbf {x} _ {i}) \right| \right|\right)
$$
where  $R_{l - 1} = b_xL^{l - 1}||W_{l - 1}||\cdot ||W_{l - 2}||\dots \cdot ||W_1||$ $R_0 = b_x$  and  $c > 0$  is a universal constant.
$$
\begin{array}{l} \mathbb {E} \sup  _ {f \in F _ {l}} \left| \left| \frac {1}{m} \sum_ {i = 1} ^ {m} \epsilon_ {i} f (\mathbf {x} _ {i}) \right| \right| = \frac {1}{m} \mathbb {E} \sup  _ {f \in \mathcal {F} _ {l - 1}} \sup  _ {W _ {l}} \left| \left| \sum_ {i = 1} ^ {m} \epsilon_ {i} \sigma_ {l} \circ W _ {l} f (\mathbf {x} _ {i}) \right| \right| \\ = \frac {1}{m} \mathbb {E} \sup  _ {f \in \mathcal {F} _ {l - 1}} \sup  _ {W _ {l}} \sqrt {\sum_ {j = 1} ^ {n} \left(\sum_ {i = 1} ^ {m} \epsilon_ {i} \sigma_ {l} \left(\mathbf {w} _ {j} ^ {\top} f (\mathbf {x} _ {i})\right)\right) ^ {2}}, \\ \end{array}
$$
where  $\mathbf{w}_j$  is the  $j$ -th row of  $W_{l}$ . Each matrix in the set  $\{W:||W||_F\leq B_l\}$  is composed of rows, whose sum of squared norms is at most  $B_{l}^{2}$ . Thus, the set can be equivalently defined as the set of
matrices, where each row  $j$  equals  $v_{j}\mathbf{w}_{j}$  for some  $v_{j} > 0$ ,  $||\mathbf{w}_{j}|| \leq 1$ , and  $||\mathbf{v}||^{2} \leq B_{l}^{2}$ . Noting that each  $v_{j}$  is positive, we can upper bound the expression in the displayed equation above as follows:
$$
\begin{array}{l} \frac {1}{m} \mathbb {E} _ {f \in \mathcal {F} _ {l - 1}} \sup  _ {\mathbf {w} _ {j}, v} \sqrt {\sum_ {j = 1} ^ {n} \left(\sum_ {i = 1} ^ {m} \epsilon_ {i} \sigma_ {l} \left(v _ {j} \mathbf {w} _ {j} ^ {\top} f (\mathbf {x} _ {i})\right)\right) ^ {2}} \\ = \frac {1}{m} \mathbb {E} \sup  _ {f \in \mathcal {F} _ {l - 1}} \sup  _ {\mathbf {w} _ {j}, v} \sqrt {\sum_ {j = 1} ^ {n} v _ {j} ^ {2} \left(\sum_ {i = 1} ^ {m} \frac {\epsilon_ {i}}{v _ {j}} \sigma_ {l} \left(v _ {j} \mathbf {w} _ {j} ^ {\top} f (\mathbf {x} _ {i})\right)\right) ^ {2}} \\ \leq \frac {1}{m} \mathbb {E} \sup  _ {f \in \mathcal {F} _ {l - 1}} \sup  _ {\mathbf {w} _ {j}, v, v ^ {\prime}} \sqrt {\sum_ {j = 1} ^ {n} v _ {j} ^ {\prime 2} \left(\sum_ {i = 1} ^ {m} \frac {\epsilon_ {i}}{v _ {j}} \sigma_ {l} \left(v _ {j} \mathbf {w} _ {j} ^ {\top} f (\mathbf {x} _ {i})\right)\right) ^ {2}}, \\ \end{array}
$$
Where  $||\mathbf{v}'||^2 \leq B_l^2$  (note that  $\mathbf{v}$  must also satisfy this constraint). Moreover, for any choice of  $\epsilon, \mathbf{v}, f$  and  $\mathbf{w}_1, \dots, \mathbf{w}_n$ , the supremum over  $\mathbf{v}'$  is clearly attained by letting  $v_{j^*}' = B_l$  for some  $j^*$ . Plugging this observation back, we can upper-bound the displayed equation by
$$
\begin{array}{l} \frac {B _ {l}}{m} \mathbb {E} \sup  _ {f \in \mathcal {F} _ {l - 1}} \sup  _ {\mathbf {w} _ {j}, \mathbf {v}} \max  _ {j} \left| \sum_ {i = 1} ^ {m} \frac {\epsilon_ {i}}{v _ {j}} \sigma_ {l} \left(v _ {j} \mathbf {w} _ {j} ^ {\top} f (\mathbf {x} _ {i})\right) \right| \\ = \frac {B _ {l}}{m} \mathbb {E} \sup  _ {f \in \mathcal {F} _ {l - 1}} \sup  _ {\mathbf {w}: \| \mathbf {w} \| \leq 1, v \in (0, B ]} \left| \sum_ {i = 1} ^ {m} \frac {\epsilon_ {i}}{v} \sigma_ {l} \left(v \mathbf {w} ^ {\top} f (\mathbf {x} _ {i})\right) \right| \\ = \frac {B _ {l}}{m} \mathbb {E} \sup  _ {f \in \mathcal {F} _ {l - 1}} \sup  _ {\mathbf {w}: \| \mathbf {w} \| \leq 1, v \in (0, B ]} \left| \sum_ {i = 1} ^ {m} \epsilon_ {i} \psi_ {v} \left(\mathbf {w} ^ {\top} f (\mathbf {x} _ {i})\right) \right|, \tag {11} \\ \end{array}
$$
where  $\psi_v(z) = \frac{\sigma_l(vz)}{v}$  for any  $z\in \mathbb{R}$ . Since  $\sigma_{l}$  is L-Lipschitz, it follows that  $\psi_v(\cdot)$  is also L-Lipschitz regardless of  $v$ , since for any  $z,z^{\prime}\in \mathbb{R}$ ,
$$
\left| \psi_ {v} (z) - \psi_ {v} \left(z ^ {\prime}\right) \right| = \frac {\left| \sigma \left(v z\right) - \sigma \left(v z ^ {\prime}\right) \right|}{v} \leq \frac {L \left| v z - v ^ {\prime} z \right|}{v} = L \left| z - z ^ {\prime} \right|.
$$
As a result, we can upper-bound Eq. 11 by
$$
\frac{B_{l}}{m}\mathbb{E}\sup_{f\in \mathcal{F}_{l - 1}}\sup_{w:\| \mathbf{w}\| \leq 1,\psi \in \Psi_{L}}\left|\sum_{i = 1}^{m}\epsilon_{i}\psi \left(\mathbf{w}^{\top}f(\mathbf{x}_{i})\right)\right|,
$$
where  $\Psi_L$  is the class of all L-Lipschitz functions which equal 0 at the origin. To continue, it will be convenient to get rid of the absolute value in the displayed expression above. This can be done by noting that the expression equals
$$
\begin{array}{l} = \frac{B_{l}}{m}\mathbb{E}\sup_{f\in \mathcal{F}_{l - 1}}\sup_{w:\| \mathbf{w}\| \leq 1,\psi \in \Psi_{L}}\max \left\{\sum_{i = 1}^{m}\epsilon_{i}\psi \left(\mathbf{w}^{\top}f(\mathbf{x}_{i})\right), - \sum_{i = 1}^{m}\epsilon_{i}\psi \left(\mathbf{w}^{\top}f(\mathbf{x}_{i})\right)\right\} \\ \stackrel {(*)} {\leq} \frac {B _ {l}}{m} \mathbb {E} \sup  _ {f \in \mathcal {F} _ {l - 1}} \left[ \sup  _ {w: \| \mathbf {w} \| \leq 1, \psi \in \Psi_ {L}} \sum_ {i = 1} ^ {m} \epsilon_ {i} \psi \left(\mathbf {w} ^ {\top} f (\mathbf {x} _ {i})\right) + \sup  _ {w: \| \mathbf {w} \| \leq 1, \psi \in \Psi_ {L}} - \sum_ {i = 1} ^ {m} \epsilon_ {i} \psi \left(\mathbf {w} ^ {\top} f (\mathbf {x} _ {i})\right) \right] \\ \stackrel {(**)} {=} \frac {2 B _ {l}}{m} \mathbb {E} \sup  _ {f \in \mathcal {F} _ {l - 1}} \sup  _ {w: \| \mathbf {w} \| \leq 1, \psi \in \Psi_ {L}} \sum_ {i = 1} ^ {m} \epsilon_ {i} \psi \left(\mathbf {w} ^ {\top} f (\mathbf {x} _ {i})\right), \\ \end{array}
$$
where  $(\ast)$  follows from the fact that  $\max \{a, b\} \leq a + b$  for non-negative  $a, b$  and the observation that the supremum is always non-negative (it is only larger, say, than the specific choice of  $\psi$  being the zero function), and  $(\ast\ast)$  is by symmetry of the function class  $\Psi_L$  (if  $\psi \in \Psi_L$ , then  $-\psi \in \Psi_L$  as well).
Note that for every  $f \in \mathcal{F}_{l-1}$  and  $w$  with  $\| \mathbf{w} \| \leq 1$  we have  $|\mathbf{w}^\top f(\mathbf{x}_i)| \leq ||\mathbf{w}^\top|| \cdot ||f(\mathbf{x}_i)|| \leq R_{l-1}$ , where  $R_{l-1} = b_x L^{l-1} ||W_{l-1}| \cdot ||W_{l-2}| \cdot \dots \cdot ||W_1|$ . This class is a subset of the class of composition of all functions from  $\mathbb{R}^d$  to  $[-R_{l-1}, R_{l-1}]$ , and all univariate  $L$ -Lipschitz functions crossing the
origin. Fortunately, the Rademacher complexity of such composed classes was analyzed in Golowich et al. [2018] for a different purpose. Applying Theorem 4 from that paper, we get the upper bound of
$$
2 c B _ {l} L \left(\frac {R _ {l - 1}}{\sqrt {m}} + \log^ {\frac {3}{2}} (m) R _ {m} (\mathbf {w} ^ {\top} F _ {l - 1})\right)
$$
where  $c > 0$  is a universal constant, and
$$
R _ {m} (\mathbf {w} ^ {\top} F _ {l - 1}) := \mathbb {E} \sup _ {w: \| \mathbf {w} \| \leq 1, f \in F _ {l - 1}} \frac {1}{m} \sum_ {i = 1} ^ {m} \epsilon_ {i} \mathbf {w} ^ {\top} f (\mathbf {x} _ {i}) \leq \mathbb {E} \sup _ {f \in F _ {l - 1}} \left| \left| \frac {1}{m} \sum_ {i = 1} ^ {m} \epsilon_ {i} f (\mathbf {x} _ {i}) \right| \right|
$$
From this, the result follows.
Proof of Theorem 7. Remember the new definition of  $\mathcal{F}_k$  (see Eq. 10), note that we just removed  $\mathbf{w}_k$ , and therefore we turn to analyze the Rademacher complexity of
$$
\left\{\mathbf {w} _ {k} ^ {\top} f: \left| \left| \mathbf {w} _ {k} \right| \right| \leq B _ {k}, f \in \mathcal {F} _ {k - 1} \right\}
$$
on  $m$  inputs from  $\{\mathbf{x} \in \mathbb{R}^d : ||\mathbf{x}|| \leq b_x\}$ . Fix some set of inputs  $\mathbf{x}_1, \ldots, \mathbf{x}_m$  with norm at most  $b_x$ . The Rademacher complexity equals
$$
\mathbb {E} \sup  _ {f \in \mathcal {F} _ {k - 1}} \sup  _ {\mathbf {w} _ {k}} \frac {1}{m} \sum_ {i = 1} ^ {m} \epsilon_ {i} \mathbf {w} _ {k} ^ {\top} f (\mathbf {x} _ {i}) \leq B _ {k} \cdot \mathbb {E} \sup  _ {f \in \mathcal {F} _ {k - 1}} \left\| \frac {1}{m} \sum_ {i = 1} ^ {m} \epsilon_ {i} f (\mathbf {x} _ {i}) \right\|. \tag {12}
$$
By applying Lemma 11 repeatedly (i.e.  $k - 1$  times) we get
$$
\begin{array}{l} \mathbb {E} \sup  _ {f \in F _ {k - 1}} \left| \left| \frac {1}{m} \sum_ {i = 1} ^ {m} \epsilon_ {i} f (\mathbf {x} _ {i}) \right| \right| \\ \leq 2 c B _ {k - 1} L \left(\frac {R _ {k - 2}}{\sqrt {m}} + \log^ {\frac {3}{2}} (m) \mathbb {E} \sup  _ {f \in F _ {k - 2}} \left| \left| \frac {1}{m} \sum_ {i = 1} ^ {m} \epsilon_ {i} f (\mathbf {x} _ {i}) \right| \right|\right) \\ \leq \frac {2 c L B _ {k - 1} R _ {k - 2}}{\sqrt {m}} + \frac {(2 c L) ^ {2} B _ {k - 1} B _ {k - 2} R _ {k - 3} \log^ {\frac {3}{2}} (m)}{\sqrt {m}} + \dots \\ \dots + \frac {(2 c L) ^ {k - 1} \left(\prod_ {j \leq k - 1} B _ {j}\right) \log^ {\frac {3 (k - 2)}{2}} (m)}{\sqrt {m}} + (2 c L) ^ {k - 1} \left(\prod_ {j \leq k - 1} B _ {j}\right) \log^ {\frac {3 (k - 1)}{2}} (m) \mathbb {E} \left| \left| \frac {1}{m} \sum_ {i = 1} ^ {m} \epsilon_ {i} \mathbf {x} _ {i} \right| \right|, \tag {13} \\ \end{array}
$$
where  $R_{k-2} = b_x L^{k-2} ||W_{k-2}|| \cdot ||W_{k-3}|| \cdots ||W_1||$  and  $R_0 = b_x$ . Note that by Cauchy-Schwarz inequality
$$
\frac {1}{m} \mathbb {E} \left| \left| \sum_ {i = 1} ^ {m} \epsilon_ {i} \mathbf {x} _ {i} \right| \right| \leq \frac {1}{m} \sqrt {\mathbb {E} \left| \left| \sum_ {i = 1} ^ {m} \epsilon_ {i} \mathbf {x} _ {i} \right| \right| ^ {2}} = \frac {1}{m} \sqrt {\mathbb {E} \left. \sum_ {i , i ^ {\prime} = 1} ^ {m} \epsilon_ {i} \epsilon_ {i ^ {\prime}} \mathbf {x} _ {i} ^ {\top} \mathbf {x} _ {i ^ {\prime}} \right.} = \frac {1}{m} \sqrt {\sum_ {i = 1} ^ {m} \| \mathbf {x} _ {i} \| ^ {2}} \leq \frac {1}{m} \sqrt {m b _ {x} ^ {2}} = \frac {b _ {x}}{\sqrt {m}}.
$$
Plugging this back into Eq. 13, we get the following upper bound:
$$
\begin{array}{l} \frac {2 c L B _ {k - 1} R _ {k - 2}}{\sqrt {m}} + \frac {(2 c L) ^ {2} B _ {k - 1} B _ {k - 2} R _ {k - 3} \log^ {\frac {3}{2}} (m)}{\sqrt {m}} + \dots + \\ \dots \frac {(2 c L) ^ {k - 1} \prod_ {j \leq k - 1} B _ {j} \log^ {\frac {3 (k - 2)}{2}} (m)}{\sqrt {m}} + \frac {(2 c L) ^ {k - 1} \prod_ {j \leq k - 1} B _ {j} \log^ {\frac {3 (k - 1)}{2}} (m) b _ {x}}{\sqrt {m}} \\ = \sum_ {i = 1} ^ {k - 1} \frac {\left(2 c L\right) ^ {i} \cdot \left(\prod_ {j = k - i} ^ {k - 1} B _ {j}\right) R _ {k - 1 - i} \cdot \log^ {\frac {3 (i - 1)}{2}} (m)}{\sqrt {m}} + \left(2 c L\right) ^ {k - 1} \prod_ {j \leq k - 1} B _ {j} \log^ {\frac {3 (k - 1)}{2}} (m) \frac {b _ {x}}{\sqrt {m}}. \\ \end{array}
$$
Altogether we have  $k$  terms, each one of them upper bound by
$$
\frac {\left(2 c L\right) ^ {k - 1} R _ {k - 2} \left(\prod_ {i \leq k - 1} B _ {i}\right) \log^ {\frac {3 (k - 1)}{2}} (m)}{\sqrt {m}},
$$
where we use the assumption that  $L$  and  $||W_i|| \geq 1$ , which implies also that  $||W_i||_F \geq 1$ . Therefore, we can upper bound the displayed Eq. by
$$
\frac {k (2 c L) ^ {k - 1} R _ {k - 2} \left(\prod_ {i \leq k - 1} B _ {i}\right) \log^ {\frac {3 (k - 1)}{2}} (m)}{\sqrt {m}}.
$$
Combining with Eq. 12, the Rademacher complexity of  $\mathcal{F}_k$  is upper bounded by
$$
\frac {k (2 c L) ^ {k - 1} b R _ {k - 2} \left(\prod_ {i \leq k - 1} B _ {i}\right) \log^ {\frac {3 (k - 1)}{2}} (m)}{\sqrt {m}},
$$