SlowGuess/NeurIPS_2023_OCR huggingface dataset:
note that in the worst-case, when all of the norm  $\| x_{i}\|$  is concentrated in the first pixel  $z_1^0 (x_i)$ , we have  $\Delta = \frac{d_0}{2}\cdot \sum_{i = 1}^{m}\| x_i\|^2$ . But in practice, the norms of the pixels  $z_{j}^{0}(x_{i})$  are typically more evenly distributed. For instance, we can assume that the pixels are  $\beta$ -balanced. Meaning,  $\forall i\in [m]:\max_{j\in [d_0]}\| z_j^0 (x_i)\|^2\leq \beta \operatorname{Avg}_{j\in [d_0]}[\| z_j^0 (x_i)\|^2 ] = \frac{\beta}{d_0}\| x_i\|^2$  (for some constant  $\beta >0$ ). Note that the rate  $\beta$  is a property exclusively dependent on the data as it is independent of the architecture and the training process. In particular, we obtain that  $\Delta \leq \frac{\beta}{2}\sum_{i = 1}^{m}\| x_{i}\|^{2}$ . In addition, we note that the second term in the bound is typically smaller than the first term as it scales with  $\sqrt{\log(\rho(w))}$  instead of  $\rho (w)$  and has no dependence on the size of the network. Therefore, in this case, our bound can be simplified to  $\mathcal{O}(\frac{\rho(w)}{\sqrt{m}}\sqrt{L\beta\operatorname{Avg}_{i = 1}^{m}[\|x_{i}\|^{2}]})$ .
Bounds for convolutional networks. As previously stated in section 2, convolutional neural networks utilize weight sharing across neurons in each layer, with each neuron in the  $l$ th layer having  $k_{l}$  input neurons (each of dimension  $c_{l-1}$ ). The norm of the network is calculated as  $\rho(w) = \prod_{l=1}^{L} \|w^l\|_F$ , and the degree at each layer is simply the kernel size  $\deg(G)_l = k_l$ . This results in a simplified version of the theorem.
Corollary 3.3 (Rademacher complexity of convolutional networks). Let  $G$  be a neural network architecture of depth  $L$  and let  $\rho > 0$ . Let  $X = \{x_{i}\}_{i=1}^{m}$  be a set of samples. Then,
$$
\mathcal {R} _ {X} (\mathcal {F} _ {G, \rho} ^ {\mathrm {s h}}) \leq \frac {\rho}{m} \cdot \left(1 + \sqrt {2 (\log (2) L + \sum_ {l = 1} ^ {L - 1} \log (k _ {l}) + \log (C))}\right) \cdot \sqrt {\prod_ {l = 1} ^ {L - 1} k _ {l} \cdot \max _ {j \in [ d _ {0} ]} \sum_ {i = 1} ^ {m} \| z _ {j} ^ {0} (x _ {i}) \| ^ {2}},
$$
where  $k_{l}$  denotes the kernel size in the l'th layer.
Comparison with the bound of [14]. The result in Corollary 3.3 is a refined version of the analysis in [14] for the specific case of convolutional networks. Theorem 1 in [14] can of course be applied to convolutional networks by treating their convolutional layers as fully-connected layers. However, this approach yields a substantially worse bound compared to the one proposed in Corollary 3.3.
Consider a convolutional neural network  $G$ . The  $l$ th convolutional layer takes the concatenation of  $(\sigma(z_1^l), \ldots, \sigma(z_{d_l}^l))$  as input and returns  $(z_1^{l+1}, \ldots, z_{d_{l+1}}^{l+1})$  as its output. Each  $z_j^{l+1}$  is computed as follows  $z_j^{l+1} = w^{l+1} \sigma(v_j^l(x))$ . Therefore, the matrix  $W^{l+1}$  associated with the convolutional layer contains  $d_{l+1}$  copies of  $w^{l+1}$  and its Frobenius norm is therefore  $\sqrt{d_{l+1}} \cdot \|w^{l+1}\|_F$ . In particular, by applying Theorem 1 in [14], we obtain a bound that scales as  $\mathcal{O}\left(\frac{\rho}{m} \sqrt{L \prod_{l=1}^{L-1} d_l \cdot \sum_{i=1}^{m} \|x_i\|^2}\right)$ . On the other hand, we have  $\prod_{l=1}^{L-1} k_l \leq \prod_{l=1}^{L-1} d_l$  and  $\|z_j^0(x_i)\| \leq \|x_i\|$ . Therefore, our bound is always smaller than  $\mathcal{O}\left(\frac{\rho}{m} \sqrt{L \prod_{l=1}^{L-1} d_l \cdot \sum_{i=1}^{m} \|x_i\|^2}\right)$ , which is the bound we obtained with [14]. In particular, if each convolutional layer has  $k_l = 2$  with no overlaps and  $d_0 = 2^L$ , then,  $d_l = 2^{L-l}$  and the bound of [14] would scale as  $\mathcal{O}\left(\frac{\rho}{\sqrt{m}} \sqrt{L 2^{0.5L(L-1)} \cdot \mathrm{Avg}_{i=1}^m[\|x_i\|^2]}\right)$ . On the other hand, in our bound we have the term  $\prod_{l=1}^{L-1} k_l = 2^{L-1}$ , and therefore, our bound scales as  $\mathcal{O}\left(\frac{\rho}{\sqrt{m}} \sqrt{L 2^L \mathrm{Avg}_{i=1}^m[\|x_i\|^2]}\right)$  which is significantly smaller. Finally, as we discussed earlier, if the norms of the pixels of each sample  $x$  are  $\beta$ -balanced (for some constant  $\beta > 0$ ), our bound scales as  $\mathcal{O}\left(\frac{\rho}{\sqrt{m}} \sqrt{L \mathrm{Avg}_{i=1}^m[\|x_i\|^2]}\right)$  which is smaller by a factor of  $2^{0.25L(L-1)}$  than the bound of [14].
Comparison with the bound of [33]. A recent paper [33] introduced generalization bounds for convolutional networks based on parameter counting. This bound roughly scales like  $\mathcal{O}\left(\sqrt{\frac{N(\sum_{l=1}^{L}\|w^{l}\|_{2} + \log(1 / \gamma)) + \log(1 / \delta)}{m}}\right)$ , where  $\gamma$  is a margin (typically smaller than 1), and  $N$  is the number of trainable parameters (taking weight sharing into account by counting each parameter of convolutional filters only once). While these bounds provide improved generalization guarantees when reusing parameters, it scales as  $\Omega(\sqrt{N / m})$  which is very large in practice. For example, the standard ResNet-50 architecture has approximately  $N = 23M$  trainable parameters while the MNIST dataset has only  $m = 50000$  training samples.
Comparison with the bounds of [32] and [47]. Recent papers [32] introduced generalization bound for convolutional networks based on covering numbers and weight sharing. For example,
the bounds in Theorem 17 of [32] roughly scale as  $\mathcal{O}\left(\frac{\prod_{l=1}^{L} \|W^l\|_2}{\sqrt{m}} E(w)^{1/\alpha} \cdot I_\alpha\right)$ , where  $E(w) = \left(\sum_{l=1}^{L-1} \frac{k_l^{\frac{\alpha}{2}} \|(w^l - u^l)^\top\|_{2,1}^\alpha}{\|w^l\|_2^\alpha} + \frac{\|w^L\|_2^\alpha}{\max_i \|w_{i,:}^L\|_2^\alpha}\right)$ ,  $k_l$  is the kernel size of the  $l$ th layer and  $W^l$  is the matrix corresponding to the linear operator associated with the  $l$ th convolutional layer,  $w_{i,:}$  is the  $i$ th row of a matrix  $w$ ,  $\alpha$  is either 2 or  $2/3$ ,  $I_\alpha = L$  if  $\alpha = 2$  and  $I_\alpha = 1$  o.w. and  $u^l$  are “reference” matrices of the same dimensions as  $w^l$ .
In general, neither our bounds nor those in [32] and [47] are inherently superior; with each being better in different cases. The main difference between their bounds and our bound, is that while their bounds include both multiplicative complexity term and additive complexity term, our bound includes only a multiplicative complexity term. For instance, the bound in Theorem 17 in [32] features both  $\prod_{l=1}^{L}\|w^l\|_2$  and  $E(w)$ , whereas our bound exclusively contains the multiplicative term  $\rho(w) = \prod_{l=1}^{L}\|w^l\|_F$ . For certain cases, this works in favor of our bound, but at the same time, our term  $\prod_{l=1}^{L}\|w^l\|_F$  is comparably larger than  $\prod_{l=1}^{L}\|w^l\|_2$  due to the smaller norms used in the former. As an example of a case where our bound is superior, consider the case described after Theorem 3.2, where each convolutional layer operates on non-overlapping patches of size 2 and the channel dimension is 1 at each layer. We choose  $u^l = 0$  for all  $l \in [L-1]$  (which is a standard choice of reference matrices). We notice that  $\|W^l\|_2 = \|w^l\|_2 = \|w^l\|_F$  since  $W^l$  is a block matrix and  $w^l$  is a vector. In addition, for any matrix  $A$ , we have  $\mathrm{rank}(A) \geq \frac{\|A^\top\|_{2,1}}{\|A\|_2} \geq \frac{\|A\|_F}{\|A\|_2} \geq 1$  and  $\mathrm{rank}(A) \geq \frac{\|A\|_2}{\max_i \|A_{i,\cdot}\|_2} \geq 1$  (see [17]). Therefore, the bound in [32] scales as at least  $\frac{\prod_{l=1}^{L}\|w^l\|_2}{\sqrt{m}} \cdot L^{3/2}$ , while our bound scales as  $\frac{\prod_{l=1}^{L}\|w^l\|_2}{\sqrt{m}} \sqrt{L}$  which is smaller by a factor of  $L$ .
Vacuous bounds? A uniform convergence bound for a class  $\mathcal{F}$  is an upper bound on the generalization gap that uniformly holds for all  $f\in \mathcal{F}$ , i.e.,  $\sup_{f\in \mathcal{F}}|\mathrm{err}_P(f) - \mathrm{err}_S(f)|\leq \epsilon (m,\mathcal{F})$  (typically tends to 0 as  $m\to \infty$ ). The Rademacher complexity bound in Lemma 2.2 is a form of uniform convergence bound. The issue with these bounds is that in interpolation regimes, where there exists a function  $f\in \mathcal{F}_G$  that fits any labeling of the samples  $\{x_i\}_{i = 1}^m$ , uniform convergence bounds are provably vacuous.
While the derivation of the bound in Theorem 3.2 follows the application of Rademacher complexities, we emphasize that it is not a uniform convergence bound and is not necessarily vacuous. Throughout the proof, we sliced the class  $\mathcal{F}_G$  into subsets  $\mathcal{F}_{G,\rho} = \{f\in \mathcal{F}_G\mid \rho (w)\leq \rho \}$  (for  $\rho \in \mathbb{N}$ ) and applied Lemma 2.2 for each of these subsets. This approach yields a bound that is proportional to  $\mathcal{O}(\rho /\sqrt{m})$  for each of the slices  $\mathcal{F}_{G,\rho}$ . We then apply a union bound to combine all of them to obtain a bound that scales as  $\mathcal{O}(\rho (w) / \sqrt{m})$ . This does not give a uniform convergence bound across all members of  $\mathcal{F}_G$ , since the bound is individualized for each member  $f_{w}\in \mathcal{F}_{G}$  based on the norm  $\rho (w)$ . For example, for  $w = 0$ , the bound will be 0 which is non-vacuous.
When the learning algorithm minimizes  $\rho (w)$  and the minimal norm required to fit the training labels is small, a tight bound can be achieved with a network that perfectly fits the training data. For example, suppose we have a dataset  $S = \{(x_{i},y_{i})\}_{i = 1}^{m}$ , a target function  $y(x) = \langle w^{*},x\rangle$ , and a hypothesis class  $\mathcal{F} = \{\langle w,x\rangle \mid w\in \mathbb{R}^d\}$ . A classic VC-theory bound scales as  $\mathcal{O}(\sqrt{d / m})$  which is vacuous when  $d\gg m$ . However, a norm-based bound scales as  $\mathcal{O}(\| w\| /\sqrt{m})$ , which is non-vacuous for any  $\| w\| \leq \| w^{*}\|$  (as long as  $m > \| w^{*}\| ^2$ ). In addition, the function  $y$  can be realized by  $\{\langle w,x\rangle \mid \| w\| \leq \| w^{*}\| \} \subset \mathcal{F}$ . Specifically, for smaller  $\| w^{*}\|$ , we need fewer samples to ensure that the bound is non-vacuous for a minimal norm model that perfectly fits the training data.
3.1 Proof Sketch
We propose an extension to a well-established method for bounding the Rademacher complexity of norm-bounded deep networks. This approach, originally developed by [13], utilizes a "peeling" argument, where the complexity bound for a depth  $L$  network is reduced to a complexity bound for a depth  $L - 1$  network and applied repeatedly. Specifically, the  $l$ th step bounds the complexity bound for depth  $l$  by using the product of the complexity bound for depth  $l - 1$  and the norm of the  $l$ th layer. By the end of this process, we obtain a bound that depends on the term  $\mathbb{E}_{\xi} g(|\sum_{i=1}^{m} \xi_i x_i|)$ .
$(g(x) = x$  in [13] and  $g = \exp$  in [14]), which can be further bounded using  $\max_{x\in X}\| x\|^2$ . The final bound scales with  $\tilde{\rho} (w)\cdot \max_{x\in X}\| x\|$ . Our extension further improves the tightness of these bounds by incorporating additional information about the network's sparsity.
To bound  $\mathcal{R}_X(\mathcal{F}_{G,\rho})$  using  $\rho (w)$ , we notice that each neuron operates on a small subset of the neurons from the previous layer. Therefore, we can bound the contribution of a certain constituent function  $z_{j}^{l}(x) = w_{j}^{l}v_{j}^{l - 1}(x)$  in the network using the norm  $\| w_j^l\| _F$  and the complexity of  $v_{j}^{l - 1}(x)$  instead of the full layer  $v^{l - 1}(x)$ . To explain this process, we provide a proof sketch of Proposition 3.1 for convolutional networks  $G = (V,E)$  with non-overlapping patches. For simplicity, we assume that  $d_0 = 2^L$ ,  $c_{l} = 1$ , and the strides and kernel sizes at each layer are  $k = 2$ . In particular, the network  $f_{w}$  can be represented as a binary tree, where the output neuron is computed as  $f_{w}(x) = z_{j_{0}}^{L}(x) = w^{L}\cdot \sigma (z_{1}^{L - 1}(x),z_{2}^{L - 1}(x)),z_{1}^{L - 1}(x) = w^{L - 1}\cdot \sigma (z_{1}^{L - 2}(x),z_{2}^{L - 2}(x))$  and  $z_{2}^{L - 1}(x) = w^{L - 1}\cdot \sigma (z_{3}^{L - 2}(x),z_{4}^{L - 2}(x))$  and so on. Similar to [14], we first bound the Rademacher complexity using Jensen's inequality,
$$
m \mathcal {R} _ {X} \left(\mathcal {F} _ {G, \rho}\right) = \frac {1}{\lambda} \log \exp \left(\lambda \mathbb {E} _ {\xi} \sup  _ {f _ {w}} \sum_ {i = 1} ^ {m} \xi_ {i} f _ {w} \left(x _ {i}\right)\right) \leq \frac {1}{\lambda} \log \left(\mathbb {E} _ {\xi} \sup  _ {f _ {w}} \exp \left(\left| \lambda \sum_ {i = 1} ^ {m} \xi_ {i} f _ {w} \left(x _ {i}\right) \right|\right)\right), \tag {2}
$$
where  $\lambda > 0$  is an arbitrary parameter. As a next step, we rewrite right-hand side as follows:
$$
\begin{array}{l} \mathbb {E} _ {\xi} \sup  _ {f _ {w}} \exp \left(\lambda \left| \sum_ {i = 1} ^ {m} \xi_ {i} \cdot f _ {w} (x _ {i}) \right|\right) = \mathbb {E} _ {\xi} \sup  _ {f _ {w}} \exp \left(\lambda \sqrt {\left| \sum_ {i = 1} ^ {m} \xi_ {i} \cdot w ^ {L} \cdot \sigma \left(z _ {1} ^ {L - 1} (x _ {i}) , z _ {2} ^ {L - 1} (x _ {i})\right) \right| ^ {2}}\right) \\ \leq \mathbb {E} _ {\xi} \sup  _ {f w} \exp \left(\lambda \sqrt {\| w ^ {L} \| _ {F} ^ {2} \cdot \sum_ {j = 1} ^ {2} \left\| \sum_ {i = 1} ^ {m} \xi_ {i} \cdot \sigma \left(z _ {j} ^ {L - 1} \left(x _ {i}\right)\right) \right\| _ {2} ^ {2}}\right). \tag {3} \\ \end{array}
$$
We notice that each  $z_{j}^{L - 1}(x)$  is itself a depth  $L - 1$  binary-tree neural network. Therefore, intuitively we would like to apply the same argument  $L - 1$  more times. However, in contrast to the above, the networks  $\sigma(z_{1}^{L - 1}(x)) = \sigma(w^{L - 1}(z_{1}^{L - 2}(x), z_{2}^{L - 2}(x)))$  and  $\sigma(z_{2}^{L - 1}(x)) = \sigma(w^{L - 1}(z_{3}^{L - 2}(x), z_{4}^{L - 2}(x)))$  end with a ReLU activation. To address this issue, [13, 14] proposed a "peeling process" based on Equation 4.20 in [51] that can be used to bound terms of the form  $\mathbb{E}_{\xi} \sup_{f' \in \mathcal{F}', W: \|W\|_{F} \leq R} \exp(\alpha \left\| \sum_{i=1}^{m} \xi_{i} \cdot \sigma(Wf'(x_{i})) \right\|)$ . However, this bound is not directly applicable when there is a sum inside the square root, as in equation 3 which includes a sum over  $j = 1, 2$ . Therefore, a modified peeling lemma is required to deal with this case.
Lemma 3.4 (Peeling Lemma). Let  $\sigma$  be a 1-Lipschitz, positive-homogeneous activation function which is applied element-wise (such as the ReLU). Then for any class of vector-valued functions  $\mathcal{F} \subset \{f = (f_1, \ldots, f_q) \mid \forall j \in [q] : f_j : \mathbb{R}^d \to \mathbb{R}^p\}$ , and any convex and monotonically increasing function  $g: \mathbb{R} \to [0, \infty)$ ,
$$
\mathbb{E}_{\xi}\sup_{\substack{f\in \mathcal{F}\\ W_{j}\colon \| W_{j}\|_{F}\leq R}}g\left(\sqrt{\sum_{j = 1}^{q}\left\|\sum_{i = 1}^{m}\xi_{i}\cdot\sigma(W_{j}f_{j}(x_{i}))\right\|_{2}^{2}}\right)\\ \leq 2\mathbb{E}_{\xi}\sup_{j\in [q], f\in \mathcal{F}}g\left(\sqrt{q} R\left\| \sum_{i = 1}^{m}\xi_{i}\cdot f_{j}(x_{i})\right\|_{2}\right).
$$
By applying this lemma  $L - 1$  times with  $g = \exp$  and  $f$  representing the neurons preceding a certain neuron at a certain layer, we can bound the term in equation 3 as follows:
$$
\begin{array}{l} \leq 2 ^ {L} \mathbb {E} _ {\xi} \sup  _ {j, w} \exp \left(\lambda \sqrt {\prod_ {l = 1} ^ {L} \| w ^ {l} \| _ {F} ^ {2} \cdot 2 ^ {L} \left| \sum_ {i = 1} ^ {m} \xi_ {i} x _ {i j} \right| ^ {2}}\right) \\ \leq 2 ^ {L} \sum_ {j = 1} ^ {d} \mathbb {E} _ {\xi} \exp \left(\lambda 2 ^ {L / 2} \rho \cdot \left| \sum_ {i = 1} ^ {m} \xi_ {i} x _ {i j} \right|\right) \leq 4 ^ {L} \sup  _ {j} \exp \left(\frac {\lambda^ {2} 2 ^ {L} \rho^ {2} \cdot \sum_ {i = 1} ^ {m} x _ {i j} ^ {2}}{2} + \lambda 2 ^ {L / 2} \rho \cdot \sqrt {\sum_ {i = 1} ^ {m} x _ {i j} ^ {2}}\right), \\ \end{array}
$$
where the last inequality follows from standard concentration bounds (see the proof for details). Finally, by equation 2 and properly adjusting  $\lambda$ , we can finally bound  $\mathcal{R}_X(\mathcal{F}_{G,\rho})$ .
In this section, we empirically evaluate the generalization bounds derived in section 3. In each experiment, we compare our bound with alternative bounds from the literature. We focus on simple
CONV-3-200
CONV-3-600
Figure 1: Comparing our bound with prior bounds in the literature during training. We plot our bound, the train and test errors, the generalization gap, and prior bounds from the literature during training. For each plot, we train a CONV-L-H network on MNIST with a different number of channels  $H$ .
CONV-3-1000
CONV-4-H
Figure 2: Varying the number of channels. We plot our bound, the train and test errors, the generalization gap, and prior bounds from the literature at the end of training. For each plot, we train a CONV- $L-H$  network on MNIST with a different number of layers  $L$  and channels  $H$ .
CONV-6-H
CONV-8-H
convolutional neural networks trained on MNIST and investigate the behavior of the bound when varying different hyperparameters. Each experiment was averaged across five runs. For additional experimental details, please refer to the appendix.
Network architecture. We used convolutional networks with  $L$  layers and  $H$  channels per layer denoted by CONV-  $L - H$ . The networks consist of a stack of  $L2\times 2$  convolutional layers with a stride of 1, 0 padding, and  $H$  output channels, utilizing ReLU activations followed by a fully-connected layer. The overall number of trainable parameters is at least  $4H + 4(L - 1)H^{2}$ .
**Optimization process.** Each model was trained using SGD for MSE-loss minimization between the logits of the network and the one-hot encodings of the training labels. We applied weight normalization [52] to all trainable layers, except for the last one, which is left un-normalized. In order to regularize the weight parameters, we used weight decay for each one of the layers of the network with the same regularization parameter  $\lambda > 0$ . To train each model, we used an initial learning rate of  $\mu = 0.01$  that is decayed by a factor of 0.1 at epochs 60, 100, 300, batch size 32, momentum of 0.9, and  $\lambda = 3\mathrm{e} - 3$  by default.
Experiments. We conducted several experiments to compare our bound to alternative bounds from the literature, when applied for neural networks trained on the MNIST dataset for classification. Throughout these experiments, we compared our bound to the one in Theorem 1 of [14], the third inequality in Theorem 2.1 of [33], Theorem 16 of [32], and Theorem 3.5 in [47] (explicitly as mentioned in their Table 3). To compute the bound in [14] for multi-class classification, we adopted a modified version based on the technique we utilized in the proof of Proposition 3.1, which allows us to extend these bounds for multi-class classification. Since [33] did not provide an explicit value for their coefficient  $C$ , we assumed it to be 1. In the first experiment we trained three models of different widths for MNIST classification. As can be seen in Figure 1, our bound is significantly smaller than the alternative bounds and is surprisingly close to 1, indicating its tightness. In the second experiment, we compared our bound with alternative bounds from previous literature, focusing on convolutional neural networks with varying depths and widths. As can be seen in Figure 2, even as
CONV-L-200
CONV-L-600
CONV-L-1000
Figure 3: Varying the number of layers. We plot our bound, the train and test errors, the generalization gap, and prior bounds from the literature at the end of training. For each plot, we train a CONV- $L-H$  network on MNIST with a different number of layers  $L$  and channels  $\bar{H}$ .
$H = 200$
Figure 4: Varying the regularization coefficient  $\lambda$ . We plot our bound, the train and test errors, the generalization gap, and prior bounds from the literature at the end of 500 epochs. For each plot, we trained a CONV-3- $H$  network on MNIST with a varying number of channels  $H$ .
$H = 600$
$H = 1000$
networks become highly overparameterized at large widths, the width does not appear to influence the results of any of the bounds. In Figure 3, we showcase the results for convolutional networks of different depths. Our bound increases exponentially with depth but rises more slowly than the bound from [14]. As an ablation study, in in Figure 4, we compared our bound with alternative bounds, this time varying the regularization coefficient  $\lambda$ . It is evident that our bound and some alternative bounds decrease concurrently with the generalization gap of the neural network, as desired outcome.
We studied the question of why certain deep learning architectures, such as convolutional networks and MLP-mixers, perform better than others on real-world datasets. To tackle this question, we derived Rademacher complexity generalization bounds for sparse neural networks, which are orders of magnitude better than a naive application of standard norm-based generalization bounds for fully-connected networks. In contrast to previous papers [33, 32], our results do not rely on parameter sharing between filters, suggesting that the sparsity of the neural networks is the critical component to their success. This sheds new light on the central question of why certain architectures perform so well and suggests that sparsity may be a key factor in their success. Even though our bounds are not practical in general, our experiments show that they are quite tight for simple classification problems, unlike other bounds based on parameter counting and norm-based bounds for fully-connected networks.
We thank Akshay Rangamani, Lorenzo Rosasco, Brian Cheung and Eran Malach for many relevant discussions. This material is based on the work supported by the Center for Minds, Brains and Machines (CBMM), funded by NSF STC award CCF-1231216. This work has also been sponsored by DOE SEA-CROGS project (DE-SC0023191).
[47] Florian Graf, Sebastian Zeng, Bastian Rieck, Marc Niethammer, and Roland Kwitt. On measuring excess capacity in neural networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 10164-10178. Curran Associates, Inc., 2022.
[48] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT Press, Cambridge, MA, 2 edition, 2018.
[49] Shai Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge eBooks, 2014.
[50] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
[51] Michel Ledoux and Michel Talagrand. Probability in Banach spaces. Springer Berlin, Heidelberg, 1991.
[52] Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks, 2016.
[53] Gábor Lugosi Stéphane Boucheron, Olivier Bousquet. Theory of classification: a survey of some recent advances. ESAIM: Probability and Statistics, 9:323-375, 3 2010.
[54] Tong Zhang. Statistical analysis of some multi-category large margin classification methods. J. Mach. Learn. Res., 5:1225-1251, dec 2004.
[55] Ambuj Tewari and Peter L. Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research, 8(36):1007-1025, 2007.
[56] Andreas Maurer. A vector-contraction inequality for rademacher complexities. In Ronald Ortner, Hans Ulrich Simon, and Sandra Zilles, editors, Algorithmic Learning Theory, pages 3–17, Cham, 2016. Springer International Publishing.
[57] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities - A Nonasymptotic Theory of Independence. Oxford University Press, 2013.