SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tomer Galanti  
Center for Brains, Mind, and Machines  
Massachusetts Institute of Technology  
galanti@mit.edu
Liane Galanti  
School of Computer Science  
Tel Aviv University  
lianegalanti@mail.tau.ac.il
Mengjia Xu  
Department of Data Science  
New Jersey Institute of Technology  
mx6@njit.edu
Tomaso Poggio  
Center for Brains, Mind, and Machines  
Massachusetts Institute of Technology  
tp@csail.mit.edu
In this paper, we derive norm-based generalization bounds for sparse ReLU neural networks, including convolutional neural networks. These bounds differ from previous ones because they consider the sparse structure of the neural network architecture and the norms of the convolutional filters, rather than the norms of the (Toeplitz) matrices associated with the convolutional layers. Theoretically, we demonstrate that these bounds are significantly tighter than standard norm-based generalization bounds. Empirically, they offer relatively tight estimations of generalization for various simple classification problems. Collectively, these findings suggest that the sparsity of the underlying target function and the model's architecture plays a crucial role in the success of deep learning.
Over the last decade, deep learning with large neural networks has significantly advanced the solution of a myriad of tasks. These include image classification [1, 2, 3], language processing [4, 5, 6], interactions with open-ended environments [7, 8], and code synthesis [9]. Contrary to traditional theories such as [10], recent findings [11, 12] indicate that deep neural networks can generalize effectively even when their size vastly exceeds the number of training samples.
To address this question, recent work has proposed different generalization guarantees for deep neural networks based on various norms of their weight matrices [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]. Many efforts have been made to improve the tightness of these bounds to realistic scales. Some studies have focused on developing norm-based generalization bounds for complex network architectures, such as residual networks [24]. Other studies investigated ways to reduce the dependence of the bounds on the product of spectral norms [21, 25], or to use compression bounds based on PAC-Bayes theory [26, 27], or on the optimization procedure used to train the networks [19, 28, 29]. However, most of this research is centered around fully-connected networks, which generally underperform compared to other architectures like convolutional networks [30], residual network [1] and transformers [4, 31]. Thus, the ability of these bounds to explain the success of contemporary architectures is rather limited.
To fully understand the success of deep learning, it is necessary to analyze a wider scope of architectures beyond fully-connected networks. An interesting recent direction [32, 33] introduces generalization bounds for neural networks with shared parameters, such as convolutional neural networks. For example, [32] showed that by taking into account the structure of the convolutional
layers, we can derive generalization bounds with a norm component smaller than the norm of the associated linear transformation. However, many questions remain unanswered, including (a) Why certain architectures, such as convolutional networks [30] and MLP-mixers [34], perform better than fully-connected neural networks? (b) Is weight sharing necessary for the success of convolutional neural networks? (c) Can we establish norm-based generalization bounds for convolutional neural networks that are reasonably tight in practical settings? In this paper, we contribute to an understanding of all three questions.
Approximation guarantees for multilayer sparse networks. While fully-connected networks, including shallow networks, are universal approximators [35, 36] of continuous functions, they are largely limited in theory and in practice. Classic results [37, 38, 39, 40, 41] show that, in the worst-case, the number of parameters required to approximate a continuously differentiable target functions (with bounded derivatives) grows exponentially with the input dimension, a property known as the "curse of dimensionality".
A recent line of work [42, 43, 44] shows that the curse of dimensionality can be avoided by deep, sparse networks, when the target function is itself compositionally sparse. Furthermore, it has been conjectured that efficiently computable functions, that is functions that are computable by a Turing machine in polynomial time, are compositionally sparse. This suggests, in turns, that, for practical functions, deep and sparse networks can avoid the curse of dimensionality. These results, however, lack any implication about generalization; in particular, they do not show that overparametrized sparse networks have good generalization.
Norm-based generalization bounds. A recent thread in the literature [13, 14, 15, 16, 17, 18, 19, 20, 21, 23] has introduced norm-based generalization bounds for neural networks. In particular, let  $S = \{(x_i,y_i)\}_{i = 1}^m$  be a training dataset of  $m$  independently drawn samples from a probability measure  $P$  defined on the sample space  $\mathcal{X}\times \mathcal{Y}$ , where  $\mathcal{X}\subset \mathbb{R}^{d}$  and  $\mathcal{Y} = \{\pm 1\}$ . A fully-connected network is defined as  $f_{w}(x) = W^{L}\sigma (W^{L - 1}\sigma (\dots \sigma (W^{2}\sigma (W^{1}x))\dots))$ , where  $W^{l}\in \mathbb{R}^{d_{l + 1}\times d_{l}}$  and  $\sigma (x)$  is the element-wise ReLU activation function  $\max (0,x)$ . A common approach for estimating the gap between the train and test errors of a neural network is to use the Rademacher complexity of the network. For example, in [13], an upper bound on the Rademacher complexity is introduced based on the norms of the weight matrices of the network of order  $\mathcal{O}(\frac{2^L}{\sqrt{m}}\prod_{l = 1}^L\| W^l\| _F)$ . Later, [14] showed that the exponential dependence on the depth can be avoided by using the contraction lemma and obtained a bound that scales with  $\mathcal{O}(\sqrt{L})$ .
While these results provide solid upper bounds on the test error of deep neural networks, they only take into account very limited information about the architectural choices of the network. In particular, when applied to convolutional networks, the matrices  $W^{l}$  represent the linear operation performed by a convolutional layer whose filters are  $w^{l}$ . However, since  $W^{l}$  applies  $w^{l}$  to several patches ( $d_{l}$  patches), we have  $\| W^{l}\|_{F} = \sqrt{d_{l}}\| w^{l}\|_{F}$ . As a result, the bound scales with  $\mathcal{O}(\sqrt{\prod_{l = 1}^{L - 1}d_l})$ , that grows exponentially with  $L$ . This means that the bound is not suitable for convolutional networks with many layers as it would be very loose in practice. In this work, we establish generalization bounds that are customized for convolutional networks and scale with  $\prod_{l = 1}^{L}\| w^{l}\|_{F}$  instead of  $\prod_{l = 1}^{L}\| W^{l}\|_{F}$ .
In [45] they conducted a large-scale experiment evaluating multiple norm-based generalization bounds, including those of [17, 14]. They argued that these bounds are extremely loose and negatively correlated with the test error. However, in all of these experiments, they trained the neural networks with the cross-entropy loss which implicitly maximizes the network's weight norms once the network perfectly fits the training data. This can explain the observed negative correlation between the bounds and the error. In this work, we empirically show that our bounds provide reasonably tight estimations of the generalization gap for convolutional networks trained with weight normalization and weight decay using the MSE loss.
Generalization bounds for convolutional networks. Several recent papers have introduced generalization bounds for convolutional networks that take into account their unique structure. In [23], they introduced a generalization bound for neural networks with weight sharing. However, this bound only holds under the assumption that the weight matrices are orthonormal, which is not real-
istic in practice. In [33], they introduced generalization bounds for convolutional networks based on parameter counting. However, this bound scales roughly as the square root of the ratio between the number of parameters and the number of samples, which is vacuous when the network is overparameterized. In [32], they extended the generalization bounds of [17] for convolutional networks where the linear transformations  $W^l$  at each layer are replaced with the trainable parameters. While this paper provides generalization bounds in which each convolutional filter contributes only once to the bound, it does not hold when different filters are used for different patches, even if their norms are the same. In short, their analysis treats different patches as "datapoints" in an augmented problem where only one linear function is applied at each layer. If several choices of linear functions (different weights for different patches) are allowed, the capacity of the function class would increase. Although all of these papers offer generalization guarantees for convolutional networks, they base their findings either on the number of trainable parameters or on weight sharing. Notably, none of these studies directly address the question of whether weight sharing is essential for the effective generalization of convolutional networks. Furthermore, none provide empirical evidence to confirm that their bounds are reasonably tight in practical settings. In our previous work [46], we derived generalization bounds using a technique similar to the one employed here. The results in this paper extend those preliminary results to a more general and more detailed formulation.
In this work, we study the generalization guarantees of a broad class of sparse deep neural networks [42], such as convolutional neural networks. Informally, a sparse neural network is a graph of neurons represented as a Directed Acyclic Graph (DAG), where each neuron is a function of a small set of other neurons. We show how a simple modification to the classic norm-based generalization bound of [14] yields significantly tighter bounds for sparse neural networks (e.g., convolutional networks). Unlike previous bounds [33, 32, 47], our analysis demonstrates how to obtain generalization guarantees for sparse networks, without incorporating weight sharing, while having a weak dependence on the actual size of the network. These results suggest that it is possible to obtain good generalization performance with sparse neural networks without relying on weight sharing. Finally, we conduct multiple experiments to evaluate our bounds for overparameterized convolutional neural networks trained on simple classification problems. These experiments show that in these settings, our bound is significantly tighter than many bounds in the literature [14, 33, 32, 47]. As a result, this research provides a better understanding of the pivotal influence of the structure of the network's architecture [30, 34, 2] on its test performance.
We consider the problem of training a model for classification. Formally, the task is defined by a distribution  $P$  over samples  $(x,y) \in \mathcal{X} \times \mathcal{Y}$ , where  $\mathcal{X} \subset \mathbb{R}^{c_0 \times d_0}$  is the instance space (e.g., images), and  $\mathcal{Y} \subset \mathbb{R}^C$  is a label space containing the  $C$ -dimensional one-hot encodings of the integers  $1, \ldots, C$ . When thinking about the samples as images, we view  $c_0$  as the number of input channels and  $d_0$  as the image size. We consider a hypothesis class  $\mathcal{F} \subset \{f': \mathcal{X} \to \mathbb{R}^C\}$  (e.g., a neural network architecture), where each function  $f_w \in \mathcal{F}$  is specified by a vector of parameters  $w \in \mathbb{R}^N$  (i.e., trainable parameters). A function  $f_w \in \mathcal{F}$  assigns a prediction to an input point  $x \in \mathcal{X}$ , and its performance on the distribution  $P$  is measured by the expected error,  $\mathrm{err}_P(f_w) := \mathbb{E}_{(x,y) \sim P}[\mathbb{I}[\max_{j \neq y}(f_w(x_i)_j) \geq f_w(x_i)_y]]$ , where  $\mathbb{I} : \{\text{True}, \text{False}\} \to \{0,1\}$  be the indicator function (i.e.,  $\mathbb{I}[\text{True}] = 1$  and vice versa). Since we do not have direct access to the full population distribution  $P$ , the goal is to learn a predictor,  $f_w$ , from some training dataset  $S = \{(x_i, y_i)\}_{i=1}^m$  of independent and identically distributed (i.i.d.) samples drawn from  $P$  along with regularization to control  $f_w$ 's complexity.
2.1 Rademacher Complexities
We examine the generalization abilities of overparameterized neural networks by investigating their Rademacher complexity. This quantity can be used to upper bound the worst-case generalization gap (i.e., the distance between train and test errors) of functions from a certain class. It is defined as the expected performance of the class when averaged over all possible labelings of the data, where the
labels are chosen independently and uniformly at random from the set  $\{\pm 1\}$ . In other words, it is the average performance of the function class on random data. For more information, see [48, 49, 50].
Definition 2.1 (Rademacher Complexity). Let  $\mathcal{F}$  be a set of real-valued functions  $f_w: \mathcal{X} \to \mathbb{R}^C$  defined over a set  $\mathcal{X}$ . Given a fixed sample  $X \in \mathcal{X}^m$ , the empirical Rademacher complexity of  $\mathcal{F}$  is defined as follows:  $\mathcal{R}_X(\mathcal{F}) := \frac{1}{m} \mathbb{E}_{\xi : \xi_{ir} \sim U[\{\pm 1\}]} \left[ \sup_{f_w \in \mathcal{F}} \left| \sum_{i=1}^{m} \sum_{r=1}^{C} \xi_{ir} f_w(x_i)_r \right| \right]$ .
In contrast to the Vapnik-Chervonenkis (VC) dimension, the Rademacher complexity has the added advantage that it can be upper bounded based on a finite sample. The Rademacher complexity can be used to upper bound the gap between test and train errors of a certain class of functions [48]. In the following lemma we bound the gap between the test error and the empirical margin error  $\mathrm{err}_S^\gamma (f_w) = \frac{1}{m}\sum_{i = 1}^m\mathbb{I}[\max_{j\neq y}(f_w(x_i)_j) + \gamma \geq f_w(x_i)_y]$ .
Lemma 2.2. Let  $P$  be a distribution over  $\mathbb{R}^{c_0d_0} \times [C]$  and  $\mathcal{F} \subset \{f' : \mathcal{X} \to \mathbb{R}^C\}$ . Let  $S = \{(x_i, y_i)\}_{i=1}^m$  be a dataset of i.i.d. samples selected from  $P$  and  $X = \{x_i\}_{i=1}^m$ . Then, with probability at least  $1 - \delta$  over the selection of  $S$ , for any  $f_w \in \mathcal{F}$ , we have
$$
\operatorname {e r r} _ {P} \left(f _ {w}\right) - \operatorname {e r r} _ {S} ^ {\gamma} \left(f _ {w}\right) \leq \frac {2 \sqrt {2}}{\gamma} \cdot \mathcal {R} _ {X} (\mathcal {F}) + 3 \sqrt {\frac {\log (2 / \delta)}{2 m}}. \tag {1}
$$
The above bound is decomposed into two parts; one is the Rademacher complexity and the second scales as  $\mathcal{O}(1 / \sqrt{m})$  which is small when  $m$  is large. In section 3 we derive norm-based bounds on the Rademacher complexity of sparse networks. This lemma and the rest of the mathematical statements are proven in the appendix.
2.2 Architectures
A neural network architecture can be formally defined using a Directed Acyclic Graph (DAG)  $G = (V, E)$ . The class of neural networks associated with this architecture is denoted as  $\mathcal{F}_G$ . The set of neurons in the network is given by  $V = \bigcup_{l=0}^{L} \{z_1^l, \ldots, z_{d_l}^l\}$ , which is organized into  $L$  layers. An edge  $(z_i^l, z_j^{l-1}) \in E$  indicates a connection between a neuron in layer  $l-1$  and a neuron in layer  $l$ . The full set of neurons at the layer  $l$ th is denoted by  $v^l := (z_j^l)_{j=1}^{d_l}$ .
A neural network  $f_{w}:\mathbb{R}^{c_{0}\times d_{0}}\to \mathbb{R}^{C}$  takes "flattened" images  $x$  as input, where  $c_{0}$  is the number of input channels and  $d_{0}$  is the image dimension represented as a vector. Each neuron  $z_{i}^{l}:\mathbb{R}^{c_{0}\times d_{0}}\rightarrow \mathbb{R}^{c_{l}}$  computes a vector of size  $c_{l}$  (the number of channels in layer  $l$ ). To avoid confusion, in our definition, we think of each neuron as a vector of dimension  $c_{l}$ . This is analogous to a pixel holding three coordinates of RGB. The set of predecessor neurons of  $z_{i}^{l}$ , denoted by  $\operatorname {pred}(l,i)$ , is the set of all  $j\in [d_{l - 1}]$  such that  $(z_i^l,z_j^{l - 1})\in E$ , and  $v_{i}^{l}\coloneqq (z_{j}^{l})_{j\in \mathrm{pred}(l,i)}$  denotes the set of predecessor neurons of  $z_{i}^{l}$ . The network is recursively defined as follows:
$$
\forall r \in [ C ]: f _ {w} (x) _ {r} := \sum_ {i = 1} ^ {d _ {L - 1}} \langle w _ {r i} ^ {L}, z _ {i} ^ {L - 1} (x) \rangle ,
$$
where  $w_{ri}^{L} \in \mathbb{R}^{c_{L} - 1}$ ,  $z_{i}^{l}(x) \coloneqq \sigma(w_{i}^{l}v_{i}^{l - 1}(x))$ ,  $w_{i}^{l} \in \mathbb{R}^{c_{l} \times (c_{l - 1} \cdot |\operatorname{pred}(l - 1,i)|)}$  is a weight matrix,  $x = (z_{j}^{0}(x))_{j = 1}^{d_{0}}$ , each  $z_{j}^{0}(x)$  is a vector of dimension  $c_{0}$  representing the  $j$ th "pixel" of  $x$  and  $\sigma$  is the ReLU activation function. For simplicity, we denote  $w^{L} \coloneqq W^{L} = (w_{ri})_{r,i}$ .
The degree of sparsity of a neural network can be measured using the degree of the graph, which is defined as the maximum number of predecessors for each neuron. Specifically, the degree of a neural network architecture  $G$  is given by:  $\deg(G) \coloneqq \max_{l \in [L]} \deg(G)_l$ , where  $\deg(G)_l \coloneqq \max_{j \in [d_l]} |\operatorname{pred}(l,j)|$  is the maximal degree of the  $l$ th layer.
Convolutional neural networks. A special type of compositionally sparse neural networks is convolutional neural networks. In such networks, each neuron acts upon a set of nearby neurons from the previous layer, using a kernel shared across the neurons of the same layer.
To formally analyze convolutional networks, we consider a broader set of neural network architectures that includes sparse networks with shared weights. Specifically, for an architecture  $G$  with  $|\operatorname{pred}(l,j)| = k_l$  for all  $j \in [d_l]$ , we define the set of neural networks  $\mathcal{F}_G^{\mathrm{sh}}$  to consist of all neural
networks  $f_{w} \in \mathcal{F}_{G}^{\mathrm{sh}}$  that satisfy the weight sharing property  $w^{l} \coloneqq w_{j_{1}}^{l} = w_{j_{2}}^{l}$  for all  $j_{1}, j_{2} \in [d_{l}]$  and  $l \in [L]$ . Convolutional neural networks are essentially sparse neural networks with shared weights and locality (each neuron is a function of a set of nearby neurons of its preceding layer).
**Norms of neural networks.** As mentioned earlier, previous papers (e.g., [14]) proposed different generalization bounds based on different types of norms for measuring the complexity of fully-connected networks. One approach that was suggested by [14] is to use the product of the norms of the weight matrices given by  $\tilde{\rho}(w) \coloneqq \prod_{l=1}^{L} \|W^l\|_F$ .
In this work, we derive generalization bounds based on the product of the maximal norms of the kernel matrices across layers, defined as:  $\rho(w) := \| w^L \|_F \cdot \prod_{l=1}^{L-1} \max_{j \in [d_l]} \| w_j^l \|_F$ , where  $\| \cdot \|_F$  and is the Frobenius norm. For a convolutional neural network, we have a simplified form of  $\rho(w) = \prod_{l=1}^{L} \| w^l \|_F$ , due to the weight sharing property. This quantity is significantly smaller than the quantity  $\tilde{\rho}(w) = \| w^L \|_F \cdot \prod_{l=1}^{L-1} \sqrt{\sum_{j=1}^{d_l} \| w_j^l \|_F^2}$  used by [14]. For instance, when weight sharing is applied, we can see that  $\tilde{\rho}(w) = \rho(w) \cdot \sqrt{\prod_{l=1}^{L-1} d_l}$  which is significantly larger than  $\rho(w)$ .
Classes of interest. In the next section, we study the Rademacher complexity of classes of compositionally sparse neural networks that are bounded in norm. We focus on two classes:  $\mathcal{F}_{G,\rho} := \{f_w \in \mathcal{F}_G \mid \rho(w) \leq \rho\}$  and  $\mathcal{F}_{G,\rho}^{\mathrm{sh}} := \{f_w \in \mathcal{F}_G^{\mathrm{sh}} \mid \rho(w) \leq \rho\}$ , where  $G$  is a compositionally sparse neural network architecture and  $\rho$  is a bound on the norm of the network parameters.
In this section, we introduce our main theoretical results. The following theorem provides a bound on the Rademacher complexity of the class  $\mathcal{F}_{G,\rho}$  of networks of architecture  $G$  of norm  $\leq \rho$ .
Proposition 3.1. Let  $G$  be a neural network architecture of depth  $L$  and let  $\rho > 0$ . Let  $X = \{x_{i}\}_{i=1}^{m}$  be a set of samples. Then,
$$
\begin{array}{l} \mathcal {R} _ {X} \left(\mathcal {F} _ {G, \rho}\right) \leq \frac {\rho}{m} \cdot \left(1 + \sqrt {2 (\log (2) L + \sum_ {l = 1} ^ {L - 1} \log (\deg (G) _ {l}) + \log (C))}\right) \\ \cdot \sqrt {\max  _ {j _ {0} , \cdots , j _ {L}} \prod_ {l = 1} ^ {L - 1} | \operatorname {p r e d} (l , j _ {l}) | \cdot \sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} (x _ {i}) \| _ {2} ^ {2}}, \\ \end{array}
$$
where the maximum is taken over  $j_0, j_1, \ldots, j_L$ , such that,  $j_{l-1} \in \operatorname{pred}(l, j_l)$  for all  $l \in [L]$ .
The proof for this theorem builds upon the proof of Theorem 1 in [14]. A sketch of the proof is presented in Section 3.1. As we show next, by combining Lemma 2.2 and Proposition 3.1 we can obtain an upper bound on the test error of compositionally sparse neural networks.
Theorem 3.2. Let  $P$  be a distribution over  $\mathbb{R}^{c_0d_0} \times \{\pm 1\}$ . Let  $S = \{(x_i, y_i)\}_{i=1}^m$  be a dataset of i.i.d. samples selected from  $P$ . Then, with probability at least  $1 - \delta$  over the selection of  $S$ , for any  $f_w \in \mathcal{F}_G$ ,
$$
\begin{array}{l} \operatorname {e r r} _ {P} \left(f _ {w}\right) - \operatorname {e r r} _ {S} ^ {\gamma} \left(f _ {w}\right) \leq \frac {2 \sqrt {2} (\rho (w) + 1)}{\gamma m} \cdot \left(1 + \sqrt {2 (\log (2) L + \sum_ {l = 1} ^ {L - 1} \log (\deg (G) _ {l}) + \log (C))}\right) \\ \cdot \sqrt {\max _ {j _ {0} , \cdots , j _ {L}} \prod_ {l = 1} ^ {L - 1} | \operatorname * {p r e d} (l , j _ {l}) | \cdot \sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} (x _ {i}) \| _ {2} ^ {2}} + 3 \sqrt {\frac {\log (2 (\rho (w) + 2) ^ {2} / \delta)}{2 m}}, \\ \end{array}
$$
where the maximum is taken over  $j_0,\ldots ,j_L$ , such that,  $j_{l - 1}\in \operatorname {pred}(l,j_l)$  for all  $l\in [L]$ .
The theorem above provides a generalization bound for neural networks of a given architecture  $G$ . To understand this bound, we first analyze the term  $\Delta := \max_{j_0, \dots, j_L} \prod_{l=1}^{L-1} |\operatorname{pred}(l, j_l)| \cdot \sum_{i=1}^{m} \|z_{j_0}^0(x_i)\|_2^2$ . We consider a setting where  $d_0 = 2^L$ ,  $c_l = 1$  and each neuron takes two neurons as input,  $k_l := |\operatorname{pred}(l, j)| = 2$  for all  $l \in [L]$  and  $j \in [d_l]$ . In particular,  $\prod_{l=1}^{L-1} k_l = 2^{L-1}$  and  $z_j^0(x_i)$  is the  $j$ th pixel of  $x_i$ . Therefore, we have  $\Delta = \frac{d_0}{2} \cdot \max_{j_0} \sum_{i=1}^{m} \|z_{j_0}^0(x_i)\|_2^2$ . We