VC dimension of partially quantized neural networks in the overparametrized regime

Vapnik-Chervonenkis ( VC ) theory has so far been unable to explain the small generalization error of overparametrized neural networks . Indeed , existing applications of VC theory to large networks obtain upper bounds on VC dimension that are proportional to the number of weights , and for a large class of networks , these upper bound are known to be tight . In this work , we focus on a subclass of partially quantized networks that we refer to as hyperplane arrangement neural networks ( HANNs ) . Using a sample compression analysis , we show that HANNs can have VC dimension significantly smaller than the number of weights , while being highly expressive . In particular , empirical risk minimization over HANNs in the overparametrized regime achieves the minimax rate for classification with Lipschitz posterior class probability . We further demonstrate the expressivity of HANNs empirically . On a panel of 121 UCI datasets , overparametrized HANNs match the performance of state-of-the-art full-precision models . 1 INTRODUCTION . Neural networks have become an indispensable tool for machine learning practitioners , owing to their impressive performance especially in vision and natural language processing ( Goodfellow et al. , 2016 ) . In practice , neural networks are often applied in the overparametrized regime and are capable of fitting even random labels ( Zhang et al. , 2021 ) . Evidently , these overparametrized models perform well on real world data despite their ability to grossly overfit , a phenomenon that has been dubbed “ the generalization puzzle ” ( Nagarajan & Kolter , 2019 ) . Toward solving this puzzle , several research directions have flourished and offer potential explanations , including implicit regularization ( Chizat & Bach , 2020 ) , interpolation ( Chatterji & Long , 2021 ) , and benign overfitting ( Bartlett et al. , 2020 ) . So far , VC theory has not been able to explain the puzzle , because existing bounds on the VC dimensions of neural networks are on the order of the number of weights ( Maass , 1994 ; Bartlett et al. , 2019 ) . It remains unknown whether there exist neural network architectures capable of modeling rich set of classfiers with low VC dimension . The focus of this work is on a class of neural networks with threshold activation that we refer to as hyperplane arrangement neural networks ( HANNs ) . Using the theory of sample compression schemes ( Littlestone & Warmuth , 1986 ) , we show that HANNs can have VC dimension that is significantly smaller than the number of parameters . Furthermore , we apply this result to show that HANNs have high expressivity by proving that HANN classifiers achieve minimax-optimality when the data has Lipschitz posterior class probability in an overparametrized setting . We benchmark the empirical performance of HANNs on a panel of 121 UCI datasets , following several recent neural network and neural tangent kernel works ( Klambauer et al. , 2017 ; Wu et al. , 2018 ; Arora et al. , 2019 ; Shankar et al. , 2020 ) . In particular , Klambauer et al . ( 2017 ) showed that , using a properly chosen activation , overparametrized neural networks perform competitively compared to classical shallow methods on this panel of datasets . Our experiments show that HANNs , a partially-quantized model , match the classification accuracy of the self-normalizing neural network ( Klambauer et al. , 2017 ) and the dendritic neural network ( Wu et al. , 2018 ) , both of which are fullprecision models . 1.1 RELATED WORK . VC dimensions of neural networks . The VC-dimension Vapnik & Chervonenkis ( 1971 ) is a combinatorial measure of the complexity of a concept class , i.e. , a set of classifiers . The Fundamental Theorem of Statistical Learning ( Shalev-Shwartz & Ben-David , 2014 , Theorem 6.8 ) states that a concept class has finite VC-dimension if and only if it is probably approximately correct ( PAC ) learnable , where the VC-dimension is tightly related to the number of samples required for PAC learning . For threshold networks , Cover ( 1965 ) ; Baum & Haussler ( 1989 ) showed a VC-dimension upper bounded of O ( w logw ) , where w is the number of parameters . Maass ( 1994 ) obtained a matching lower bound attained by a network architecture with two hidden layers . More recently , Bartlett et al . ( 2019 ) obtained the upper and lower bounds O ( w ` logw ) and Ω ( w ` log ( w/ ` ) ) respectively for the case when the activation is piecewise linear , where ` is the number of layers . These lower bounds are achieved by somewhat unconventional network architectures . The architectures we consider exclude these and thus we are able to achieve a smaller upper bound on the VC dimensions . The generalization puzzle . In practice , neural networks that achieve state-of-the-art performance use significantly more parameters than samples , a phenomenon that can not be explained by classical VC theory if the VC dimension ≥ number of weights . This has been dubbed the generalization puzzle ( Zhang et al. , 2021 ) . To explain the puzzle , researchers have pursued new directions including margin-based bounds ( Neyshabur et al. , 2017 ; Bartlett et al. , 2017 ) , PAC-Bayes bounds ( Dziugaite & Roy , 2017 ) , and implicit bias of optimization methods ( Gunasekar et al. , 2018 ; Chizat & Bach , 2020 ) . We refer the reader to the recent article by Bartlett et al . ( 2021 ) for a comprehensive coverage of this growing literature . The generalization puzzle is not specific to deep learning . For instance , AdaBoost has been observed to continue to decrease the test error while the VC dimension grows linearly with the number of boosting rounds ( Schapire , 2013 ) . Other learning algorithms that exhibits similarly surprising behavior include random forests ( Wyner et al. , 2017 ) and kernel methods ( Belkin et al. , 2018 ) . Minimax-optimality . Whereas VC theory is distribution-independent , minimax theory is concerned with the question of optimal estimation/classification under distributional assumptions1 . A minimax optimality result shows that the expected excess classification error goes to zero at the fastest rate possible , as the sample size tend to infinity . For neural networks , this often involves a hyperparameter selection scheme in terms of the sample size . Faragó & Lugosi ( 1993 ) show minimax-optimality of ( underparametrized ) neural networks for learning to classify under certain assumptions on the Fourier transform of the data distribution . Schmidt-Hieber ( 2020 ) shows minimax-optimality of s-sparse neural networks for regression over Hölder classes , where at most s = O ( n log n ) network weights are nonzero , and n = the number of training samples . Kim et al . ( 2021 ) extends the results of Schmidt-Hieber ( 2020 ) to the classification setting , remarking that effective optimization under sparsity constraint is lacking . Kohler & Langer ( 2020 ) and Langer ( 2021 ) proved minimax-optimality without the sparsity assumption , however in an underparametrized setting . To the best of our knowledge , our result is the first to establish minimax optimality of overparametrized neural networks without a sparsity assumption . ( Partially ) quantized neural networks . Quantizing some of the weights and/or activations of neural networks has the potential to reduce the high computational burden of neural networks at test time ( Qin et al. , 2020 ) . Many works have focused on the efficient training of quantized neural networks to close the performance gap with full-precision architectures ( Hubara et al. , 2017 ; Rastegari et al. , 2016 ; Lin et al. , 2017 ) . Several works have observed that quantization of the activations , rather than of the weights , leads to a larger accuracy gap ( Cai et al. , 2017 ; Mishra et al. , 2018 ; Kim et al. , 2019 ) . Towards explaining this phenomenon , researchers have focused on understanding the so-called coarse gradient , a term coined by Yin et al . ( 2019 ) , often used in training QNNs as a surrogate for the usual gradient . One commonly used heuristic is the straight-through-estimator ( STE ) first introduced in an online course by Hinton et al . ( 2012 ) . Theory supporting the STE heuristic has recently been studied in Li et al . ( 2017 ) and Yin et al . ( 2019 ) . 1Without distributional assumptions , no classifier can be minimax optimal in light of the No-Free-Lunch Theorem ( Devroye , 1982 ) . QNNs have also been analyzed from other theoretical angles , including mean-field theory ( Blumenfeld et al. , 2019 ) , memory capacity ( Vershynin , 2020 ) , Boolean function representation capacity ( Baldi & Vershynin , 2019 ) and adversarial robustness ( Lin et al. , 2018 ) . Of particular relevance to our work , Maass ( 1994 ) constructed an example of a QNN architecture with VC dimension on the order of the number of weights in the network . In contrast , our work shows that there exist QNN architectures with much smaller VC dimensions . Sample compression schemes . Many concept classes with geometrically structured decision regions , such as axis-parallel rectangles , can be trained on a properly chosen size σ subset of an arbitrarily large training dataset without affecting the result . Such a concept class is said to admit a sample compression schemes of size σ , a notion introduced by Littlestone & Warmuth ( 1986 ) who showed that the VC dimension of the class is upper bounded by O ( σ ) . Furthermore , the authors posed the Sample Compression Conjecture . See Moran & Yehudayoff ( 2016 ) for the best known partial result and an extensive review of research in this area . Besides the conjecture , sample compression schemes have also been applied to other long-standing problems in learning theory ( Hanneke et al. , 2019 ; Bousquet et al. , 2020 ; Ashtiani et al. , 2020 ) . To the best of our knowledge , our work is the first to apply sample compression schemes to neural networks . 2 NOTATIONS . The set of real numbers is denoted R. The unit interval is denoted [ 0 , 1 ] . For an integer k ≥ 1 , let [ k ] = { 1 , . . . , k } . We use X to denote the feature space , which in this work will either be Rd or [ 0 , 1 ] d where d ≥ 1 is the ambient dimension/number of features . Denote by I { input } the indicator function which returns 1 if input is true and 0 otherwise . The sign function is given by σsgn ( t ) = I { t ≥ 0 } − I { t < 0 } . For vector inputs , σsgn applies entry-wise . The set of labels for binary classification is denoted B : = { ±1 } . Joint distributions on X × B are denoted by P , whereX , Y ∼ P denotes a random instance-label pair distributed according to P . Let f : X → B be a binary classifier . The risk with respect to P is denoted byRP ( f ) : = P ( f ( X ) 6= Y ) . For an integer n ≥ 1 , the empirical risk is the random variable R̂P , n ( f ) : = 1n ∑n i=1 I { f ( Xi ) 6= Yi } , where ( X1 , Y1 ) , . . . , ( Xn , Yn ) ∼ P are i.i.d . The Bayes risk inff : X→BRP ( f ) with respect to P is denoted by R∗P . Let f , g : { 1 , 2 , . . . } → R≥0 be nonnegative functions on the natural numbers . We write f g if there exists α , β > 0 such that for all n = 1 , 2 , . . . we have αg ( n ) ≤ f ( n ) ≤ βg ( n ) .

Summary: This paper considers VC dimension for a class of partially quantized networks referred to as hyperplane augmented neural networks (HANNs). The architecture of this class of networks consists of a first hidden layer of width r and a second hidden layer of width k with Boolean (sign) activations. The name HANNs seems to be inspired by this Boolean layer which introduces a hyperplane arrangement. An upper bound of O(k^r) is established on the VC dimension of this network that is independent of the rest of the network (a Boolean function). In addition to this result on the VC dimension of HANNs, a result is proven on their representation power. In particular, it is shown that HANNs achieve the minimax rate for classification when the posterior probability is Lipschitz.