VC dimension of partially quantized neural networks in the overparametrized regime

Vapnik-Chervonenkis ( VC ) theory has so far been unable to explain the small generalization error of overparametrized neural networks . Indeed , existing applications of VC theory to large networks obtain upper bounds on VC dimension that are proportional to the number of weights , and for a large class of networks , these upper bound are known to be tight . In this work , we focus on a subclass of partially quantized networks that we refer to as hyperplane arrangement neural networks ( HANNs ) . Using a sample compression analysis , we show that HANNs can have VC dimension significantly smaller than the number of weights , while being highly expressive . In particular , empirical risk minimization over HANNs in the overparametrized regime achieves the minimax rate for classification with Lipschitz posterior class probability . We further demonstrate the expressivity of HANNs empirically . On a panel of 121 UCI datasets , overparametrized HANNs match the performance of state-of-the-art full-precision models . 1 INTRODUCTION . Neural networks have become an indispensable tool for machine learning practitioners , owing to their impressive performance especially in vision and natural language processing ( Goodfellow et al. , 2016 ) . In practice , neural networks are often applied in the overparametrized regime and are capable of fitting even random labels ( Zhang et al. , 2021 ) . Evidently , these overparametrized models perform well on real world data despite their ability to grossly overfit , a phenomenon that has been dubbed “ the generalization puzzle ” ( Nagarajan & Kolter , 2019 ) . Toward solving this puzzle , several research directions have flourished and offer potential explanations , including implicit regularization ( Chizat & Bach , 2020 ) , interpolation ( Chatterji & Long , 2021 ) , and benign overfitting ( Bartlett et al. , 2020 ) . So far , VC theory has not been able to explain the puzzle , because existing bounds on the VC dimensions of neural networks are on the order of the number of weights ( Maass , 1994 ; Bartlett et al. , 2019 ) . It remains unknown whether there exist neural network architectures capable of modeling rich set of classfiers with low VC dimension . The focus of this work is on a class of neural networks with threshold activation that we refer to as hyperplane arrangement neural networks ( HANNs ) . Using the theory of sample compression schemes ( Littlestone & Warmuth , 1986 ) , we show that HANNs can have VC dimension that is significantly smaller than the number of parameters . Furthermore , we apply this result to show that HANNs have high expressivity by proving that HANN classifiers achieve minimax-optimality when the data has Lipschitz posterior class probability in an overparametrized setting . We benchmark the empirical performance of HANNs on a panel of 121 UCI datasets , following several recent neural network and neural tangent kernel works ( Klambauer et al. , 2017 ; Wu et al. , 2018 ; Arora et al. , 2019 ; Shankar et al. , 2020 ) . In particular , Klambauer et al . ( 2017 ) showed that , using a properly chosen activation , overparametrized neural networks perform competitively compared to classical shallow methods on this panel of datasets . Our experiments show that HANNs , a partially-quantized model , match the classification accuracy of the self-normalizing neural network ( Klambauer et al. , 2017 ) and the dendritic neural network ( Wu et al. , 2018 ) , both of which are fullprecision models . 1.1 RELATED WORK . VC dimensions of neural networks . The VC-dimension Vapnik & Chervonenkis ( 1971 ) is a combinatorial measure of the complexity of a concept class , i.e. , a set of classifiers . The Fundamental Theorem of Statistical Learning ( Shalev-Shwartz & Ben-David , 2014 , Theorem 6.8 ) states that a concept class has finite VC-dimension if and only if it is probably approximately correct ( PAC ) learnable , where the VC-dimension is tightly related to the number of samples required for PAC learning . For threshold networks , Cover ( 1965 ) ; Baum & Haussler ( 1989 ) showed a VC-dimension upper bounded of O ( w logw ) , where w is the number of parameters . Maass ( 1994 ) obtained a matching lower bound attained by a network architecture with two hidden layers . More recently , Bartlett et al . ( 2019 ) obtained the upper and lower bounds O ( w ` logw ) and Ω ( w ` log ( w/ ` ) ) respectively for the case when the activation is piecewise linear , where ` is the number of layers . These lower bounds are achieved by somewhat unconventional network architectures . The architectures we consider exclude these and thus we are able to achieve a smaller upper bound on the VC dimensions . The generalization puzzle . In practice , neural networks that achieve state-of-the-art performance use significantly more parameters than samples , a phenomenon that can not be explained by classical VC theory if the VC dimension ≥ number of weights . This has been dubbed the generalization puzzle ( Zhang et al. , 2021 ) . To explain the puzzle , researchers have pursued new directions including margin-based bounds ( Neyshabur et al. , 2017 ; Bartlett et al. , 2017 ) , PAC-Bayes bounds ( Dziugaite & Roy , 2017 ) , and implicit bias of optimization methods ( Gunasekar et al. , 2018 ; Chizat & Bach , 2020 ) . We refer the reader to the recent article by Bartlett et al . ( 2021 ) for a comprehensive coverage of this growing literature . The generalization puzzle is not specific to deep learning . For instance , AdaBoost has been observed to continue to decrease the test error while the VC dimension grows linearly with the number of boosting rounds ( Schapire , 2013 ) . Other learning algorithms that exhibits similarly surprising behavior include random forests ( Wyner et al. , 2017 ) and kernel methods ( Belkin et al. , 2018 ) . Minimax-optimality . Whereas VC theory is distribution-independent , minimax theory is concerned with the question of optimal estimation/classification under distributional assumptions1 . A minimax optimality result shows that the expected excess classification error goes to zero at the fastest rate possible , as the sample size tend to infinity . For neural networks , this often involves a hyperparameter selection scheme in terms of the sample size . Faragó & Lugosi ( 1993 ) show minimax-optimality of ( underparametrized ) neural networks for learning to classify under certain assumptions on the Fourier transform of the data distribution . Schmidt-Hieber ( 2020 ) shows minimax-optimality of s-sparse neural networks for regression over Hölder classes , where at most s = O ( n log n ) network weights are nonzero , and n = the number of training samples . Kim et al . ( 2021 ) extends the results of Schmidt-Hieber ( 2020 ) to the classification setting , remarking that effective optimization under sparsity constraint is lacking . Kohler & Langer ( 2020 ) and Langer ( 2021 ) proved minimax-optimality without the sparsity assumption , however in an underparametrized setting . To the best of our knowledge , our result is the first to establish minimax optimality of overparametrized neural networks without a sparsity assumption . ( Partially ) quantized neural networks . Quantizing some of the weights and/or activations of neural networks has the potential to reduce the high computational burden of neural networks at test time ( Qin et al. , 2020 ) . Many works have focused on the efficient training of quantized neural networks to close the performance gap with full-precision architectures ( Hubara et al. , 2017 ; Rastegari et al. , 2016 ; Lin et al. , 2017 ) . Several works have observed that quantization of the activations , rather than of the weights , leads to a larger accuracy gap ( Cai et al. , 2017 ; Mishra et al. , 2018 ; Kim et al. , 2019 ) . Towards explaining this phenomenon , researchers have focused on understanding the so-called coarse gradient , a term coined by Yin et al . ( 2019 ) , often used in training QNNs as a surrogate for the usual gradient . One commonly used heuristic is the straight-through-estimator ( STE ) first introduced in an online course by Hinton et al . ( 2012 ) . Theory supporting the STE heuristic has recently been studied in Li et al . ( 2017 ) and Yin et al . ( 2019 ) . 1Without distributional assumptions , no classifier can be minimax optimal in light of the No-Free-Lunch Theorem ( Devroye , 1982 ) . QNNs have also been analyzed from other theoretical angles , including mean-field theory ( Blumenfeld et al. , 2019 ) , memory capacity ( Vershynin , 2020 ) , Boolean function representation capacity ( Baldi & Vershynin , 2019 ) and adversarial robustness ( Lin et al. , 2018 ) . Of particular relevance to our work , Maass ( 1994 ) constructed an example of a QNN architecture with VC dimension on the order of the number of weights in the network . In contrast , our work shows that there exist QNN architectures with much smaller VC dimensions . Sample compression schemes . Many concept classes with geometrically structured decision regions , such as axis-parallel rectangles , can be trained on a properly chosen size σ subset of an arbitrarily large training dataset without affecting the result . Such a concept class is said to admit a sample compression schemes of size σ , a notion introduced by Littlestone & Warmuth ( 1986 ) who showed that the VC dimension of the class is upper bounded by O ( σ ) . Furthermore , the authors posed the Sample Compression Conjecture . See Moran & Yehudayoff ( 2016 ) for the best known partial result and an extensive review of research in this area . Besides the conjecture , sample compression schemes have also been applied to other long-standing problems in learning theory ( Hanneke et al. , 2019 ; Bousquet et al. , 2020 ; Ashtiani et al. , 2020 ) . To the best of our knowledge , our work is the first to apply sample compression schemes to neural networks . 2 NOTATIONS . The set of real numbers is denoted R. The unit interval is denoted [ 0 , 1 ] . For an integer k ≥ 1 , let [ k ] = { 1 , . . . , k } . We use X to denote the feature space , which in this work will either be Rd or [ 0 , 1 ] d where d ≥ 1 is the ambient dimension/number of features . Denote by I { input } the indicator function which returns 1 if input is true and 0 otherwise . The sign function is given by σsgn ( t ) = I { t ≥ 0 } − I { t < 0 } . For vector inputs , σsgn applies entry-wise . The set of labels for binary classification is denoted B : = { ±1 } . Joint distributions on X × B are denoted by P , whereX , Y ∼ P denotes a random instance-label pair distributed according to P . Let f : X → B be a binary classifier . The risk with respect to P is denoted byRP ( f ) : = P ( f ( X ) 6= Y ) . For an integer n ≥ 1 , the empirical risk is the random variable R̂P , n ( f ) : = 1n ∑n i=1 I { f ( Xi ) 6= Yi } , where ( X1 , Y1 ) , . . . , ( Xn , Yn ) ∼ P are i.i.d . The Bayes risk inff : X→BRP ( f ) with respect to P is denoted by R∗P . Let f , g : { 1 , 2 , . . . } → R≥0 be nonnegative functions on the natural numbers . We write f g if there exists α , β > 0 such that for all n = 1 , 2 , . . . we have αg ( n ) ≤ f ( n ) ≤ βg ( n ) .

Summary: This paper proposes a network architecture, HANN. HANN contains a hidden layer of binary activations, and can has a VC dimension smaller than the number of network parameters. HANN can be potentially a VC theory for studying the generalization of overparameterized neural networks. HANN is competitive with sota on UCI datasets.