Provable Memorization via Deep Neural Networks using Sub-linear Parameters

1 INTRODUCTION . The modern trend of over-parameterizing neural networks has shifted the focus of deep learning theory from analyzing their expressive power toward understanding the generalization capabilities of neural networks . While the celebrated universal approximation theorems state that over-parameterization enables us to approximate the target function with a smaller error ( Cybenko , 1989 ; Pinkus , 1999 ) , the theoretical gain is too small to satisfactorily explain the observed benefits of over-parameterizing already-big networks . Instead of “ how well can models fit , ” the question of “ why models do not overfit ” has become the central issue ( Zhang et al. , 2017 ) . Ironically , a recent breakthrough on the phenomenon known as the double descent ( Belkin et al. , 2019 ; Nakkiran et al. , 2020 ) suggests that answering the question of “ how well can models fit ” is in fact an essential element in fully characterizing their generalization capabilities . In particular , the double descent phenomenon characterizes two different phases according to the capability/incapability of the network size for memorizing training samples . If the network size is insufficient for memorization , the traditional bias-variance trade-off occurs . However , after the network reaches the capacity that memorizes the dataset , i.e. , “ interpolation threshold , ” larger networks exhibit better generalization . Under this new paradigm , identifying the minimum size of networks for memorizing finite input-label pairs becomes a key issue , rather than function approximation that considers infinite inputs . The memory capacity of neural networks is relatively old literature , where researchers have studied the minimum number of parameters for memorizing arbitrary N input-label pairs . Existing results showed that O ( N ) parameters are sufficient for various activation functions ( Baum , 1988 ; Huang and Babri , 1998 ; Huang , 2003 ; Yun et al. , 2019 ; Vershynin , 2020 ) . On the other hand , Sontag ( 1997 ) established the negative result that for any network using analytic definable activation functions with o ( N ) parameters , there exists a set of N input-label pairs that the network can not memorize . The sub-linear number of parameters also appear in a related topic , namely the VC-dimension of neural networks . It has been proved that there exists a set of N inputs such that a neural network with o ( N ) parameters can “ shatter , ” i.e. , memorize arbitrary labels ( Maass , 1997 ; Bartlett et al. , 2019 ) . Comparing the two results on o ( N ) parameters , Sontag ( 1997 ) showed that not all sets of N inputs can be memorized for arbitrary labels , whereas Bartlett et al . ( 2019 ) showed that at least one set of N inputs can be shattered . This suggests that there may be a reasonably large family of N input-label pairs that can be memorized with o ( N ) parameters , which is our main interest . 1.1 SUMMARY OF RESULTS . In this paper , we identify a mild condition satisfied by many practical datasets , and show that o ( N ) parameters suffice for memorizing such datasets . In order to bypass the negative result by Sontag ( 1997 ) , we introduce a condition to the set of inputs , called the ∆-separateness . Definition 1 . For a set X ⊂ Rdx , we say X is ∆-separated if sup x , x′∈X : x 6=x′ ‖x− x′‖2 < ∆× inf x , x′∈X : x 6=x′ ‖x− x′‖2 . This condition requires that the ratio of the maximum distance to the minimum distance between distinct points is bounded by ∆ . Note that the condition is milder when ∆ is bigger . By Definition 1 , any given finite set of ( distinct ) inputs is ∆-separated for some ∆ , so one might ask why ∆- separateness is different from having distinct inputs in a dataset . The key difference is that even if the number of data points N grows , the ratio of the maximum to the minimum should remain bounded by ∆ . Given the discrete nature of computers , there are many practical datasets that satisfy ∆-separateness , as we will see shortly . Also , this condition is more general than the minimum distance assumptions ( ∀i , ‖xi‖2 = 1 , ∀i 6= j , ‖xi − xj‖2 ≥ ρ > 0 ) that are employed in existing theoretical results ( Hardt and Ma , 2017 ; Vershynin , 2020 ) . To see this , note that the minimum distance assumption implies 2/ρ-separateness . In our theorem statements , we will use the phrase “ ∆-separated set of input-label pairs ” for denoting the set of inputs is ∆-separated . In our main theorem sketched below , we prove the sufficiency of o ( N ) parameters for memorizing any ∆-separated set of N pairs ( i.e. , any ∆-separated set of N inputs with arbitrary labels ) even for large ∆ . More concretely , our result is of the following form : Theorem 1 ( Informal ) . For any w ∈ ( 23 , 1 ] , there exists aO ( N 2−2w/ logN+log ∆ ) -layer , O ( Nw+ log ∆ ) -parameter fully-connected network with sigmoidal or RELU activation that can memorize any ∆-separated set of N input-label pairs . We note that log has base 2 . Theorem 1 states that if the number of layers increases with the number of pairs N , then any ∆-separated set of N pairs can be memorized by a network with o ( N ) parameters . Here , we can check from Definition 1 that the log ∆ term does not usually dominate the depth or the number of parameters , especially for modern deep architectures and practical datasets . For example , it is easy to check that any dataset consisting of 3-channel images ( values from { 0 , 1 , . . . , 255 } ) of size a× b satisfies log ∆ < ( 9 + 12 log ( ab ) ) ( e.g. , log ∆ < 17 for the ImageNet dataset ) , which is often much smaller than the depth of modern deep architectures . For practical datasets , we can show that networks with parameters fewer than the number of pairs can successfully memorize the dataset . For example , in order to perfectly classify one million images in ImageNet dataset1 with 1000 classes , our result shows that 0.7 million parameters are sufficient . The improvement is more significant for large datasets . To memorize 15.8 million bounding boxes in Open Images V62 with 600 classes , our result shows that only 4.5 million parameters suffice . Theorem 1 improves the sufficient number of parameters for memorizing a large class of N pairs ( i.e. , 2O ( N w ) -separated ) from O ( N ) down to O ( Nw ) for any w ∈ ( 23 , 1 ) , for deep networks . Then , it is natural to ask whether the depth increasing with N is necessary for memorization with a sub-linear number of parameters . The following existing result on the VC-dimension implies that this is indeed necessary for memorization with o ( N/ logN ) parameters , at least for RELU networks . Theorem [ Bartlett et al . ( 2019 ) ] . ( Informal ) For L-layer RELU networks , Ω ( N/ ( L logN ) ) parameters are necessary for memorizing at least a single set of N inputs with arbitrary labels . The above theorem implies that for RELU networks of constant depth , Ω ( N/ logN ) parameters are necessary for memorizing at least one set of N inputs with arbitrary labels . In contrast , by increasing depth with N , Theorem 1 shows that there is a large class of datasets that can be memorized with o ( N/ logN ) parameters . Combining these two results , one can conclude that increasing depth is necessary and sufficient for memorizing a large class of N pairs with o ( N/ logN ) parameters . Given that the depth is critical for the memorization power , is the width also critical ? We prove that it is not the case , via the following theorem . Theorem 2 ( Informal ) . For a fully-connected network of width 3 with a sigmoidal or RELU activation function , O ( N2/3 + log ∆ ) parameters ( i.e. , layers ) suffice for memorizing any ∆-separated set of N input-label pairs . Theorem 2 states that under 2O ( N 2/3 ) -separateness of inputs , the network width does not necessarily have to increase with N for memorization with sub-linear parameters . Furthermore , it shows that 1http : //www.image-net.org/ 2https : //storage.googleapis.com/openimages/web/index.html even a surprisingly narrow network of width 3 has a superior memorization power than a fixed-depth network , requiring only O ( N2/3 ) parameters . Theorems 1 and 2 show the existence of network architectures that memorize N points with o ( N ) parameters , under the condition of ∆-separateness . This means that these theorems do not answer the question of how many such data points can a given network memorize . We provide generic criteria for identifying the maximum number of points given general networks ( Theorem 3 ) . In a nutshell , our criteria indicate that to memorize more pairs under the same budget for the number of parameters , a network must have a deep and narrow architecture at the final layers of the network . In contrast to the prior results that the number of arbitrary pairs that can be memorized is at most proportional to the number of parameters ( Yamasaki , 1993 ; Yun et al. , 2019 ; Vershynin , 2020 ) , our criteria successfully incorporate with the characteristics of datasets , the number of parameters , and the architecture , which enable us to memorize ∆-separated datasets with number of pairs super-linear in the number of parameters . Finally , we provide empirical results corroborating our theoretical findings that deep networks often memorize better than their shallow counterparts with a similar number of parameters . Here , we emphasize that better memorization power does not necessarily imply better generalization . We indeed observe that shallow and wide networks often generalize better than deep and narrow networks , given the same ( or similar ) training accuracy . Organization . We first introduce related works in Section 2 . In Section 3 , we introduce necessary notation and the problem setup . We formally state our main results and discuss them in Section 4 . In Section 6 , we provide empirical observations on the effect of depth and width in neural networks . Finally , we conclude the paper in Section 7 . 2 RELATED WORKS . 2.1 NUMBER OF PARAMETERS FOR MEMORIZATION . Sufficient number of parameters for memorization . Identifying the sufficient number of parameters for memorizing arbitrary N pairs has a long history . Earlier works mostly focused on bounding the number of hidden neurons of shallow networks for memorization . Baum ( 1988 ) proved that for 2-layer STEP3 networks , O ( N ) hidden neurons ( i.e. , O ( N ) parameters ) are sufficient for memorizing arbitrary N pairs when inputs are in general position . Huang and Babri ( 1998 ) showed that the same bound holds for any bounded and nonlinear activation function σ satisfying that either limx→−∞ σ ( x ) or limx→∞ σ ( x ) exists , without any condition on inputs . The O ( N ) bounds on the number of hidden neurons was improved to O ( √ N ) by exploiting an additional hidden layer by Huang ( 2003 ) ; nevertheless , this construction still requires O ( N ) parameters . With the advent of deep learning , the study has been extended to modern activation functions and deeper architectures . Zhang et al . ( 2017 ) proved that O ( N ) hidden neurons are sufficient for 2-layer RELU networks to memorize arbitrary N pairs . Yun et al . ( 2019 ) showed that for deep RELU ( or hard tanh ) networks having at least 3 layers , O ( N ) parameters are sufficient . Vershynin ( 2020 ) proved a similar result for STEP ( or RELU ) networks with an additional logarithmic factor , i.e. , Õ ( N ) parameters are sufficient , for memorizing arbitrary { xi : ‖xi‖2 = 1 } Ni=1 satisfying ‖xi − xj‖22 = Ω ( log log dmax log dmin ) and N , dmax = eO ( d 1/5 min ) where dmax and dmin denote the maximum and the minimum hidden dimensions , respectively . In addition , the memorization power of modern network architectures has also been studied . Hardt and Ma ( 2017 ) showed that RELU networks consisting of residual blocks with O ( N ) hidden neurons can memorize arbitrary { xi : ‖xi‖2 = 1 } Ni=1 satisfying ‖xi − xj‖2 ≥ ρ for some absolute constant ρ > 0 . Nguyen and Hein ( 2018 ) studied a broader class of layers and proved that O ( N ) hidden neurons suffice for convolutional neural networks consisting of fully-connected , convolutional , and max-pooling layers for memorizing arbitrary N pairs having different patches . Necessary number of parameters for memorization . On the other hand , the necessary number of parameters for memorization has also been studied . Sontag ( 1997 ) showed that for any neural network using analytic definable activation functions , Ω ( N ) parameters are necessary for memorizing arbitrary 3STEP denotes the binary threshold activation function : x 7→ 1 [ x ≥ 0 ] . N pairs . Namely , given any network using analytic definable activation with o ( N ) parameters , there exists a set of N pairs that the network can not memorize . The Vapnik-Chervonenkis ( VC ) dimension is also closely related to the memorization power of neural networks . While the memorization power studies the number of parameters for memorizing arbitrary N pairs , the VC-dimension studies the number of parameters for memorizing at least a single set of N inputs with arbitrary labels . Hence , it naturally provides the lower bound on the necessary number of parameters for memorizing arbitrary N pairs . The VC-dimension of neural networks has been studied for various types of activation functions . For memorizing at least a single set of N inputs with arbitrary labels , it is known that Θ ( N/ logN ) parameters are necessary ( Baum and Haussler , 1989 ) and sufficient ( Maass , 1997 ) for STEP networks . Similarly , Karpinski and Macintyre ( 1997 ) proved that Ω ( √ N/U ) parameters are necessary for sigmoid networks of U neurons . Recently , Bartlett et al . ( 2019 ) showed that Θ ( N/ ( L̄ logN ) ) parameters are necessary and sufficient for L-layer networks using any piecewise linear activation function where L̄ : = 1WL ∑L ` =1W ` and W ` denotes the number of parameters up to the ` -th layer .

Summary: The paper studies the memorization capacity of deep networks as a function of the number of parameters. Many prior works have shown that to memorize $N$ examples $O(N)$ parameters are sufficient and that to memorize any set of $N$ examples $\Omega(N)$ parameters are necessary. This work shows that under very mild and commonly satisfied conditions, $O(N^{\frac{2}{3}})$ parameters and layers are sufficient for memorizing $N$ examples, a significant improvement over prior results. Additionally, they show that even very narrow width-bounded networks can memorize with sub-linear parameters, but the same is not true for depth-bounded networks, demonstrating a new capability unlocked by deeper networks. Finally, they characterize the properties sufficient for memorizing $N$ examples by a given network.