An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks

√ n ) neurons and Õ ( dδ + n ) weights are sufficient . Our construction uses Gaussian random weights only in the first layer , while all the subsequent layers use binary or integer weights . We also prove new lower bounds by connecting memorization in neural networks to the purely geometric problem of separating n points on a sphere using hyperplanes . 1 Introduction . The current paradigm of training neural networks is to train the networks until they fit the training dataset perfectly ( Zhang et al. , 2017 ; Arpit et al. , 2017 ) . This means that the network is able to output the exact label for every sample in the training set , a phenomenon known as interpolation . Quite interestingly , modern deep networks have been known to be powerful enough to interpolate even randomized labels ( Zhang et al. , 2017 ; Liu et al. , 2020 ) , a phenomenon that is usually referred to as memorization ( Yun et al. , 2019 ; Vershynin , 2020 ; Bubeck et al. , 2020 ) , where the networks can interpolate any arbitrary labeling of the dataset . Given that memorization is a common phenomenon in modern deep learning , a reasonable question to ask is that of how big a neural network needs to be so that it can memorize a dataset of n points in d dimensions . This question has been of interest since the 60s ( Cover , 1965 ; Baum , 1988 ; Mitchison & Durbin , 1989 ; Sontag , 1990 ; Huang et al. , 1991 ; Sartori & Antsaklis , 1991 ) . In particular , Baum ( 1988 ) proved that a single hidden layer threshold network withO ( max ( n , d ) ) weights andO ( dn/de ) neurons can memorize any set of n points in d dimensions , as long as they are in general position.1 Baum asked if going deeper could increase the memorization power of threshold networks , and in particular if one could reduce the number of neurons needed for memorization to O ( √n ) . While for ReLU activated networks , Yun et al . ( 2019 ) were able to prove that indeed onlyO ( √n ) neurons were sufficient for memorization using deeper networks , the same question remained open for threshold networks . 1The “ general position ” assumption of n vectors in Rd means that any collection of d of these n vectors are linearly independent . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Recently , Vershynin ( 2020 ) was able to answer Baum ’ s question in the positive by proving that indeed Õ ( e1/δ2 +√n ) neurons were sufficient to memorize a set of n points on a unit sphere separated by a distance at least δ . Many recent works study the memorization power of neural networks under separation assumptions , e.g. , Bubeck et al . ( 2020 ) ; Vershynin ( 2020 ) ; Park et al . ( 2020 ) . One reason why before Vershynin ’ s work , it was unclear whether going deeper was helpful for threshold networks was that unlike ReLU or sigmoid functions , the threshold activation function , i.e. , σ ( z ) = 1z≥0 , prohibits neurons from passing ‘ amplitude information ’ to the next layer . Although the dependence ofO ( √ n ) on the number of neurons was achieved by the work of Vershynin , it is unclear that the exponential dependence on distance and the requirement that the points lie on a sphere is fundamental . In this work , we lift the spherical requirement and offer an exponential improvement on the dependence on δ. Theorem 1 . ( Informal ) There exists a threshold neural network with Õ ( 1 δ + √ n ) neurons and Õ ( d δ + n ) weights that can memorize any δ-separated dataset of size n in d dimensions . Please see Definition 2 for the formal definition of δ-separation ; informally , it means that all the points have bounded norm and a distance of at least δ between them . Comparing the theorem above with Vershynin ’ s result , we see that firstly we have reduced the dependence on δ from exponential to nearly linear ; and secondly , in our upper bound for the number of weights , the δ and n terms appear in summation rather than product ( ignoring the logarithmic factors ) . Further , note that Vershynin ( 2020 ) needs the points to lie on a sphere , whereas we only need them to have a bounded ` 2 norm . We have compared our results with the existing work in Table 1 . Our construction has O ( log lognδ ) layers , of which only the first layer in our construction has real weights , which are i.i.d Gaussian . This layer has O ( lognδ ) neurons and O ( d logn δ ) weights . All the other layers have integer weights which are bounded and are on the of order O ( log n ) . Baum ( 1988 ) also proved that there exists a dataset of n points such that any threshold neural network would need at least dn/de neurons in the first layer to memorize it . However , the distance for this dataset is O ( 1/n ) .2 Later , Baum & Haussler ( 1989 ) proved VC-dimension bounds which imply that min { √n , n/d } neurons are necessary in a network to memorize a dataset of n points in d dimensions , and this bound is independent of the minimum distance δ . Our upper bound ( Theorem 1 ) shows that if δ = Ω ( 1/n ) , then we can memorize with only O ( lognδ ) neurons in the first layer . To complement our upper bound , we introduce a new , δ-dependent lower bound below : Theorem 2 . ( Informal ) There exists a δ-separated dataset of size n ∈ [ d2 δ , ( 1 δ ) d 2 ] such that any threshold network that can memorize it needs Ω̃ ( 1√ δ ) neurons in the first layer . The rest of the paper is divided into 7 sections . In Section 2 , we provide the related works for memorization using neural networks . We provide definitions and notation in Section 3 and our main results in Section 4 . Then , we briefly explain the constructions of our upper bound in Section 5 . 2Here , we have rescaled the dataset to have maximum norm of any sample to be 1 . This is done to make the dataset consistent with our minimum distance definition ( see Assumption 2 ) . The assumption in Park et al . ( 2020 ) also uses this kind of normalization . Before exploring lower bounds for threshold networks , we provide sharp bounds on the minimum parameterization needed for any model ( not necessarily a neural network ) to memorize a δ-separated dataset , in Section 6 . We discuss our lower bound for threshold networks in Section 7 . Finally , we conclude our paper with a brief conclusion in Section 8 . 2 Related works . Memorization with threshold activation . There is a rich history of research into the memorization capacity of threshold networks . Baum ( 1988 ) showed that a single hidden layer network with O ( max ( n , d ) ) weights andO ( dn/de ) neurons are sufficient to memorize a dataset , if the samples are in general positions . Baum also provided an instance of a dataset , where any threshold based network aiming to memorize the dataset would need at least dn/de neurons in the first layer . However , the construction was such that δ scaled inversely with n. In this work , we show that for any minimum distance 0 < δ ≤ 12 , the number of neurons in the first layer can be much smaller . Baum also proved , using the Function Counting Theorem ( Cover , 1965 ) , that O ( n/ log n ) connections are needed in any neural network that aims to memorize points in general position , regardless of the minimum distance . Huang et al . ( 1991 ) and Sartori & Antsaklis ( 1991 ) proved that threshold networks could memorize any arbitrary set of points , that is , the points need not be in general position . However , these results needed ( n− 1 ) d weights and n− 1 neurons in the networks for memorization . Mitchison & Durbin ( 1989 ) and Kowalczyk ( 1997 ) provided upper and lower bounds when the requirement of memorizing the dataset exactly was relaxed to allow a recall error of up 50 % . Sontag ( 1990 ) provided evidence that suggested that adding skip connections can double the memorization capacity of single hidden layer networks . Kotsovsky et al . ( 2020 ) extended Baum ’ s result to multiclass classification setting , using bithresholding activation functions . Recently , Vershynin ( 2020 ) was able to show thatO ( e1/δ2 ( d+√n ) +n ) weights andO ( e1/δ2 +√n ) neurons are sufficient to memorize a set of n points on the unit sphere , separated by a distance of at least δ . This answered the question by Baum ( 1988 ) , which was whether the number of neurons could be reduced to O ( √n ) by considering deeper networks . Note that Vershynin ’ s and Baum ’ s assumptions are fundamentally different and neither is stronger than the other . For example , Baum ’ s construction would still need only O ( max ( n , d ) ) weights even if the points are infinitesimally close to each other , whereas Vershynin ’ s construction would grow as δ decreases . On the other hand , Vershynin ’ s construction works even if all the points lie on a low dimensional hyperplane , whereas Baum ’ s general position assumption would get violated in this situation . Note that naïvely extending Baum ’ s construction to the case where the points lie on a very low dimensional hyperplane would result in Õ ( nd ) weights and Õ ( n ) neurons3 , which is identical to the bounds by Huang et al . ( 1991 ) . Memorization with ReLU and sigmoid activation . Due to the popularity of the ReLU activation function , there has also been an interest in their memorization capacity . Zhang et al . ( 2017 ) proved that a single hidden layer ReLU network with 2n+d weights and n neurons can memorize n arbitrary points . Hardt & Ma ( 2017 ) proved that ReLU networks with residual connections also need only O ( n ) weights to memorize n points . Yun et al . ( 2019 ) were able to prove that for ReLU , in fact , one needs only O ( √n ) neurons to memorize n points . Quite interestingly , the construction by Vershynin ( 2020 ) works for both threshold activated networks ( as described above ) as well as ReLU activated networks ; and gives the same bounds for both architectures . Bubeck et al . ( 2020 ) show that under some minimum distance assumptions , ReLU networks can memorize ( even real labels ) with O ( n/d ) neurons such that the total weight ( Bartlett , 1998 ) of the network is O ( √n ) , which is proved to be optimal for single hidden layer networks . Huang ( 2003 ) proved that a two-hidden layer network with O ( √n ) sigmoid activated neurons suffices to memorize n points . Recently , Park et al . ( 2020 ) have shown that ReLU or sigmoid activated networks with onlyO ( n2/3 + log ( 1/δ ) ) weights are sufficient to memorize n points separated by a normalized distance of δ . Over parameterization and generalization . A large body of recent work attempts to explain why large neural networks , which are capable memorizers , can also generalize well , e.g. , Neyshabur et al . ( 2018 , 2019 ) ; Belkin et al . ( 2020 ) . In this work , we only study memorization , and omit any discussion about generalization . We do not believe our results explain anything about generalization . 3Please see Appendix A for a detailed explanation .

Summary: This paper considers the memorization of data using deep neural networks and aims to minimize both the number of neurons and the number of connections in the network, where the activation function is a step function.  The main result is a construction of a deep and relatively sparse network that memorizes arbitrary labels of 'well-separated' data sets. If the size of the data set is $n$, the number of neurons in the network is proportional to $\sqrt{n}$ plus a linear dependence on the well-separateness parameter. This is an exponential improvement over a recent result of Vershynin (which itself generalized a classical result of Baum from the 80's), which had an exponential dependence on the well-separateness parameter. The authors also complement their work by proving some lower bounds, which, in particular, show that some polynomial dependence on the well-separateness parameter is necessary.