An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks

√ n ) neurons and Õ ( dδ + n ) weights are sufficient . Our construction uses Gaussian random weights only in the first layer , while all the subsequent layers use binary or integer weights . We also prove new lower bounds by connecting memorization in neural networks to the purely geometric problem of separating n points on a sphere using hyperplanes . 1 Introduction . The current paradigm of training neural networks is to train the networks until they fit the training dataset perfectly ( Zhang et al. , 2017 ; Arpit et al. , 2017 ) . This means that the network is able to output the exact label for every sample in the training set , a phenomenon known as interpolation . Quite interestingly , modern deep networks have been known to be powerful enough to interpolate even randomized labels ( Zhang et al. , 2017 ; Liu et al. , 2020 ) , a phenomenon that is usually referred to as memorization ( Yun et al. , 2019 ; Vershynin , 2020 ; Bubeck et al. , 2020 ) , where the networks can interpolate any arbitrary labeling of the dataset . Given that memorization is a common phenomenon in modern deep learning , a reasonable question to ask is that of how big a neural network needs to be so that it can memorize a dataset of n points in d dimensions . This question has been of interest since the 60s ( Cover , 1965 ; Baum , 1988 ; Mitchison & Durbin , 1989 ; Sontag , 1990 ; Huang et al. , 1991 ; Sartori & Antsaklis , 1991 ) . In particular , Baum ( 1988 ) proved that a single hidden layer threshold network withO ( max ( n , d ) ) weights andO ( dn/de ) neurons can memorize any set of n points in d dimensions , as long as they are in general position.1 Baum asked if going deeper could increase the memorization power of threshold networks , and in particular if one could reduce the number of neurons needed for memorization to O ( √n ) . While for ReLU activated networks , Yun et al . ( 2019 ) were able to prove that indeed onlyO ( √n ) neurons were sufficient for memorization using deeper networks , the same question remained open for threshold networks . 1The “ general position ” assumption of n vectors in Rd means that any collection of d of these n vectors are linearly independent . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Recently , Vershynin ( 2020 ) was able to answer Baum ’ s question in the positive by proving that indeed Õ ( e1/δ2 +√n ) neurons were sufficient to memorize a set of n points on a unit sphere separated by a distance at least δ . Many recent works study the memorization power of neural networks under separation assumptions , e.g. , Bubeck et al . ( 2020 ) ; Vershynin ( 2020 ) ; Park et al . ( 2020 ) . One reason why before Vershynin ’ s work , it was unclear whether going deeper was helpful for threshold networks was that unlike ReLU or sigmoid functions , the threshold activation function , i.e. , σ ( z ) = 1z≥0 , prohibits neurons from passing ‘ amplitude information ’ to the next layer . Although the dependence ofO ( √ n ) on the number of neurons was achieved by the work of Vershynin , it is unclear that the exponential dependence on distance and the requirement that the points lie on a sphere is fundamental . In this work , we lift the spherical requirement and offer an exponential improvement on the dependence on δ. Theorem 1 . ( Informal ) There exists a threshold neural network with Õ ( 1 δ + √ n ) neurons and Õ ( d δ + n ) weights that can memorize any δ-separated dataset of size n in d dimensions . Please see Definition 2 for the formal definition of δ-separation ; informally , it means that all the points have bounded norm and a distance of at least δ between them . Comparing the theorem above with Vershynin ’ s result , we see that firstly we have reduced the dependence on δ from exponential to nearly linear ; and secondly , in our upper bound for the number of weights , the δ and n terms appear in summation rather than product ( ignoring the logarithmic factors ) . Further , note that Vershynin ( 2020 ) needs the points to lie on a sphere , whereas we only need them to have a bounded ` 2 norm . We have compared our results with the existing work in Table 1 . Our construction has O ( log lognδ ) layers , of which only the first layer in our construction has real weights , which are i.i.d Gaussian . This layer has O ( lognδ ) neurons and O ( d logn δ ) weights . All the other layers have integer weights which are bounded and are on the of order O ( log n ) . Baum ( 1988 ) also proved that there exists a dataset of n points such that any threshold neural network would need at least dn/de neurons in the first layer to memorize it . However , the distance for this dataset is O ( 1/n ) .2 Later , Baum & Haussler ( 1989 ) proved VC-dimension bounds which imply that min { √n , n/d } neurons are necessary in a network to memorize a dataset of n points in d dimensions , and this bound is independent of the minimum distance δ . Our upper bound ( Theorem 1 ) shows that if δ = Ω ( 1/n ) , then we can memorize with only O ( lognδ ) neurons in the first layer . To complement our upper bound , we introduce a new , δ-dependent lower bound below : Theorem 2 . ( Informal ) There exists a δ-separated dataset of size n ∈ [ d2 δ , ( 1 δ ) d 2 ] such that any threshold network that can memorize it needs Ω̃ ( 1√ δ ) neurons in the first layer . The rest of the paper is divided into 7 sections . In Section 2 , we provide the related works for memorization using neural networks . We provide definitions and notation in Section 3 and our main results in Section 4 . Then , we briefly explain the constructions of our upper bound in Section 5 . 2Here , we have rescaled the dataset to have maximum norm of any sample to be 1 . This is done to make the dataset consistent with our minimum distance definition ( see Assumption 2 ) . The assumption in Park et al . ( 2020 ) also uses this kind of normalization . Before exploring lower bounds for threshold networks , we provide sharp bounds on the minimum parameterization needed for any model ( not necessarily a neural network ) to memorize a δ-separated dataset , in Section 6 . We discuss our lower bound for threshold networks in Section 7 . Finally , we conclude our paper with a brief conclusion in Section 8 . 2 Related works . Memorization with threshold activation . There is a rich history of research into the memorization capacity of threshold networks . Baum ( 1988 ) showed that a single hidden layer network with O ( max ( n , d ) ) weights andO ( dn/de ) neurons are sufficient to memorize a dataset , if the samples are in general positions . Baum also provided an instance of a dataset , where any threshold based network aiming to memorize the dataset would need at least dn/de neurons in the first layer . However , the construction was such that δ scaled inversely with n. In this work , we show that for any minimum distance 0 < δ ≤ 12 , the number of neurons in the first layer can be much smaller . Baum also proved , using the Function Counting Theorem ( Cover , 1965 ) , that O ( n/ log n ) connections are needed in any neural network that aims to memorize points in general position , regardless of the minimum distance . Huang et al . ( 1991 ) and Sartori & Antsaklis ( 1991 ) proved that threshold networks could memorize any arbitrary set of points , that is , the points need not be in general position . However , these results needed ( n− 1 ) d weights and n− 1 neurons in the networks for memorization . Mitchison & Durbin ( 1989 ) and Kowalczyk ( 1997 ) provided upper and lower bounds when the requirement of memorizing the dataset exactly was relaxed to allow a recall error of up 50 % . Sontag ( 1990 ) provided evidence that suggested that adding skip connections can double the memorization capacity of single hidden layer networks . Kotsovsky et al . ( 2020 ) extended Baum ’ s result to multiclass classification setting , using bithresholding activation functions . Recently , Vershynin ( 2020 ) was able to show thatO ( e1/δ2 ( d+√n ) +n ) weights andO ( e1/δ2 +√n ) neurons are sufficient to memorize a set of n points on the unit sphere , separated by a distance of at least δ . This answered the question by Baum ( 1988 ) , which was whether the number of neurons could be reduced to O ( √n ) by considering deeper networks . Note that Vershynin ’ s and Baum ’ s assumptions are fundamentally different and neither is stronger than the other . For example , Baum ’ s construction would still need only O ( max ( n , d ) ) weights even if the points are infinitesimally close to each other , whereas Vershynin ’ s construction would grow as δ decreases . On the other hand , Vershynin ’ s construction works even if all the points lie on a low dimensional hyperplane , whereas Baum ’ s general position assumption would get violated in this situation . Note that naïvely extending Baum ’ s construction to the case where the points lie on a very low dimensional hyperplane would result in Õ ( nd ) weights and Õ ( n ) neurons3 , which is identical to the bounds by Huang et al . ( 1991 ) . Memorization with ReLU and sigmoid activation . Due to the popularity of the ReLU activation function , there has also been an interest in their memorization capacity . Zhang et al . ( 2017 ) proved that a single hidden layer ReLU network with 2n+d weights and n neurons can memorize n arbitrary points . Hardt & Ma ( 2017 ) proved that ReLU networks with residual connections also need only O ( n ) weights to memorize n points . Yun et al . ( 2019 ) were able to prove that for ReLU , in fact , one needs only O ( √n ) neurons to memorize n points . Quite interestingly , the construction by Vershynin ( 2020 ) works for both threshold activated networks ( as described above ) as well as ReLU activated networks ; and gives the same bounds for both architectures . Bubeck et al . ( 2020 ) show that under some minimum distance assumptions , ReLU networks can memorize ( even real labels ) with O ( n/d ) neurons such that the total weight ( Bartlett , 1998 ) of the network is O ( √n ) , which is proved to be optimal for single hidden layer networks . Huang ( 2003 ) proved that a two-hidden layer network with O ( √n ) sigmoid activated neurons suffices to memorize n points . Recently , Park et al . ( 2020 ) have shown that ReLU or sigmoid activated networks with onlyO ( n2/3 + log ( 1/δ ) ) weights are sufficient to memorize n points separated by a normalized distance of δ . Over parameterization and generalization . A large body of recent work attempts to explain why large neural networks , which are capable memorizers , can also generalize well , e.g. , Neyshabur et al . ( 2018 , 2019 ) ; Belkin et al . ( 2020 ) . In this work , we only study memorization , and omit any discussion about generalization . We do not believe our results explain anything about generalization . 3Please see Appendix A for a detailed explanation .

Summary: The paper considers a feed-forward neural network that takes inputs in $\mathbb{R}^d$ and outputs binary labels. The neural network has $L$ layers, with each layers consisting of a bias term and the activation function is the threshold function. The paper considers the following question: What is the minimum size of a threshold network so that it can memorize a dataset of $n$ samples in $\mathbb{R}^d$. Memorization refers to learning the parameters of the threshold network so that it can accurately output labels of the $n$ samples, for any arbitrary labeling. The paper shows that a deep threshold network can memorize $n$ points in $d$ dimension using $O(\frac{1}{\delta} + \sqrt{n})$ neurons $O(\frac{d}{\delta} +n)$ weights (up to log factors), where $\delta$ is the minimum distance between the points (considered in angular metric and $\ell_2$-metric).