On the Optimal Memorization Power of ReLU Neural Networks

( √ N ) parameters . Known VC-dimension upper bounds imply that memorizing N samples requires Ω ( √ N ) parameters , and hence our construction is optimal up to logarithmic factors . We also give a generalized construction for networks with depth bounded by 1 ≤ L ≤ √ N , for memorizing N samples using Õ ( N/L ) parameters . This bound is also optimal up to logarithmic factors . Our construction uses weights with large bit complexity . We prove that having such a large bit complexity is both necessary and sufficient for memorization with a sub-linear number of parameters . 1 INTRODUCTION . The expressive power of neural networks has been widely studied in many previous works . These works study different aspects of expressiveness , such as the universal approximation property ( Cybenko , 1989 ; Leshno et al. , 1993 ) , and the benefits of depth in neural networks ( Telgarsky , 2016 ; Eldan and Shamir , 2016 ; Safran and Shamir , 2017 ; Daniely , 2017 ; Chatziafratis et al. , 2019 ) . Another central and well studied question is about their memorization power . The problem of memorization in neural networks can be viewed in the following way : For every dataset of N labeled samples ( x1 , y1 ) , . . . , ( xN , yN ) ∈ X × Y , construct a network N : X → Y such that N ( xi ) = yi for every i = 1 , . . . , N . Many works have shown results regarding the memorization power of neural networks , using different assumptions on the activation function and data samples ( see e.g . Huang and Babri ( 1998 ) ; Huang ( 2003 ) ; Baum ( 1988 ) ; Vershynin ( 2020 ) ; Daniely ( 2019 ; 2020 ) ; Bubeck et al . ( 2020 ) ; Park et al . ( 2020 ) ; Hardt and Ma ( 2016 ) ; Yun et al . ( 2019 ) ; Zhang et al . ( 2021 ) ; Nguyen and Hein ( 2018 ) ; Rajput et al . ( 2021 ) ; Sontag ( 1997 ) ) . The question of memorization also have practical implications on phenomenons such as ” double descent ” ( Belkin et al . ( 2019 ) ; Nakkiran et al . ( 2019 ) ) which connects the memorization power of neural networks with their generalization capabilities . A trivial lower bound on the required size of the network for memorizingN labeled points is implied by the VC dimension of the network ( cf . Shalev-Shwartz and Ben-David ( 2014 ) ) . That is , if a network with a certain size can not shatter any specific set of N points , then it certainly can not memorize all sets of N points . Known VC dimension bounds for networks with W parameters is on the order ofO ( W 2 ) ( Goldberg and Jerrum ( 1995 ) ; Bartlett et al . ( 1998 ; 2019 ) ) . Hence , it follows that memorizing N samples would require at least Ω ( N1/2 ) parameters . The best known upper bound is given in Park et al . ( 2020 ) , where it is shown that memorizing N data samples can be done using a neural network with O ( N2/3 ) parameters . Thus , there is a clear gap between the lower and upper bounds , although we note that the upper bound is for memorization of any set of data samples , while the lower bound is for shattering a single set of data samples . In this paper we ask the following questions : What is the minimal number of parameters that are required to memorize N labeled data samples ? Is the task of memorizing any set of N data samples more difficult than shattering a single set of N samples ? We answer these questions by providing a construction of a ReLU feedforward neural network which achieves the lower bound up to logarithmic factors . In this construction we use a very deep neural network , but with a constant width of 12 . In more details , our main result is the following : Theorem 1.1 ( informal statement ) . Let ( x1 , y1 ) , . . . , ( xN , yN ) ∈ Rd × { 1 , . . . , C } be a set of N labeled samples of a constant dimension d , with ‖xi‖ ≤ r for every i and ‖xi − xj‖ ≥ δ for every i 6= j . Then , there exists a ReLU neural network F : Rd → R with width 12 , depth Õ ( √ N ) , and Õ ( √ N ) parameters , such that F ( xi ) = yi for every i ∈ [ N ] , where the notation Õ ( · ) hides logarithmic factors in N , C , r , δ−1 . Comparing this result to the known VC bounds , we show that , up to logarithmic factors , our construction is optimal . This also shows , quite surprisingly , that up to logarithmic factors , the task of shattering a single set of N points is not more difficult than memorizing any set of N points , under the mild separability assumption of the data samples . We note that this result can also be extended to regression tasks ( see Remark 3.3 ) . In our construction , the depth of the network is Θ̃ ( √ N ) . We also give a generalized construction where the depth of the network is limited to some 1 ≤ L ≤ √ N . In his case , the number of param- eters in our construction is Õ ( N L ) ( see Theorem 5.1 ) . We compare this result to the VC-dimension bound from Bartlett et al . ( 2019 ) , and show that our construction is optimal up to logarithmic factors . Our construction uses a bit extraction technique , inspired by Telgarsky ’ s triangle function ( Telgarsky ( 2016 ) ) , and by Safran and Shamir ( 2017 ) . Using this technique , we are able to use weights with bit complexity Θ̃ ( √ N ) , and deep neural networks to “ extract ” the bits of information from the specially crafted weights of the network . We also generalize our results to the case of having a bounded bit complexity restriction on the weights . We show both lower ( Theorem 6.1 ) and upper ( Theorem 6.2 ) bounds , proving that memorizing N points using a network with N1− parameters , for some ∈ [ 0 , 0.5 ] can be done if the bit complexity of each weight is Θ̃ ( N ) . Hence , our construction is also optimal , up to logarithmic factors , w.r.t the bit complexity of the network . We emphasize that also in previous works showing non-trivial VC bounds ( e.g . Bartlett et al . ( 1998 ; 2019 ) ) weights with large bit complexity are used . We note that increasing the bit complexity beyond N1/2 can not be used to further reduce the number of parameters ( see the discussion in section 4 ) . RELATED WORK . MEMORIZATION – UPPER BOUNDS . The problem of memorizing arbitrary data points with neural networks has a rich history . Baum ( 1988 ) studied memorization in single-hidden-layer neural networks with the threshold activation , and showed that dNd e neurons suffice to memorize N arbitrary points in general position in R d with binary labels . Bubeck et al . ( 2020 ) extended the construction of Baum ( 1988 ) and showed that single-hidden-layer ReLU networks with 4 · dNd e hidden neurons can memorize N points in general position with arbitrary real labels . In Huang et al . ( 1991 ) and Sartori and Antsaklis ( 1991 ) it is shown that single-hidden-layer networks with the threshold activation can memorize any arbitrary set of N points , even if they are not in general position , using N − 1 neurons . Huang and Babri ( 1998 ) proved a similar result for any bounded non-linear activation function σ where either limz→∞ σ ( z ) or limz→−∞ σ ( z ) exists . Zhang et al . ( 2021 ) proved that single-hidden-layer ReLU networks can memorize arbitraryN points in Rd with arbitrary real labels usingN neurons and 2N+d parameters . Huang ( 2003 ) showed that two-hidden-layers networks with the sigmoid activation can memorize N points with O ( √ N ) neurons , but the number of parameters is still linear in N . Yun et al . ( 2019 ) proved a similar result for ReLU ( and hard-tanh ) networks . Vershynin ( 2020 ) showed that threshold and ReLU networks can memorize N binary-labeled unit vectors in Rd separated by a distance of δ > 0 , using Õ ( e1/δ 2 + √ N ) neurons and Õ ( e1/δ 2 ( d+ √ N ) +N ) parameters . Rajput et al . ( 2021 ) improved the dependence on δ by giving a construction with Õ ( 1 δ + √ N ) neurons and Õ ( d δ +N ) parameters . This result holds only for threshold networks , but does not assume that the inputs are on the unit sphere . The memorization power of more specific architectures was also studied . Hardt and Ma ( 2016 ) proved that residual ReLU networks with O ( N ) neurons can memorize N points on the unit sphere separated by a constant distance . Nguyen and Hein ( 2018 ) considered convolutional neural networks and showed , under certain assumptions , memorization using O ( N ) neurons . Note that in all the results mentioned above the number of parameters is at least linear in N . Our work is inspired by Park et al . ( 2020 ) , that established a first memorization result with a sub-linear number of parameters . They showed that neural networks with sigmoidal or ReLU activations can memorize N points in Rd separated by a normalized distance of δ , using O ( N2/3 + log ( 1/δ ) ) parameters ( where the dimension d is constant ) . Thus , in this work we improve the dependence on N from N2/3 to √ N ( up to logarithmic factors ) , which is optimal . We also note that the first stage in our construction is similar to the first stage in theirs . Finally , optimization aspects of memorization were studied in Bubeck et al . ( 2020 ) ; Daniely ( 2019 ; 2020 ) . MEMORIZATION – LOWER BOUNDS . An Ω ( N ) lower bound on the number of parameters required for memorizing arbitrary N points using neural networks with standard activations ( e.g. , threshold , sigmoid and ReLU ) is given in Sontag ( 1997 ) . Thus , for networks with o ( N ) parameters , there is a set of size N that can not be shattered . It implies that in order to obtain memorization with a sub-linear number of parameters some assumptions are required . Our positive result circumvents this lower bound by assuming that the data is separated . Moreover , lower bounds on the number of parameters required for memorization are implied by bounds on the VC dimension of neural networks . Indeed , if W parameters are not sufficient for shattering even a single set of size N , then they are clearly not sufficient for memorizing all sets of size N . The VC dimension of neural networks has been extensively studied in recent decades ( cf . Anthony and Bartlett ( 2009 ) ; Bartlett et al . ( 2019 ) ) . The most relevant results for our work are by Goldberg and Jerrum ( 1995 ) and Bartlett et al . ( 2019 ) . We discuss these results and their implications in Sections 4 and 5 . Trade-offs between the number of parameters of the network and the Lipschitz parameter of the prediction function in memorizing a given dataset are studies in Bubeck et al . ( 2021 ) ; Bubeck and Sellke ( 2021 ) .

Summary: This paper studies memorization capacity of deep ReLU networks. For arbitrary $N$ data points in a ball of size $r$ satisfying minimum separation $\delta$, the authors show that there exists a ReLU network of constant width and depth $\tilde O(\sqrt{N})$ that perfectly memorizes the entire dataset (Theorem 3.1). This means that memorizing arbitrary $N$ data points can be done using only $\tilde O(\sqrt{N})$ parameters, when depth increases with $N$. Combined with a classical upper bound on VC dimension (Goldberg and Jerrum (1995)), the construction is optimal up to log factors. Theorem 3.1 is extended to the case of fixed depth $L \leq \sqrt{N}$ (Theorem 5.1) and fixed bit complexity per parameter (Theorem 6.2), and these additional results are also optimal modulo log factors.