Title: Generalizablity of Memorization Neural Network
Venue: NeurIPS 2024
Accepted: yes
Keywords: generalization

memorization

sample complexity
Abstract: The neural network memorization problem is to study the expressive power of neural networks to interpolate a finite dataset. Although memorization is widely believed to have a close relationship with the strong generalizability of deep learning when using overparameterized models, to the best of our knowledge, there exists no theoretical study on the generalizability of memorization neural networks. In this paper, we give the first theoretical analysis of this topic. Since using i.i.d. training data is a necessary condition for a learning algorithm to be generalizable, memorization and its generalization theory for i.i.d. datasets are developed under mild conditions on the data distribution. First, algorithms are given to construct memorization networks for an i.i.d. dataset, which have the smallest number of parameters and even a constant number of parameters. Second, we show that, in order for the memorization networks to be generalizable, the width of the network must be at least equal to the dimension of the data, which implies that the existing memorization networks with an optimal number of parameters are not generalizable. Third, a lower bound for the sample complexity of general memorization algorithms and the exact sample complexity for memorization algorithms with constant number of parameters are given. As a consequence, it is shown that there exist data distributions such that, to be generalizable for them, the memorization network must have an exponential number of parameters in the data dimension. Finally, an efficient and generalizable memorization algorithm is given when the number of training samples is greater than the efficient memorization sample complexity of the data distribution.

Generalizability of Memorization Neural Networks

Abstract

The neural network memorization problem is to study the expressive power of neural networks to interpolate a finite dataset. Although memorization is widely believed to have a close relationship with the strong generalizability of deep learning when using over-parameterized models, to the best of our knowledge, there exists no theoretical study on the generalizability of memorization neural networks. In this paper, we give the first theoretical analysis of this topic. Since using i.i.d. training data is a necessary condition for a learning algorithm to be generalizable, memorization and its generalization theory for i.i.d. datasets are developed under mild conditions on the data distribution. First, algorithms are given to construct memorization networks for an i.i.d. dataset, which have the smallest number of parameters and even a constant number of parameters. Second, we show that, in order for the memorization networks to be generalizable, the width of the network must be at least equal to the dimension of the data, which implies that the existing memorization networks with an optimal number of parameters are not generalizable. Third, a lower bound for the sample complexity of general memorization algorithms and the exact sample complexity for memorization algorithms with constant number of parameters are given. It is also shown that there exist data distributions such that, to be generalizable for them, the memorization network must have an exponential number of parameters in the data dimension. Finally, an efficient and generalizable memorization algorithm is given when the number of training samples is greater than the efficient memorization sample complexity of the data distribution.

1 Introduction

Memorization is to study the expressive power of neural networks to interpolate a finite dataset [9]. The main focus of the existing work is to study how many parameters are needed to memorize. For any dataset D tr of size N and neural networks of the form F : R n → R , memorization networks with O ( N ) parameters have been given with various model structures and activation functions [31, 50, 30, 29, 26, 47, 56, 11, 65]. On the other hand, it is shown that in order to memorize an arbitrary dataset of size N [64, 56], the network must have at least Ω( N ) parameters, so the above algorithms are approximately optimal. Under certain assumptions, it is shown that sublinear O ( N 2 / 3 ) parameters are sufficient to memorize D tr [49]. Furthermore, Vardi et al. [55] give a memorization network with optimal number of parameters: O ( √ N ) .

Recently, it is shown that memorization is closely related to one of the most surprising properties of deep learning, that is, over-parameterized neural networks are trained to nearly memorize noisy data and yet can still achieve a very nice generalization on the test data [45, 7, 4]. More precisely, the

double descent phenomenon [45] indicates that when the networks reach the interpolation threshold, larger networks tend to have more generalizability [41, 10]. It is also noted that memorizing helps generalization in complex learning tasks, because data with the same label have quite diversified features and need to be nearly memorized [19, 20]. A line of research to harvest the help of memorization to generalization is interpolation learning . Most of recent work in interpolation learning shows generalizability of memorization models in linear regimes [7, 12, 38, 53, 59, 66].

As far as we know, the generazability of memorization neural networks has not been studied theoretically, which is more challenging compared to the linear models, and this paper provides a systematic study of this topic. In this paper, we consider datasets that are sampled i.i.d. from a data distribution, because i.i.d. training dataset is a necessary condition for learning algorithms to have generalizability [54, 44]. More precisely, we consider binary data distributions D over R n ×{-1 , 1 } and use D tr ∼ D N to mean that D tr is sampled i.i.d. from D and |D tr | = N . All neural networks are of the form F : R n → R . The main contributions of this paper include four aspects.

First, we give the smallest number of parameters required for a network to memorize an i.i.d. dataset.

Theorem 1.1 (Informal. Refer to Section 4) . Under mild conditions on D , if D tr ∼ D N , it holds

(1) There exists an algorithm to obtain a memorization network of D tr with width 6 and depth O ( √ N ) .

(2) There exists a constant N D ∈ Z + depending on D only, such that a memorization network of D tr with at most N D parameters can be obtained algorithmically.

N D is named as the memorization parameter complexity of D , which measures the complexity of D under which a memorization network with ≤ N D parameters exists for almost all D tr ∼ D N .

Theorem 1.1 allows us to give the memorization network for i.i.d dataset with the optimal number of parameters. When N is small so that √ N ≪ N D , the memorization network needs at least Ω( √ N ) parameters as proved in [6] and (1) of Theorem 1.1 gives the optimal construction. When N is large, (2) of Theorem 1.1 shows that a constant number of parameters is enough to memorize.

Second, we give a necessary condition for the structure of the memorization networks to be generalizable, and shows that even if there is enough data, memorization network may not have generalizability.

Theorem 1.2 (Informal. Refer to Section 5) . Under mild conditions on D , if D tr ∼ D N , it holds

(1) Let H be a set of neural networks with width w . Then, there exist an integer n > w and a data distribution D over R n × {-1 , 1 } such that, any memorization network of D tr in H is not generalizable.

√

(2) For almost any D , there exists a memorization network of D tr , which has O ( N ) parameters and is not generalizable.

√

Theorem 1.2 indicates that memorization networks with the optimal number of parameters O ( N ) may have poor generalizability, and commonly used algorithms for constructing fixed-width memorization networks have poor generalization for some distributions. These conclusions demonstrate that the commonly used network structures for memorization is not generalizable and new network structures are needed to achieve generalization.

Third, we give a lower bound for the sample complexity of general memorization networks and the exact sample complexity for certain memorization networks.

Theorem 1.3 (Informal. Refer to Section 6) . Let N D be the memorization parameter complexity defined in Theorem 1.1. Under mild conditions on D , we have

(1) Lower bound . In order for a memorization network of any D tr ∼ D N to be generalizable, N must be ≥ Ω( N 2 D ln 2 ( N D ) ) 2 .

(2) Upper bound . For any memorization network with at most N D parameters for D tr ∼ D N , if N = O ( N 2 D ln N D ) , then the network is generalizable.

Notice that the lower bound is for general memorization networks and the upper bound is for memorization networks with ≤ N D parameters, which always exist by (2) of Theorem 1.1. In the latter case, the lower and upper bounds are approximately the same, which gives the exact sample complexity O ( N 2 D ) in this case. In other words, a necessary and sufficient condition for the memorization network in (2) of Theorem 1.1 to be generalizable is N = O ( N 2 D ) .

Remark 1.4 . Unfortunately, these generalizable memorization networks cannot be computed efficiently, as shown by the following results proved by us.

̸

(1) If P = NP , then all networks in (2) of Theorem 1.3 cannot be computed in polynomial time.

(2) For some data distributions, an exponential (in the data dimension) number of samples is required for memorization networks to achieve generalization.

Finally, we want to know that does there exist a polynomial time memorization algorithm that can ensure generalization, and what is the sample complexity of such memorization algorithm? An answer is given in the following theorem.

Theorem 1.5 (Informal. Refer to Section 7) . There exists an S D ∈ Z + depending on D only such that, under mild conditions on D , if N = O ( S D ) , then we can construct a generalizable memorization network with O ( N 2 n ) parameters for any D tr ∼ D N in polynomial time.

S D is named as the efficient memorization sample complexity for D , which measures the complexity of D so that the generalizable memorization network of any D tr ∼ D N can be computed efficiently if N = O ( S D ) .

√

The memorization network in Theorem 1.5 has more parameters than the optimal number O ( N ) of parameters required for memorization. The main reason is that building memorization networks with O ( √ N ) parameters requires special technical skill that may break the generalization. On the other hand, as mention in [7], over-parametrization is good for generalization, so it is reasonable for us to use more parameters for memorization to achieve generalization.

Remark 1.6 . We explain the relationship between our results and interpolation learning [7]. Interpolation learning uses optimization to achieve memorization, which is a more practical approach, while our approach gives a theoretical foundation for memorization networks. Once an interpolation is achieved, Theorem 1.2, (1) of Theorem 1.3, and Theorem 1.5 are valid for interpolation learning. For example, according to (1) of Theorem 1.3, Ω( N 2 D ) is a lower bound for the sample complexity of interpolation learning, and by Theorem 1.5, O ( S D ) is an upper bound for the sample complexity of efficient interpolation learning.

Main Contributions . Under mild conditions for the data distribution D , we have

We define the memorization parameter complexity N D ∈ Z + of D such that, a memorization network for any D tr ∼ D N can be constructed, which has O ( √ N ) or ≤ N D parameters. Here, the memorization network has the optimal number of parameters.

We give two necessary conditions for the construction of generalizable memorization networks for any D tr in terms of the width and number of parameters of the memorization network.

We give a lower bound Ω( N 2 D ) of the sample complexity for general memorization networks as well as the exact sample complexity O ( N 2 D ) for memorization networks with ≤ N D parameters. We also show that for some data distribution, an exponential number of samples in n is required to achieve generalization.

We define the efficient memorization sample complexity S D ∈ Z + for D , so that generalizable memorization network of any D tr ∼ D N can be computed in polynomial time, if N = O ( S D ) .

2 Related work

Memorization . The problem of memorization has a long history. In [9], it is shown that networks with depth 2 and O ( N ) parameters can memorize a binary dataset of size N . In subsequent work,

it is shown that networks with O ( N ) parameters can be a memorization for any dataset [31, 50, 11, 30, 65, 29, 64, 56, 26, 47] and such memorization networks are approximately optimal for generic dataset [64, 56]. Since the VC dimension of neural networks with N parameters and depth D and with ReLU as the activation function is at most O ( ND ) [24, 5, 6], memorizing some special datasets of size N requires at least Ω( √ N ) parameters and there exists a gap between this lower bound Ω( √ N ) and the upper bound O ( N ) . Park et al. [49] show that a network with O ( N 2 / 3 ) parameters is enough for memorization under certain assumptions. Vardi et al. [55] further give the memorization network with optimal number of parameters O ( √ N ) . In [22], strengths of both generalization and memorization are combined in a single neural network. Recently, robust memorization has been studied [35, 62]. As far as we know, the generazability of memorization neural networks has not been studied theoretically.

Interpolation Learning . Another line of related research is interpolation learning, that is, leaning under the constraint of memorization, which can be traced back to [52]. Most recent works establish various generalizability of interpolation learning in linear regimes [7, 12, 38, 53, 59, 66]. For instance, Bartlett et al. [7] prove that over-parametrization allows gradient methods to find generalizable interpolating solutions for the linear regime. In relation to this, how to achieve memorization via gradient descent is studied in [13, 14]. Results of this paper can be considered to give sample complexities for interpolation learning.

Generalization Guarantee . There exist several ways to ensure generalization of networks. The common way is to estimate the generalization bound or sample complexity of leaning algorithms. Generalization bounds for neural networks are given in terms of the VC dimension [24, 5, 6], under the normal training setting [27, 44, 8], under the differential privacy training setting [1], and under the adversarial training setting [60, 58]. In most cases, these generalization bounds imply that when the training set is large enough, a well-trained network with fixed structure has good generalizability. On the other hand, the relationship between memorization and generalization has also been extensively studied [45, 41, 10, 19, 20]. In [25], sample complexity of neural networks is given when the norm of the transition matrix is limited, in [36], sample complexity of shallow transformers is considered. This paper gives the lower bound and upper bound (in certain cases) of the sample complexities for interpolation learning.

3 Notation

In this paper, we use O ( A ) to mean a value not greater than cA for some constant c , and O to mean that small quantities, such as logarithm, are omitted. We use Ω( A ) to mean a value not less than cA for some constant c , and Ω to mean that small quantities, such as logarithm, are omitted. We say for all ( x, y ) ∼ D there is event A stand means that P ( x,y ) ∼D ( A ) = 1 .

3.1 Neural network

In this paper, we consider feedforward neural networks of the form F : R n → R and the l -th hidden layer of F ( x ) can be written as

where σ = Relu is the activation function, X 0 = x and N 0 = n . The last layer of F is F ( x ) = W L +1 X L + b L +1 ∈ R , where L is the number of hidden layers in F . The depth of F is depth ( F ) = L +1 , the width of F is width ( F ) = max L i =1 { n i } , the number of parameters of F is para ( F ) = ∑ L i =0 n i ( n i +1 +1) . Denote H ( n ) to be the set of all neural networks in the above form.

3.2 Data distribution

In this paper, we consider binary classification problems and use D to denote a joint distribution on D ( n ) = [0 , 1] n ×{-1 , 1 } . To avoid extreme cases, we focus mainly on a special kind of distribution to be defined in the following.

Definition 3.1. For n ∈ Z + and c ∈ R + , D ( n, c ) is the set of distributions D on D ( n ) , which has a positive separation bound: inf ( x, 1) , ( z, -1) ∼D || x -z || 2 ≥ c .

The accuracy of a network F on a distribution D is defined as

We use D tr ∼ D N to mean that D tr is a set of N data sampled i.i.d. according to D . For convenience, dataset under distribution means that the dataset is i.i.d selected from a data distribution.

̸

Remark 3.2 . We define the distribution with positive separation bound in for the following reasons. (1) If D tr ∼ D N and D ∈ D ( n, c ) , then x i = x j when y i = y j . Such property ensures that D tr can be memorized. (2) Proposition 3.3 shows that there exists a D such that any network is not generalizable over D , and this should be avoided. Therefore, distribution D needs to meet certain requirements for a dataset sampled from D to have generalizability. Proof of Proposition 3.3 is given in Appendix A. (3) Most commonly used classification distributions should have positive separation bound.

̸

Proposition 3.3. There exists a distribution D such that A D ( F ) ≤ 0 . 5 for any neural network F .

3.3 Memorization neural network

Definition 3.4. A neural network F ∈ H ( n ) is a memorization of a dataset D tr over D ( n ) , if Sgn ( F ( x )) = y for any ( x, y ) ∈ D tr .

Remark 3.5 . Memorization networks can also be defined more strictly as F ( x ) = y for any ( x, y ) ∈ D tr . In Proposition 4.10 of [62], it is shown that these two types of memorization networks need essentially the same number of parameters.

To be more precise, we treat memorization as a learning algorithm in this paper, as defined below.

Definition 3.6. L : ∪ n ∈ Z + 2 D ( n ) → ∪ n ∈ Z + H ( n ) is called a memorization algorithm if for any n and D tr ∈ D ( n ) , L ( D tr ) is a memorization network of D tr .

Furthermore, a memorization algorithm L is called an efficient memorization algorithm if there exists a polynomial poly : R → R such that L ( D tr ) can be computed in time poly ( size ( D tr )) , where size ( D tr ) is the bit-size of D tr .

Remark 3.7 . It is clear that if L is an efficient memorization algorithm, then para ( L ( D tr )) is also polynomial in size ( D tr ) .

There exist many methods which can construct memorization networks in polynomial times, and all these memorization methods are efficient memorization algorithms, which are summarized in the following proposition.

Proposition 3.8. The methods given in [9, 62] are efficient memorization algorithms. The methods given in [55, 49] are probabilistic efficient memorization algorithms, which can be proved similar to that of Theorem 4.1. More precisely, they are Monte Carlo polynomial-time algorithms.

4 Optimal memorization network for dataset under distribution

By the term 'dataset under distribution', we mean datasets that are sampled i.i.d. from a data distribution, and is denoted as D tr ∼ D N . In this section, we show how to construct the memorization network with the optimal number of parameters for dataset under distribution.

4.1 Memorization network with optimal number of parameters

To memorize N samples, ˜ Ω( √ N ) parameters are necessary [6]. In [55], a memorization network is given which has O ( √ N ) parameters under certain conditions, where O means that some logarithm factors in N and polynomial factors of other values are omitted. Therefore, O ( √ N ) is the optimal number of parameters for a network to memorize certain dataset. In the following theorem, we show that such a result can be extended to dataset under distribution.

Theorem 4.1. Let D ∈ D ( n, c ) and D tr ∼ D N . Then there exists a memorization algorithm L such that L ( D tr ) has width 6 and depth (equivalently, the number of parameters) O ( √ N ln( Nn/c )) . Furthermore, for any ϵ ∈ (0 , 1) , L ( D tr ) can be computed in time poly ( size ( D tr ) , ln(1 /ϵ )) with probability ≥ 1 -ϵ .

̸

Proof Idea. This theorem can be proven using the idea from [55]. Let D tr = { ( x i , y i ) } N i =1 . The mainly different is that in [55], it requires || x i -x j || ≥ c for all i = j , which is no longer valid when D tr is sampled i.i.d. from distribution D . Since D has separation bound c > 0 , we have || x i -x j || ≥ c for all i, j satisfying y i = y j , which is weaker. Despite this difference, the idea of [55] can still be modified to prove the theorem. In constructing such a memorization network, we need to randomly select a vector, and each selection has a probability of 0.5 to give the correct vector. So, repeat the selection ln(1 /ϵ ) times, with probability 1 -ϵ , we can get at least one correct vector. Then we can construct the memorization network based on this vector. Detailed proof is given in Appendix B.

Remark 4.2 . The algorithm in Theorem 4.1 is a Monte Carlo polynomial-time algorithm, that is, it gives a correct answer with arbitrarily high probability. The algorithm given in [55] is also a Monte Carlo algorithm.

4.2 Memorization network with constant number of parameters

In this section, we prove an interesting fact of memorization for dataset under distribution. We show that for a distribution D ∈ D ( n, c ) , there exists a constant N D ∈ Z + such that for all datasets sampled i.i.d. from D , there exists a memorization network with N D parameters.

Theorem 4.3. There exists a memorization algorithm L such that for any D ∈ D ( n, c ) , there is an N ′ D ∈ Z + satisfying that for any N > 0 , with probability 1 of D tr ∼ D N , we have para ( L ( D tr )) ≤ N ′ D . The smallest N ′ D of the distribution D is called the memorization parameter complexity of D , written as N D .

Proof Idea. It suffices to show that we can find a memorization network of D tr with a constant number of parameters, which depends on D only. The main idea is to take a subset D ′ tr of D tr such that D tr is contained in the neighborhood of D ′ tr . It can be proven that the number of elements in this subset is limited. Then construct a robust memorization network of D ′ tr with certain budget [62], we obtain a memorization network of D tr , which has a constant number of parameters. The proof is given in Appendix C.

Combining Theorems 4.1 and 4.3, we can give a memorization network with the optimal number of parameters.

Remark 4.4 . What we have proven in Theorem 4.3 is that a memorization algorithm with a constant number of parameters can be found, but in most of times, we have N ′ D > N D . Furthermore, if N ′ D is large for the memorization algorithm, the algorithm can be efficient. Otherwise, if N ′ D is closed to N D , the algorithm is usually not efficient.

Remark 4.5 . It is obvious that the memorization parameter compelxity N D is the minimum number of parameters required to memorize any dataset sampled i.i.d. from D . N D is mainly determined by the characteristic of D ∈ D ( n, c ) , so N D may be related to n and c . It is an interesting problem to estimate N D .

5 Condition on the network structure for generalizable memorization

In the preceding section, we show that for the dataset under distribution, there exists a memorization algorithm to generate memorization networks with the optimal number of parameters. In this section, we give some conditions for the generalizable memorization networks in terms of width and number of parameters of the network. As a consequence, we show that the commonly used memorization networks with fixed width is not generalizable.

First, we show that networks with fixed width do not have generazability in some situations. Reducing the width and increasing depth is a common way for parameter reduction, but it inevitably limits the network's power, making it unable to achieve good generalization for specific distributions, as shown in the following theorem.

Theorem 5.1. Let w ∈ Z + and L be a memorization algorithm such that L ( D tr ) has width not more than w for all D tr . Then, there exist an integer n > w , c ∈ R + , and a distribution D ∈ D ( n, c ) such that, for any D tr ∼ D N , it holds A D ( L ( D tr )) ≤ 0 . 51 .

̸

Proof Idea. As shown in [40, 48], networks with small width are not dense in the space of measurable functions, but this is not enough to estimate the upper bound of the generalization. In order to further measure the upper bound of generalization, we define a special class of distributions. Then, we calculate the upper bound of the generalization of networks with fixed width on this class of distribution. Based on the calculation results, it is possible to find a specific distribution within this class of distributions, such that the fixed-width network exhibits a poor generalization of this distribution. The proof is given in Appendix D.

It is well known that width of the network is important for the network to be robust [2, 17, 18, 37, 67]. Theorem 5.1 further shows that large width is a necessary condition for generalizabity.

Note that Theorem 5.1 is for a specific data distribution. We will show that for most distributions, providing enough data does not necessarily mean that the memorization algorithm has generalization ability. This highlights the importance of constructing appropriate memorization algorithms to ensure generalization. We need to introduce another parameter for data distribution.

Definition 5.2. The distribution D is said to have density r , if P x ∼D ( x ∈ A ) /V ( A ) ≤ r for any closed set A ⊂ [0 , 1] n , where V ( A ) is the volume of A .

Loosely speaking, the density of a distribution is the upper bound of the density function.

Theorem 5.3. For any n ∈ Z + , r, c ∈ R + , if distribution D ∈ D ( n, c ) has density r , then for any N ∈ Z + and D tr ∼ D N , there exists a memorization network F for D tr such that para ( F ) = O ( n + √ N ln( Nnr/c )) and A D ( F ) ≤ 0 . 51 .

Proof Idea. We refer to the classical memorization construction idea [55]. The main body includes three parts. Firstly, compress the data in D tr into one dimension. Secondly, map the compressed data to some specific values. Finally, use such a value to get the label of input. Moreover, we will pay more attention to points outside the dataset. We use some skills to control the classification results of points that do not appear in the dataset D tr , so that the memorization network will give the wrong label to the points that are not in D tr as much as possible to reduce its accuracy. The general approach is the following: (1) Find a set in which each point is not presented in D tr and has the same label under distribution D . Without loss of generality, let they have label 1 . (2) In the second step mentioned in the previous step, ensure that the mapped results of the points in the set mentioned in (1) are similar to the samples with label -1 . This will cause the third step to output the label -1 , leading to an erroneous classification result for the points in the set. The proof is given in Appendix E.

Remark 5.4 . Theorem 5.1 shows that the width of the generazable memorization network needs to increase with the increase of the data dimension. Theorem 5.3 shows that when para ( F ) = O ( √ N ) , the memorization network may have poor generalizability for most distributions. The above two theorems indicate that no matter how large the dataset is, there always exist memorization networks with poor generalization. In terms of sample complexity, it means that for the hypotheses of neural networks with fixed width or with optimal number of parameters, the sample complexity is infinite, contrary to the uniform generalization bound for feedforward neural networks [63, Lemma D.16].

Remark 5.5 . It is worth mentioning that the two theorems in this section cannot be obtained from the lower bound of the generalization gap [44], and more details are shown in Appendix E.

6 Sample complexity for memorization algorithm

As said in the preceding section, generalization of memorization inevitably requires certain conditions. In this section, we give the necessary and sufficient condition for generalization for the memorization algorithm in Section 4 in terms of sample complexity.

We first give a lower bound for the sample complexity for general memorization algorithms and then an upper bound for memorization algorithms which output networks with an optimal number of parameters. The lower and upper bounds are approximately the same, thus giving the exact sample complexity in this case.

6.1 Lower bound for sample complexity of memorization algorithm

Roughly speaking, the sample complexity of a learning algorithm is the number of samples required to achieve generalizability [44]. The following theorem gives a lower bound for the sample complexity of memorization algorithms based on N D , which has been defined in Theorem 4.3.

Theorem 6.1. There exists no memorization algorithm L which satisfies that for any n ∈ Z + , c ∈ R + , ϵ, δ ∈ (0 , 1) , if D ∈ D ( n, c ) and N ≥ v N 2 D 2 (1 -2 ϵ -δ ) , it holds

where v is an absolute constant which does not depend on N,n,c,ϵ, δ .

Proof Idea. The mainly idea is that: for a dataset D tr ⊂ [0 , 1] n ×{-1 , 1 } with |D tr | = N , we can find some distributions D 1 , D 2 , . . . , such that if D tr,i ∼ ( D i ) N , then with a positive probability, it hold D tr,i = D tr . In addition, each distribution has a certain degree of difference from the others. It is easy to see that L ( D tr ) is a fixed network for a given L , so L ( D tr ) cannot fit all D i well because D i are different to some degree. So, if a memorization algorithm L satisfies the condition in the theorem, we try to construct some distributions {D i } n i =1 , and use the above idea to prove that L cannot fit one of the distributions in {D i } n i =1 , and obtain contradictions. The proof of the theorem is given in Appendix F.

Remark 6.2 . In general, the sample complexity depends on the data distribution, hypothesis space, learning algorithms, and ϵ, δ . Since N D is related to n and c , the lower bound in Theorem 6.1 also depends on n and c . Here, the hypothesis space is the memorization networks, which is implicitly reflected in N D .

Remark 6.3 . Roughly strictly, if we consider interpolation learning, that is, training network under the constraint of memorizing the dataset, then Theorem 6.1 also provides a lower bound for the sample complexity.

This theorem shows that if we want memorization algorithms to have guaranteed generalization, then about O ( N 2 D ) samples are required. As a consequence, we show that, for some data distribution, it need an exponential number of samples to achieve generalization. The proof is also in Appendix F.

Corollary 6.4. For any memorization algorithm L and any ϵ, δ ∈ (0 , 1) , there exist n ∈ Z + , c > 0 and a distribution D ∈ D ( n, c ) , such that in order for L to have generalizability on D , that is for all N ≥ N 0 , there is

N 0 must be more than v (2 2[ n ⌈ c 2 ⌉ ] c 4 (1 -2 ϵ -δ ) /n 2 ) , where v is an absolute constant not depending on N,n,c,ϵ, δ .

6.2 Exact sample complexity of memorization algorithm with N D parameters

In Theorem 6.1, it is shown that Ω( N 2 D ) samples are necessary for generalizability of memorization. The following theorem shows that there exists a memorization algorithm that can reach generalization with O ( N 2 D ) samples.

Theorem 6.5. For all memorization algorithms L satisfies that L ( D tr ) has at most N D parameters, with probability 1 for D tr ∼ D N , we have

where v is an absolute constant which does not depend on N,n,c,ϵ, δ .

̸

(2) If P = NP, then all such algorithms are not efficient.

Proof Idea. For the proof of (1), we need to use the N D to calculate the VC-dimension [6], and take such a dimension in the generalization bound theorem [44] to obtain the result. For the proof of (2), we show that, if such algorithm is efficient, then we can solve the following reversible 6-SAT [43] problem, which is defined below and is an NPC problem. The proof of the theorem is given in Appendix G.

Definition 6.6. Let φ be a Boolean formula and φ the formula obtained from φ by negating each variable. The Boolean formula φ is called reversible if either both φ and φ are satisfiable or both are not satisfiable. The reversible satisfiability problem is to recognize the satisfiability of reversible formulae in conjunctive normal form (CNF). By the reversible 6-SAT , we mean the reversible satisfiability problem for CNF formulae with six variables per clause. In [43], it is shown that the reversible 6-SAT is NPC.

Combining Theorems 6.1 and 6.5, we see that N = O ( N 2 D ) is the necessary and sufficient condition for the memorization algorithm to generalize, and hence O ( N 2 D ) is the exact sample complexity for memorization algorithms with N D parameters over the distribution D ( n, c ) .

Unfortunately, by (2) of Theorem 6.5, this memorization algorithm is not efficient when the memorization has no more than N D parameters. Furthermore, we conjecture that there exist no efficient memorization algorithms that can use O ( N 2 D ) samples to reach generalization in the general case, as shown in the following conjecture.

̸

Conjecture 6.7 . If P = NP, there exist no efficient memorization algorithms that can reach generalization with O ( N 2 D ) samples for all D ∈ D ( n, c ) .

Remark 6.8 . This result also provides certain theoretical explanation for the over-parameterization mystery [45, 7, 4]: for memorization algorithms with N D parameters, the exact sample complexity O ( N 2 D ) is greater than the number of parameters. Thus, the networks is under-parameterized and for such a network, even if it is generalizable, it cannot be computed efficiently.

7 Efficient memorization algorithm with guaranteed generalization

In the preceding section, we show that there exist memorization algorithms that are generalizable when N = O ( N 2 D ) , but such an algorithm is not efficient. In this section, we give an efficient memorization algorithm with guaranteed generalization.

First, we define the efficient memorization sample complexity of D .

Definition 7.1. For ( x, y ) ∼ D , let L ( x,y ) = min ( z, -y ) ∼D || x -z || 2 and B (( x, y )) = B 2 ( x, L ( x,y ) / 3 . 1) = { z ∈ R n : ∥ z -x ∥ 2 ≤ L ( x,y ) / 3 . 1 } . The nearby set S of D is a subset of sample ( x, y ) which is in distribution D and satisfies: (1) for any ( x, y ) ∼ D , x ∈ ∪ ( z,w ) ∈ S B (( z, w )) ; (2) | S | is minimum.

Evidently, for any D ∈ D ( n, c ) , its nearby set is finite, as shown by Proposition 7.7. S D = | S | is called the efficient memorization sample complexity of D , the meaning of which is given in Theorem 7.3.

Remark 7.2 . In the above definition, we use L ( x,y ) / 3 . 1 to be the radius of B (( x, y )) . In fact, when 3 . 1 is replaced by any real number greater than 3 , the following theorem is still valid.

Theorem 7.3. There exists an efficient memorization algorithm L such that for any c ∈ R , ϵ, δ ∈ (0 , 1) , n ∈ Z + , and D ∈ D ( n, c ) , if N ≥ S D ln( S D /δ ) ϵ , then

Moreover, for any D tr ∼ D N , L ( D tr ) has at most O ( N 2 n ) parameters.

Proof Idea. For a given dataset D tr ⊂ [0 , 1] n ×{-1 , 1 } , we use the following two steps to construct a memorization network.

Step 1. Find suitable convex sets { C i } in [0 , 1] n such that each sample in D tr is in at least one of these convex sets. Furthermore, if x, z ∈ C i and ( x, y x ) , ( z, y z ) ∈ D tr , then y x = y z , and define y ( C i ) = y x .

Step 2. Construct a network F such that for any x ∈ C i , Sgn ( F ( x )) = y ( C i ) . This network must be a memorization of D tr , because each sample in D tr is in at least one of { C i } . Hence, if x ∈ C i and ( x, y x ) ∈ D tr , then Sgn ( F ( x )) = y ( C i ) = y x . The proof of the theorem is given in Appendix H.

Remark 7.4 . Theorem 7.3 shows that there exists an efficient and generalizable memorization algorithm when N = O ( S D ) . Thus, S D is an intrinsic complexity measure of D on whether it is

easy to learn and generalize. By Theorem 6.1, S D ≥ N 2 D for some D , but for some 'nice' D , S D could be small. It is an interesting problem to estimate S D .

Remark 7.5 . Theorem 7.3 uses O ( N 2 n ) parameters, highlight the importance of overparameterization [45, 7, 4]. Interestingly, Remark 6.8 shows that if the network has O ( √ N ) parameters, even if it is generalizable, it cannot be computed efficiently.

The experimental results of the memorization algorithm mentioned in Theorem 7.3 are given in Appendix I. Unfortunately, for commonly used datasets such as CIFAR-10, this algorithm cannot surpass the network obtained by training with SGD, in terms of test accuracy. Thus, the main purpose of the algorithm is theoretical, that is, it provides a polynomial-time memorization algorithm that can achieve generalization when the training dataset contains O ( S D ) samples. In comparison of theoretical works, training networks is NP-hard for small networks [32, 51, 39, 15, 3, 42, 16, 23, 21] and the guarantee of generalization needs strong assumptions on the loss function [46, 27, 34, 61, 60, 58].

Finally, we give an estimate for S D . From Corollary 6.4 and Theorem 7.3, we obtain a lower bound for S D .

Corollary 7.6. There exists a distribution D ∈ D ( n, c ) such that S D ln( S D /δ ) ≥ Ω( c 4 n 2 2 2[ n ⌈ c 2 ⌉ ] ) .

We will give an upper bound for S D in the following proposition, and the proof is given in Appendix H.1. From the proposition, it is clear that S D is finite.

Proposition 7.7. For any D ∈ D ( n, c ) , we have S D ≤ ([6 . 2 n/c ] + 1) n .

Remark 7.8 . The above proposition gives an upper bound of S D when D ∈ D ( n, c ) , and this does not mean that S D is exponential for all D ∈ D ( n, c ) . Determining the conditions under which S D is small for a given D is a compelling problem.

8 Conclusion

Memorization originally focuses on theoretical study of the expressive power of neural networks. Recently, memorization is believed to be a key reason why over-parameterized deep learning models have excellent generalizability and thus the more practical interpolation learning approach has been extensively studied. But the generalizability theory of memorization algorithms is not yet given, and this paper fills this theoretical gap in several aspects.

We first show how to construct memorization networks for dataset sampled i.i.d from a data distribution, which have the optimal number of parameters, and then show that some commonly used memorization networks do not have generalizability even if the dataset is drawn i.i.d. from a data distribution and contains a sufficiently large number of samples. Furthermore, we establish the sample complexity of memorization algorithm in several situations, including a lower bound for the memorization sample complexity and an upper bound for the efficient memorization sample complexity.

Limitation and future work Two numerical complexities N D and S D for a data distribution D are introduced in this paper, which are used to describe the size of the memorization networks and the efficient memorization sample complexity for any i.i.d. dataset of D . N D is also a lower bound for the sample complexity of memorization algorithms. However, we do not know how to compute N D and S D , which is an interesting future work. Conjecture 6.7 tries to give a lower bound for the efficient memorization sample complexity. More generally, can we write N D and S D as functions of the probability density function p ( x, y ) of D ?

Corollary 6.4 indicates that even for the 'nice' data distributions D ( n, c ) , to achieve generalization for some data distribution requires an exponential number of parameters. This indicates that there exists 'data curse of dimensionality' , that is, to achieve generalizability for certain data distribution, neural networks with exponential number of parameters are needed. Considering the practical success of deep learning and the double descent phenomenon [45], the data distributions used in practice should have better properties than D ( n, c ) , and finding data distributions with polynomial size efficient memorization sample complexity E D is an important problem.

Finally, finding a memorization algorithm that can achieve SOTA results in solving practical image classification problems is also a challenge problem.

Acknowledgments

This work is supported by CAS Project for Young Scientists in Basic Research, Grant No.YSBR-040, ISCAS New Cultivation Project ISCAS-PYFX-202201, and ISCAS Basic Research ISCAS-JCZD202302. This work is also supported by NKRDP grant No.2018YFA0704705, grant GJ0090202, and NSFC grant No.12288201. The authors thank anonymous referees for their valuable comments.

References

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 242-252. PMLR, 09-15 Jun 2019.

Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. arXiv preprint arXiv:1611.01491 , 2016.

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning , pages 233-242. PMLR, 2017.

Peter Bartlett, Vitaly Maiorov, and Ron Meir. Almost linear vc dimension bounds for piecewise polynomial networks. In M. Kearns, S. Solla, and D. Cohn, editors, Advances in Neural Information Processing Systems , volume 11. MIT Press, 1998.

Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight vcdimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research , 20(1):2285-2301, 2019.

Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica , 30:87-201, 2021.

Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems , 33:4381-4391, 2020.

Eric B. Baum. On the capabilities of multilayer perceptrons. Journal of Complexity , 4(3): 193-215, 1988.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Sebastien Bubeck, Ronen Eldan, Yin Tat Lee, and Dan Mikulincer. Network size and size of the weights in memorization with two-layers neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 4977-4986. Curran Associates, Inc., 2020.

Niladri S Chatterji and Philip M Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. Journal of Machine Learning Research , 22(129):1-30, 2021.

Amit Daniely. Neural networks learning and memorization with (almost) no overparameterization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 9007-9016. Curran Associates, Inc., 2020.

Amit Daniely. Memorizing gaussians with no over-parameterizaion via gradient decent on neural networks. arXiv preprint arXiv:2003.12895 , 2020.

Bhaskar DasGupta, Hava T. Siegelmann, and Eduardo Sontag. On a learnability question associated to neural networks with continuous activations (extended abstract). In Proceedings of the Seventh Annual Conference on Computational Learning Theory , COLT '94, page 47-56, New York, NY, USA, 1994. Association for Computing Machinery.

Santanu S. Dey, Guanyi Wang, and Yao Xie. Approximation algorithms for training one-node relu neural networks. IEEE Transactions on Signal Processing , 68:6696-6706, 2020.

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 1675-1685. PMLR, 09-15 Jun 2019.

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations , 2019.

Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , STOC 2020, page 954-959, New York, NY, USA, 2020. Association for Computing Machinery.

Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 2881-2891. Curran Associates, Inc., 2020.

Vincent Froese, Christoph Hertrich, and Rolf Niedermeier. The computational complexity of relu network training parameterized by data dimensionality. Journal of Artificial Intelligence Research , 74:1775-1790, 2022.

Prachi Garg, Shivang Agarwal, Alexis Lechervy, and Frederic Jurie. Memorization and generalization in deep cnns using soft gating mechanisms. https://prachigarg23.github.io/reports/Report-GREYC.pdf , 2019.

Surbhi Goel, Adam Klivans, Pasin Manurangsi, and Daniel Reichman. Tight hardness results for training depth-2 relu networks. arXiv preprint arXiv:2011.13550 , 2020.

Paul Goldberg and Mark Jerrum. Bounding the vapnik-chervonenkis dimension of concept classes parameterized by real numbers. In Proceedings of the sixth annual conference on Computational learning theory , pages 361-369, 1993.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference On Learning Theory , volume 75 of Proceedings of Machine Learning Research , pages 297-299. PMLR, 06-09 Jul 2018.

Moritz Hardt and Tengyu Ma. Identity matters in deep learning. arXiv preprint arXiv:1611.04231 , 2016.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pages 1225-1234. PMLR, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016.

Guang-Bin Huang. Learning capability and storage capacity of two-hidden-layer feedforward networks. IEEE Transactions on Neural Networks , 14(2):274-281, 2003.

Guang-Bin Huang and H.A. Babri. Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions. IEEE Transactions on Neural Networks , 9(1):224-229, Jan 1998.

Shih-Chi Huang and Yih-Fang Huang. Bounds on number of hidden neurons of multilayer perceptrons in classification and recognition. In Proceedings of 1990 IEEE International Symposium on Circuits and Systems (ISCAS) , pages 2500-2503, 1990.

Adam R. Klivans and Alexander A. Sherstov. Cryptographic hardness for learning intersections of halfspaces. Journal of Computer and System Sciences , 75(1):2-12, 2009.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report TR-2009 , 2009.

Ilja Kuzborskij and Christoph Lampert. Data-dependent stability of stochastic gradient descent. In International Conference on Machine Learning , pages 2815-2824. PMLR, 2018.

Binghui Li, Jikai Jin, Han Zhong, John E Hopcroft, and Liwei Wang. Why robust generalization in deep learning is difficult: Perspective of expressive power. arXiv preprint arXiv:2205.13863 , 2022.

Hongkang Li, Meng Wang, Sijia Liu, and Pin-Yu Chen. A theoretical understanding of shallow vision transformers: Learning, generalization, and sample complexity. arXiv preprint arXiv:2302.06015 , 2023.

Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Tengyuan Liang and Benjamin Recht. Interpolating classifiers make few mistakes. Journal of Machine Learning Research , 24:1-27, 2023.

Roi Livni, Shai Shalev-Shwartz, and Ohad Shamir. On the computational efficiency of training neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 3325-3334. PMLR, 10-15 Jul 2018.

Pasin Manurangsi and Daniel Reichman. The computational complexity of training relu (s). arXiv preprint arXiv:1810.04207 , 2018.

Nimrod Megiddo. On the complexity of polyhedral separability. Discrete & Computational Geometry , 3:325-337, 1988.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124003, 2021.

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. Advances in neural information processing systems , 30, 2017.

Quynh Nguyen and Matthias Hein. Optimization landscape and expressivity of deep cnns. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 3730-3739. PMLR, 10-15 Jul 2018.

Sejun Park, Chulhee Yun, Jaeho Lee, and Jinwoo Shin. Minimum width for universal approximation. arXiv preprint arXiv:2006.08859 , 2020.

Sejun Park, Jaeho Lee, Chulhee Yun, and Jinwoo Shin. Provable memorization via deep neural networks using sub-linear parameters. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 3627-3661. PMLR, 15-19 Aug 2021.

Michael A. Sartori and Panos J. Antsaklis. A simple method to derive bounds on the size and to train multilayer neural networks. IEEE Transactions on Neural Networks , 2(4):467-471, 1991.

Shalev-Shwartz Shai and Ben-David Shai. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Rahul Sharma, Aditya V. Nori, and Alex Aiken. Interpolants as classifiers. In CAV 2012, LNCS 7358 , pages 71-87, 2012.

Ryan Theisen, Jason M. Klusowski, and Michael W. Mahoney. Good classifiers are abundant in the interpolating regime. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics , pages 15532-15543, 2021.

Vladimir N. Vapnik. An overview of statistical learning theory. IEEE Transactions On Neural Networks , 10(5):988-999, 1999.

Gal Vardi, Gilad Yehudai, and Ohad Shamir. On the optimal memorization power of relu neural networks. arXiv preprint arXiv:2110.03187 , 2021.

Roman Vershynin. Memory capacity of neural networks with threshold and rectified linear unit activations. Siam J. Math. Data Sci. , 2(4):1004-1033, 2020.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Yihan Wang, Shuang Liu, and Xiao-Shan Gao. Data-dependent stability analysis of adversarial training. arXiv preprint arXiv:2401.03156 , 2021.

Zhen Wang, Lan Bai, and Yuanhai Shao. Generalization memorization machine with zero empirical risk for classification. Pattern Recognition , 152:110469, 2024.

Jiancong Xiao, Yanbo Fan, Ruoyu Sun, Jue Wang, and Zhi-Quan Luo. Stability analysis and generalization bounds of adversarial training. Advances in Neural Information Processing Systems , 35:15446-15459, 2022.

Yue Xing, Qifan Song, and Guang Cheng. On the algorithmic stability of adversarial training. Advances in neural information processing systems , 34:26523-26535, 2021.

Lijia Yu, Xiao-Shan Gao, and Lijun Zhang. Optimal robust memorization with relu neural networks. In International Conference on Learning Representations , 2024.

Lijia Yu, Shuang Liu, Yibo Miao, Xiao-Shan Gao, and Lijun Zhang. Generalization bound and new algorithm for clean-label backdoor attack. In Proceedings of the 41st International Conference on Machine Learning , pages 235:57559-57596, 2024.

Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small relu networks are powerful memorizers: a tight analysis of memorization capacity. In Advances in Neural Information Processing Systems , volume 32, pages 15532-15543, 2019.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3): 107-115, 2021.

Lijia Zhou, Frederic Koehler, Danica J. Sutherland, and Nathan Srebro. Optimistic rates: A unifying theory for interpolation learning and regularization in linear regression. ACM/JMS Journal of Data Science , 1(2):1-51, 2024.

Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep relu networks. arXiv preprint arXiv:1811.08888 , 2018.

A Proof of Proposition 3.3

Using the following steps, we construct a distribution D in [0 , 1] ×{-1 , 1 } . We use ( x, y ) ∼ D to mean that

Randomly select a number in {-1 , 1 } as the label y .

If we get 1 as the label, then randomly select an irrational number in [0 , 1] as samples x ; if we get -1 as the label, then randomly select a rational number in [0 , 1] as samples x .

Then Proposition 3.3 follows from the following lemma.

Lemma A.1. For any neural network F , we have A D ( F ) ≤ 0 . 5 .

Proof. Let F be a network. Firstly, we show that F can be written as

) = 1

x

I

(

L

i

(

x

)

I

(

if

x

x

∈

is true or

A

i

)

I

(

x

) = 0

. In addition,

A

i

is an interval and is a non-negative or non-positive function for any

where

L

A

j

∩

A

i

∈

[

i

i

M

]

.

It is obvious that the network is a locally linear function with a finite number of linear regions, so we can write

̸

where L ′ i are linear functions, A ′ i is an interval and A ′ j ∩ A ′ i = ∅ when j = i .

Consider that L ′ i ( x ) I ( x ∈ A ′ i ) = L ′ i ( x ) I ( x ∈ A ′ i , L ′ i ( x ) > 0) + L ′ i ( x ) I ( x ∈ A ′ i , L ′ i ( x ) < 0) , and L ′ i ( x ) I ( x ∈ A ′ i , L ′ i ( x ) > 0) is a non-negative function, { x ∈ A ′ i , L ′ i ( x ) > 0 } is an interval which is disjoint with { x ∈ A ′ i , L ′ i ( x ) < 0 } . Similarly as L ′ i ( x ) I ( x ∈ A ′ i , L ′ i ( x ) < 0) , so we use L ′ i ( x ) I ( x ∈ A ′ i ) in (2) instead of L ′ i ( x ) I ( x ∈ A ′ i , L ′ i ( x ) > 0) + L ′ i ( x ) I ( x ∈ A ′ i , L ′ i ( x ) < 0) . Then we get the equation (1).

By equation (2), we have that

The second equation uses A i ∩ A j = ∅ .

For convenience, we use x ∈ R r to mean that x is an irrational number and x / ∈ R r to mean that x is a rational number. Then, if L i ( x ) I ( x ∈ A i ) is a non-negative function, then we have P ( x,y ) ∼D ( Sgn ( L i ( x ) I ( x ∈ A i )) = y | x ∈ A i ) ≤ P ( x,y ) ∼D ( x ∈ R r | x ∈ A i ) . Moreover, we have that

.

By (2) in the definition of D , we have P ( x,y ) ∼D ( x ∈ A i | x ∈ R r ) = P ( x,y ) ∼D ( x ∈ A i | x / ∈ R r ) Substituting this in equation (3), we have that P ( x,y ) ∼D ( Sgn ( L i ( x ) I ( x ∈ A i )) = y | x ∈ A i ) ≤ P ( x,y ) ∼D ( x ∈ R r | x ∈ A i ) = P ( x,y ) ∼D ( x ∈ A i | x ∈ R r ) P ( x,y ) ∼D ( x ∈ A i | x ∈ R r )+ P ( x,y ) ∼D ( x ∈ A i | x/ ∈ R r ) = 0 . 5 . Proof is similar when L i ( x ) I ( x ∈ A i ) is a non-positive function.

are linear functions,

=

∅

when

j

̸

=

i

, and

Using this in equation (2), we have that

The lemma is proved.

B Proof of Theorem 4.1

For the proof of this theorem, we mainly follow the constructive approach of the memorization network in [55]. Our proof is divided into four parts.

B.1 Data Compression

The general method of constructing memorization networks will compress the data into a low dimensional space at first, and we follow this approach. We are trying to compress the data into a 1-dimensional space, and we require the compressed data to meet some conditions, as shown in the following lemma.

Lemma B.1. Let D be a distribution in [0 , 1] n ×{-1 , 1 } with separation bound c and D tr ∼ D N . Then, there exist w ∈ R n and b ∈ R such that

(1): O ( nN 2 /c ) ≥ wx + b ≥ 1 for all x ∈ [0 , 1] n ;

To prove this lemma, we need the following lemma.

Lemma B.2. For any v ∈ R n and T ≥ 1 , let u ∈ R n be uniformly randomly sampled from the hypersphere S n -1 . Then we have P ( |⟨ u, v ⟩| < || v || 2 T √ 8 nπ ) < 2 T .

This is Lemma 13 in [49]. Now, we prove the lemma B.1.

Proof. Let c 0 = min ( x, -1) , ( z, 1) ∈D tr || x -z || 2 . Then, we prove the following result:

Result R1: Let u ∈ R n be uniformly randomly sampled from the hypersphere S n -1 , then there are P ( |⟨ u, ( x -z ) ⟩| ≥ c 0 4 N 2 √ 8 nπ , ∀ ( x, -1) , ( z, 1) ∈ D tr ) > 0 . 5 .

By lemma B.2, and take T = 4 N 2 , for any x, z which satisfies ( x, -1) , ( z, 1) ∈ D tr , we have that: let u ∈ R n be uniformly randomly sampled from the hypersphere S n -1 , then there are

We proved Result R1.

In practice, to find such a vector, we can randomly select a vector u in hypersphere S n -1 , and verify that if it satisfies |⟨ u, ( x -z ) ⟩| ≥ c 0 4 N 2 √ 8 nπ , ∀ ( x, -1) , ( z, 1) ∈ D tr . Verifying such a fact needs poly ( B ( D tr )) times. If such a u is not what we want, randomly select a vector u and verify it again. In each selection, with probability 0 . 5 , we can get a vector we need, so with ln 1 /ϵ times the selections, we can get a vector we need with probability 1 -ϵ .

Construct w,b and verify their rationality

By the above result, we have that: there exists a u ∈ R n such that || u || 2 = 1 and |⟨ u, ( x -z ) ⟩| ≥ c 0 4 N 2 √ 8 nπ , ∀ ( x, -1) , ( z, 1) ∈ D tr , and we can find such a u in poly ( B ( D tr ) , ln(1 /ϵ )) times.

Now, let w = 16 √ nN 2 c 0 u and b = || w || 2 √ n +1 , then we show that w and b are what we want:

(1): We have

O

(

nN

2

/c

)

wx

+

b

1

for all

x

∈

[0

,

1]

n

.

Firstly, because D is defined in [0 , 1] n ×{-1 , 1 } , so it holds || x || 2 ≤ √ n for any ( x, y ) ∈ D tr , and consequently wx + b ≥ b -|| w || 2 √ n ≥ 1 .

On the other hand, | wx | ≤ || w || 2 √ n ≤ O ( nN 2 c 0 ) , so wx + b ≤ | wx | + b ≤ O ( nN 2 /c 0 ) ≤ O ( nN 2 /c ) .

(2): We have

|

w

(

x

-

z

)

4

for all

(

x,

1)

,

(

z,

-

1)

∈ D

tr

.

By Definition 3.1, we know that c 0 ≥ c . So, w and b are what we want. The lemma is proved.

B.2 Data Projection

The purpose of this part is to map the compressed data into appropriate values.

Let w ∈ R n and b ∈ R be given and D tr = { ( x i , y i ) } N i =1 . Without losing generality, we assume that 0 < wx i < wx i +1 .

In this section, we show that, after compressing the data into 1-dimension, we can use a network F to map wx i + b to v [ i [ √ N ] ] , where { v j } [ N [ √ N ] ] j =0 ∈ R + are given values. This network has O ( √ N ) parameters and width 4, as shown in the following lemma.

Lemma B.3. Let { x i } N i =1 ⊂ R + , { v j } [ N [ √ N ] ] j =0 ⊂ R + . Assume that x i < x i +1 . Then a network F with width 4 and depth O ( √ N ) (at most O ( √ N ) parameters) can be obtained such that F ( x i ) = v [ i √ N ] for all i ∈ [ N ] .

Proof. Let F i ( x ) be the i -th hidden layer of network F , ( F i ) j be the j -th nodes of i -th hidden layer of network F .

Let q i = x i +1 -x i and t ( i ) = argmax j ∈ [ N ] { [ j/ √ N ] = i } . Consider the following network F :

The 2 i +1 hidden layer has width 4, and each node is:

For the case i = 0 , let ( F 0 ) 2 ( x ) = x and ( F 1 ) 3 ( x ) = v 0 .

The (2 i +2) -th hidden layer is:

The output is F ( x ) = ( F 2[ N/ √ N ] ) 1 ( x ) .

√

This network has width 4 and O ( N ) hidden layers. We can verify that such a network is what we want as follows.

Then, for v i +1 -v i q t ( i ) / 3 (( F 2 i +1 ) 1 ( x ) -( F 2 i +1 ) 2 ( x )) = v i -v i -1 q t ( i ) / 3 (Relu( x -x t ( i )+1 +2 q t ( i ) / 3) -Relu( x -x t ( i )+1 + q t ( i ) / 3) , easy to verify that, when x ≤ x t ( i ) , it is 0 ; when x ≥ x t ( i +1) , it is v i +1 -v i .

By the above two results, we have that ( F 2 i +2 ) 1 ( x ) = Relu(( F 2 i +1 ) 3 ( x )+ v i -v i -1 q t ( i ) / 3 (( F 2 i +1 ) 1 ( x ) -( F 2 i +1 ) 2 ( x ))) = Relu(( F 2 i ) 1 ( x )) when x ≤ x t ( i ) ; and ( F 2 i +2 ) 1 ( x ) = Relu(( F 2 i +1 ) 3 ( x ) + v i -v i -1 q t ( i ) / 3 (( F 2 i +1 ) 1 ( x ) -( F 2 i +1 ) 2 ( x ))) = Relu(( F 2 i ) 1 ( x ) + v i +1 -v i ) when x ≥ x t ( i )+1 .

So, we have that, if t ( i -1)+1 ≤ j ≤ t ( i ) , there are ( F 2 ) 1 ( x j ) = v 0 , ( F 4 ) 1 ( x j ) = v 1 -v 0 + v 0 = v 1 , ( F 6 ) 1 ( x j ) = v 2 -v 1 + v 1 = v 2 , . . . , ( F 2 i ) 1 ( x j ) = v i -v i -1 + v i -1 = v i ; and F ( x j ) = ( F 2[ N/ √ N ] ) 1 ( x j ) = ( F 2[ N/ √ N ] -2 ) 1 ( x j ) = · · · = ( F 2 i ) 1 ( x j ) = v i .

So, by the definition of t ( i ) , we have that F ( x j ) = v [ j √ N ] , such F is what we what and the lemma is proved.

B.3 Label determination

The purpose of this part is to use the values to which the compressed data are mapped, mentioned in the above section, to determine the labels of the data.

Assuming x i is compressed to c i where c i ≥ 1 is given in section B.1. Value v i in section B.2 is designed as: v i = [ c i [ √ N ]+1 ] . . . [ c ( i +1)[ √ N ] ] , where we treat [ c j ] as a w digit number for all j ( w is a given number). If there exist not enough digits for some c j , we fill in 0 before it, and we use ab to denote the integer by putting a and b together.

First, prove a lemma.

Lemma B.4. For a given N , there exists a network f : R → R 2 with width 4 and at most O ( w ) parameters such that, for any w digit number a i > 0 , we have f ( a 1 a 2 . . . a N ) = ( a 1 , a 2 . . . a N ) .

Proof. Firstly, we show that, for any a > b > 0 , there exists a network F a,b ( x ) : R + → R + with depth 2 and width 3, such that F a,b ( x ) = x when x ∈ [0 , a ] , and F a,b ( x ) = x -a when x ∈ [ a + b, 2 a ] .

We just need to take F a,b ( x ) = Relu( x ) -a/b Relu( x -a ) + a/b Relu( x -( a + b )) .l It is easy to verify that this is what we want.

Now, let q ∈ N + satisfy 2 q ≤ 10 w +1 -1 an 2 q +1 > 10 w +1 and p < 1 10 wN . We consider the following network:

and show that, F ( a 1 a 2 . . . a N / 10 w ( N -1) ) = a 2 . . . a N / 10 w ( N -1) .

Firstly, we have F 2 q ,p ( a 1 a 2 . . . a N / 10 w ( N -1) ) = a 1 ( q ) a 2 . . . a N / 10 w ( N -1) , where a 1 ( q ) = a 1 if a 1 a 2 . . . a N / 10 w ( N -1) ≤ 2 q and a 1 ( q ) = a 1 -2 q if a 1 a 2 . . . a N / 10 w ( N -1) > 2 q + p . Just by the definition of q , we know that there must be a 1 a 2 . . . a N / 10 w ( N -1) ≤ 2 q +1 . Further by the definition of p , one of the following two inequalities is true:

a 1 a 2 . . . a N / 10 w ( N -1) < 2 q or a 1 a 2 . . . a N / 10 w ( N -1) > 2 q + p .

So using the definition of F 2 q ,p , we get the desired result.

Similarly as before, for k = q -1 , q -2 , . . . , 0 , we have F 2 k ,p ( a 1 ( k +1) a 2 . . . a N / 10 w ( N -1) ) = a 1 ( k ) a 2 . . . a N / 10 w ( N -1) , where a 1 ( k ) = a 1 ( k + 1) if a 1 ( k +1) a 2 . . . a N / 10 w ( N -1) ≤ 2 k and a 1 ( k ) = a 1 ( k +1) -2 k if a 1 ( k +1) a 2 . . . a N / 10 w ( N -1) > 2 k + p .

Then we have the following result: a 1 ( k ) < 2 k for any k = 0 , 1 , . . . , q . By the definition, it is easy to see that a 1 < 2 q +1 . If a 1 < 2 q , then a 1 ( q ) ≤ a 1 < 2 q ; if a 1 ≥ 2 q , then a 1 a 2 . . . a N / 10 w ( N -1) > 2 q + p , so a 1 ( q ) = a 1 -2 q < 2 q +1 -2 q = 2 q . Thus a 1 ( q ) < 2 q . When a 1 ( t ) < 2 t for a t ∈ [ q ] , similar as before, we have a 1 ( t -1) < 2 t -1 . And t = q is proved, so we get the desired result.

It is easy to see that, a 1 ( k ) are non negative integers, so there must be F ( a 1 a 2 . . . a N / 10 w ( N -1) ) = a 1 (0) a 2 . . . a N / 10 w ( N -1) = a 2 . . . a N / 10 w ( N -1) , by a 1 (0) < 2 0 = 1 , which implies a 1 (0) = 0 .

Now we construct a network F b as follows:

F b 1 ( x ) : R → R 2 and F b 1 ( x ) = ( F ( x/ 10 w ( N -1) ) , x ) where x is defined as before.

Now we verify that F b is what we want.

By the structure of F , F b has width 4 and depth O ( w ) , so there are at most O ( w ) parameters.

It is easy to see that F b 1 ( a 1 a 2 . . . a N ) = ( a 2 . . . a N / 10 w ( N -1) , a 1 a 2 . . . a N ) . Then by the definition of F b 2 ( x ) , we have F b ( x ) = ( a 1 , a 2 . . . a N ) , this is what we want. The lemma is proved.

By the preceding lemma, we have the following lemma.

Lemma B.5. There is a network R 2 → R with at most O ( Nw ) parameters and width 6 , and for any { a i } N i =1 where a j is a w digit number and a j ≥ 1 , which satisfies f ( x, a 1 a 2 . . . a N ) > 0 . 1 if | x -a k | < 1 for some k ∈ [ N ] , and f ( x, a 1 a 2 . . . a N ) = 0 if | x -a k | ≥ 1 . 1 for all k ∈ [ N ] .

Proof. The proof idea is as follows: First, we use x and a 1 a 2 . . . a N to judge if | x -a 1 | < 1 as follows: Using lemma B.4, we calculate a 1 and a 2 . . . a N and then calculate | x -a 1 | .

If | x -a 1 | < 1 , then we let the network output a positive number; if | x -a 1 | ≥ 1 , then calculate a 2 . . . a N , and use x and a 2 . . . a N to repeat the above process until all | x -a i | have been calculated.

The specific structure of the network is as follows:

step 1: Firstly, for a given N , we introduce a sub-network f s : R 2 → R 2 , which satisfies ( f s ) 1 ( x, a 1 a 2 . . . a N ) > 0 . 1 if | x -a 1 | < 1 , and f s ( x, a 1 a 2 . . . a N ) = 0 if | x -a 1 | ≥ 1 . 1 , and ( f s ) 2 ( x, a 1 a 2 . . . a N ) = a 2 . . . a N . And f s has O ( w ) parameters and width 5 .

The first part of f s is to calculate a 1 and a 2 . . . a N by lemma B.4. We also need to keep x , and the network has width 5 . The second part of f s is to calculate | x -a 1 | and keep a 2 . . . a N by using | x | = Relu( x ) + Relu( -x ) , which has width 4. The output of f s is Relu(1 . 1 -| x -a 1 | ) . Easy to check that this is what we want.

step 2: Now we build the f mentioned in the lemma.

For each i ∈ [ N ] , we will let the input of f i which is also the output of f i -1 when i > 1 be the form ( x, a i a i +1 . . . a N , q i ) , where q 1 = 0 . The detail is as follows:

For i ∈ [ N ] , in f i , construct f s ( x, a i a i +1 . . . a N ) at first, and then let q i +1 = q i + ( f s ) 1 ( x, a i a i +1 . . . a N ) , to keep q i in each layer, where we need one more width than f s . Then, output ( x, a i +1 a i +2 . . . a N , q i +1 ) , which is also the input of ( i +1) -th part.

The output of f is q N +1 , that is, g ( x, 0 , q N +1 ) = q N +1 . Now, we show that, f is what we want.

(1): f has at most O ( Nw ) parameters and width 6, which is obvious, because each part f i , f i has O ( w ) parameters by lemma B.4, and f has at most N parts, so we get the result.

This is because when | x -a k | < 1 , the k -th part will make q k +1 = q k + f s ( x, a k a k +1 . . . a N ) > 0 . 1 , because ( f s ) 1 ( x, a k a k +1 . . . a N ) > 0 . 1 as said in step 1. Since q j +1 = q j +( f s ) 1 ≥ q j , we have f ( x, a 1 a 2 . . . a N ) = q N +1 ≥ q k +1 > 0 . 1 .

This is because when | x -a k | ≥ 1 . 1 , the k -th part will make q k +1 = q k + f s ( x, a k a k +1 . . . a N ) = q k , because f s ( x, a k a k +1 . . . a N ) = 0 as said in step 1. Since f s ( x, a k a k +1 . . . a N ) = 0 for all k , we have f ( x, a 1 a 2 . . . a N ) = q N +1 = q N + f s ( x, a N ) = q N = · · · = q 0 = 0 .

B.4 The proof of Theorem 4.1

Now, we will prove Theorem 4.1. As we mentioned before, three steps are required: data compression, data projection, and label determination. The proof is as follows.

̸

Proof. Assume that D tr = { x i } N i =1 , without loss of generality, let x i = x j . Now, we show that there is a memorization network F of D tr with O ( √ N ) parameters.

Part One, data compression.

The part is to compress the data in D tr into R . Let w,b satisfy (1) and (2) in lemma B.1. Then, the first part of F is f 1 ( x ) = Relu( wx + b ) .

Part two, data projection.

Let c i = f 1 ( x i ) , without loss of generality, we assume c i ≤ c i +1 and y 1 = 1 . We define c ′ i as: c ′ i = c i if x i has label 1 ; otherwise c ′ i = c 1 .

In this part, the second part of F ( x ) , named as f 2 ( x ) : R → R 2 , need to satisfy f 2 ( c i ) = ( v [ i √ N 0 ] , c i ) for any i ∈ [ N ] .

By lemma B.3, a network with O ( √ N ) parameters and width 4 is enough to map x i to v [ i √ N ] and for keeping the input, and one node is needed at each layer. So f 2 just need O ( √ N ) parameters and width 5 .

Part Three, Label determination.

In this part, we will use the v k mentioned in part two to output the label of input. The third part, nameed as f 3 ( v, c ) , should satisfy that:

Because the number of digits for c i is O (ln( nN/c )) by (1) in lemma B.1 and lemma B.5, we know that such a network need O ( √ N ln( Nn/c )) parameters.

Construction of F and verify it:

Let F ( x ) = f 3 ( f 2 ( f 1 ( x ))) -0 . 05 . We show that F is what we want.

√

(1): By parts one, two, three, it is easy to see that F has at most O ( N ln( Nn/c )) parameters and width 6 .

(2): F ( x ) is a memorization of D tr . For any ( x i , y i ) ∈ D tr , consider two sub-cases:

(1.1: if y i = 1 ): Using the symbols in Part Two, f 2 ( f 1 ( x i )) will output ( v [ i √ N ] , f 1 ( x i )) . Since c ′ i = c i because y i = 1 , by part three, we have f 3 ( f 2 ( f 1 ( x ))) -0 . 05 ≥ 0 . 1 -0 . 05 > 0 .

The Running Time: In Part One, it takes poly ( B ( D tr ) , ln ϵ ) times to find such w and b with probability 1 -ϵ , as said in lemma B.1. In other parts, the parameters are calculated deterministically. We proved the theorem.

C Proof of Theorem 4.3

Proof. It suffices to show that there exists a memorization algorithm L , such that if D ∈ D ( n, c ) and D tr ∼ D N , then the network L ( D tr ) has a constant number of parameters (independent of N ). The construction has four steps.

Step One: Calculate the min ( x,y x ) , ( z,y z ) ∈D s || x -z || 2 , name it as c 0 .

Step Two: There is a D tr ⊂ D tr , such that:

(c1): For any ( x, y x ) , ( z, y z ) ∈ D s , it holds || x -z || 2 > c 0 / 3 ;

(c2): For any ( x, y x ) ∈ D tr , it holds || x -z || 2 ≤ c 0 / 3 for some ( z, y z ) ∈ D s .

It is obvious that such D s exists.

Step Three: We prove that |D s | ≤ (1+2 c 0 / 3) n C n ( c 0 / 3) n , where C n is the volume of unit ball in R n . Let Q = (1+2 c/ 3) n C n ( c/ 3) n , consider that c 0 ≥ c , so there are |D s | ≤ Q .

Let B 2 ( x, r ) = { z : || z -x || 2 ≤ r } , and V ( A ) the volume of A .

Due to D s ⊂ D tr ⊂ [0 , 1] n × {-1 , 1 } , so ∪ ( x,y ) ∈D s B 2 ( x, c 0 / 3) ∈ [ -c 0 / 3 , 1 + c 0 / 3] n . By condition (c1), we have B 2 ( x, c 0 / 3) ∩ B 2 ( z, c 0 / 3) = ∅ for any ( x, y x ) , ( z, y z ) ∈ D s , so we have ∑ ( x,y ) ∈D s V ( B 2 ( x, c 0 / 3)) ≤ (1 + 2 c 0 / 3) n , which means |D s | ≤ (1+2 c 0 / 3) n C n ( c 0 / 3) n < Q .

Step Four: There is a robust memorization network [62] with at most O ( Qn ) parameters for D s with robust radius c 0 / 3 , and this memorization network is a memorization of D tr .

By condition (c1), there is a robust memorization network F rm with O ( |D s | n ) parameters for D s with radius c 0 / 3 [62]. By step three, we have |D s | ≤ Q , so that such a network has at most O ( Qn ) parameters.

̸

By condition (c2), for any ( x, y x ) ∈ D tr , there is a ( z, y z ) ∈ D s satisfying || x -z || 2 ≤ c 0 / 3 . Firstly, there must be y x = y z , because the distribution D has separation bound c 0 , and if y x = y z then || x -z || 2 ≥ c 0 > c 0 / 3 . Then, since robust memorization F rm has robust radius c 0 / 3 , we have Sgn ( F rm ( x )) = Sgn ( F rm ( z )) = y z = y x , so F rm is a memorization network of D tr . The theorem is proved.

D Proof for Theorem 5.1

In this section, we will prove that networks with small width cannot have a good generalization for some distributions. For a given width w , we will construct a distribution on which any network with width w will have poor generalization. The proof consists of the following parts.

D.1 Disadvantages of network with small width

In this section, we demonstrate that a network with a small width may have some unfavorable properties. We have the following simple fact.

Lemma D.1. Let the first transition weight matrix of network F be W . Then if Wx = Wz , we have F ( x ) = F ( z ) .

If W is not full-rank, then there exist x and z satisfying Wx = Wz . Moreover, if x and z have different labels, according to lemma D.1, we have F ( x ) = F ( z ) , so there must be an incorrect result given between F ( x ) and F ( z ) .

According to the theorem of matrices decomposition, we also have the following fact.

Lemma D.2. Let the first transition weight matrix of network F : R n → R be W . If W has width w < n , then exists a W 1 ∈ R w × n , whose rows are orthogonal and unit such that W 1 x = W 1 z implies F ( x ) = F ( z ) .

Proof. Using matrix decomposition theory, we can write W = NW 1 , where N ∈ R w × w and W 1 ∈ R w × n and the rows of W 1 are orthogonal to each other and unit.

Next, we only need to consider W 1 as the first transition matrix of the network F and use lemma D.1.

At this point, we can try to construct a distribution where any network with small width will have poor generalization.

D.2 Some useful lemmas

In this section, we introduce some lemmas which are used in the proof in section D.3.

Lemma D.3. Let B ( r ) be the ball with radius r in R n . For any given δ > 0 , let ϵ = 2 δ/n . Then we have V ( B ( √ 1 -ϵr )) V ( B ( r )) > 1 -δ .

For w ∈ R a,b and q ∈ R a , let q ◦ w = ∑ a i =1 q i w i , where q i is the i -th weight of q , w i is the i -th row of w i . Then we have

Lemma D.4. Let W ∈ R w × n , and its rows are unit and orthogonal.

̸

(1): For any q = q ∈ R , we have

(3): For any q ∈ R w , { x ∈ R n : Wx = W ( q ◦ W ) , x ∈ S } is a ball in R n -w with volume (1 -|| q || 2 2 ) ( n -w ) / 2 C n -w , where C i is the volume of the unit ball in R i .

Proof. First, we define an orthogonal coordinate system { W i } n i =1 in R n . Let W i be the i -th row of W when i ≤ w . When i > w , let W i be a unit vector orthogonal with all W j where j < i .

Then for all x ∈ R n , we say ˜ x i is the i -th weight of x under such coordinate system. Then, Wx = Wz if and only ˜ x i = ˜ z i for i ∈ [ w ] .

Now, we can prove the lemma.

(1): The first weight w of q 1 ◦ W under orthogonal coordinate system { W i } n i =1 is q 1 , so if x ∈ { x ∈ R n : Wx = W ( q 1 ◦ W ) } , we have ˜ x i = ( q 1 ) i for i ∈ [ w ] .

̸

The first w weight of q 2 ◦ W under orthogonal coordinate system { W i } n i =1 is q 2 , so if x ∈ { x ∈ R n : Wx = W ( q 2 ◦ W ) } , we have ˜ x i = ( q 2 ) i for i ∈ [ w ] . Because q 1 = q 2 ∈ R w , we get the result.

(2): For any x ∈ R n , let q ( x ) = ( ˜ x 1 , ˜ x 2 , . . . , ˜ x w ) ∈ R w . It is easy to see that || x || 2 = √ ∑ n i =1 ˜ x i 2 , so || q ( x ) || 2 ≤ 1 when || x || 2 ≤ 1 .

Now we verify that: for any s ∈ S , we have s ∈ { x ∈ R n : Wx = W ( q ( s ) ◦ W ) , x ∈ S } .

Firstly, we have Ws = ∑ w i =1 < w i , ∑ N i =1 ˜ s i w i > = ∑ w i =1 ˜ s i .

Secondly, we have W ( q ( s ) ◦ W ) = ∑ w i =1 < w i , ∑ w i =1 ˜ s i w i > = ∑ w i =1 ˜ s i . So Ws = W ( q ( s ) ◦ W ) , resulting in s ∈ { x ∈ R n : Wx = W ( q ( s ) ◦ W ) , x ∈ S } , which implies that S = ∪ q ∈ R w , || q || 2 ≤ 1 { x ∈ R n : Wx = W ( q ◦ W ) , x ∈ S } .

(3): By the proof of (2), we know that if x satisfies ˜ x i = q i for i ∈ [ w ] , then x ∈ { x ∈ R n : Wx = W ( q ◦ W ) } . By (1), { x ∈ R n : Wx = W ( q ◦ W ) } will not intersect for different q . Therefore, x ∈ { x ∈ R n : Wx = W ( q ◦ W ) } equals ˜ x i = q i for i ∈ [ w ] .

Since || x || 2 = √ ∑ n i =1 ˜ x i 2 , when x ∈ { x ∈ R n : Wx = W ( q ◦ W ) } , we have ˜ x i = q i for i ∈ [ w ] , so ∑ n i = w +1 ˜ x i 2 = || x || 2 2 -|| q || 2 2 , and such n -w weight is optional.

Therefore, { x ∈ R n : Wx = W ( q ◦ W ) , x ∈ S } is a ball in R n -w with radius √ 1 -|| q || 2 2 , so we get the result.

Proof. Let f ( x ) = ( r 3 -x ) n -( r 2 -x ) n ( r 1 -x ) n . We just need to prove f ( x ) ≥ f (0) when x ≤ r 1 . We calculate the derivative f ( x ) at first:

It is easy to calculate that (( r 3 -x ) n -( r 2 -x ) n ) ′ = -n (( r 3 -x ) n -1 -( r 2 -x ) n -1 ) and (( r 1 -x ) n ) ′ = -n ( r 1 -x ) n -1 . Putting this into the above equation, we have

Where P ( x ) is a positive value about x . Since

we have f ′ ( x ) ≥ 0 , resulting in f ( x ) ≥ f (0) . The lemma is proved.

Lemma D.6. Let a > b > 1 , n > m ≥ 1 . If a n -b n = 1 . Then a m -b m ≤ 1 .

Lemma D.7. Let a > qb where q < 1 and a, b > 0 . Then min { a, b } ≥ qb .

Proof. When min { a, b } = b , by q < 1 , the result is obvious. When min { a, b } = a , by a > qb , the result is obvious.

Lemma D.8. For any w > 0 , there exist r 1 , r 2 , r 3 and n such that

Proof. Because the equations are all homogeneous, without loss of generality, we assume that r 1 = 1 . We take α = 2 1 /n -1 , β + α = 3 1 /n -1 , and n to satisfy 3 w/n < 1 . 001 . Let r 2 = 1 + α , r 3 = 1 + α + β . We show that this is what we want.

At first, we have r n 3 -r n 2 = (1 + α + β ) n -(1 + α ) n = 3 -2 = 1 = r n 1 . We also have (1 + α + β ) w < 1 . 001 , named (k1). So we have

The lemma is proved.

D.3 Construct the distribution

In this section, we construct the distribution in Theorem 5.1.

Definition D.9. Let q be a point in [0 , 1] n , 0 < r 1 < r 2 < r 3 , and we define B k 2 ( z, t ) = { x ∈ R k : || x -z || 2 ≤ t } , where k ∈ N + , z ∈ R k and t ≥ 0 .

The distribution D ( n, q, r 1 , r 2 , r 3 ) is defined as:

(1): This is a distirbution on R n ×{-1 , 1 } .

(2): A point has label 1 if and only if it is in B n 2 ( q, r 1 ) . A point has label -1 if and only if it is in B n 2 ( q, r 3 ) /B n 2 ( q, r 2 ) .

(3): The points with label 1 or -1 satisfy the uniform distribution, and let the density function be f ( x ) = λ = 1 V ( B n 2 ( q,r 3 )) -V ( B n 2 ( q,r 2 ))+ V ( B n 2 ( q,r 1 )) .

We now prove Theorem 5.1.

Proof. Use the notations in Definition D.9.

Lemma D.8 ensures that such r i , q, n exist.

Let distribution D = D ( n, q, r 1 , r 2 , r 3 ) , where D ( n, q, r 1 , r 2 , r 3 ) is given in Definition D.9. Now, we show that D is what we want. We prove that for any given F with width w , we have A D ( F ) < 0 . 51 .

Firstly, we define some symbols. Using lemma D.2, let W ∈ R w × n whose rows are unit and orthogonal and satisfy that Wx = Wz implying F ( x ) = F ( z ) .

By lemma D.2, we know that, for any given x , the points in S 1 ,x ∪ S 2 ,x have the same output after inputting to F , but the points in S 1 ,x have label 1 and the points in S 2 ,x have label -1. So F must give the wrong label to the point in S 1 ,x or S 2 ,x .

The proof is then divided into two parts.

Part One: Let h ∈ B w 2 (0 , r 1 ) , and x ( h ) = q + h ◦ W ∈ R n , where ◦ is defined in section D.2. Consider that for any given h , F must give the wrong label to the point in S 1 x ( h ) or S 2 x ( h ) , we have that F will give the wrong label with probability at least min { P ( x,y ) ∼D ( x ∈ S 1 x ( h ) ) , P ( x,y ) ∼D ( x ∈ S 2 x ( h ) ) } . So, now we only need to sum these values about h .

For any different h 1 , h 2 ∈ B w 2 (0 , r 1 ) , we have S 1 x ( h 1 ) ∩ S 1 x ( h 2 ) = ∅ , S 2 x ( h 1 ) ∩ S 2 x ( h 2 ) = ∅ , and ∪ h ∈ B w 2 (0 ,r 1 ) S 1 x ( h ) = B n 2 ( q, r 1 ) . By (1) and (2) in lemma D.4. Proof is similar for S 2 x ( h ) . Then, by the volume of S 1 x ( h ) , S 2 x ( h ) calculated in lemma D.4, we know that, the probability of F producing an error on distribution D is at least

where C n -w is the volume of the unit ball in R n -w as mentioned in lemma D.4. Next, we will estimate the lower bound of this value

Then, by lemma D.6 and (c2) , we know that ( r 2 3 ) ( n -w ) / 2 -( r 2 2 ) ( n -w ) / 2 ( r 2 1 ) ( n -w ) / 2 ≤ 1 . Thus by lemma D.7, we have

From r n 3 -r n 2 = r n 1 , we know that λV ( B n 2 ( q, r 1 )) = λ ( V ( B n 2 ( q, r 3 )) -V ( B n 2 ( q, r 2 ))) = 0 . 5 , so P ( x,y ) ∼D ( y = -1) = P ( x,y ) ∼D ( y = -1) = 0 . 5 , and further consider the (c3), we have

The theorem is proved.

E Proof of Theorem 5.3

Firstly, note that Theorem 5.3 cannot be proved by the following classic result.

Theorem E.1 ([57]) . Let D be any joint distribution over R n ×{-1 , 1 } , D tr a dataset of size N selected i.i.d. from D , and H = { h : R n → R } the hypothesis space. Then with probability at least 1 -δ ,

where R ( h, D ) is the population risk, R ( h, D tr ) is the empirical risk, and Rad N ( H ) is the Radermecher complexity of H .

Theorem E.1 is the classical conclusion about the lower bound of generalization error, and theorem 5.3 and Theorem E.1 are different. Firstly, Theorem E.1 is established on the basis of probability, whereas Theorem 5.3 is not. Secondly, Theorem E.1 highlights the existence of a gap between the empirical error and the generalization error for certain functions within the hypothesis space, and does not impose any constraints on the value of empirical error. However, memorization networks, which perfectly fit the training set, will inherently have a zero empirical error, so Theorem E.1 cannot directly address Theorem 5.3. Lastly, Theorem E.1 relies on Radermacher complexity, which can be challenging to calculate, while Theorem 5.3 does not have such a requirement.

For the proof of Theorem 5.3, we mainly follow the constructive approach of memorization network in [55], but during the construction process, we will also consider the accuracy of the memorization network. Our proof is divided into four parts.

E.1 Data Compression

The general method of constructing memorization networks compresses the data into a low dimensional space at first, and we adopt this approach. We are trying to compress the data into 1-dimension space. However, we require the compressed data to meet some conditions, as stated in the following lemma.

Lemma E.2. Let D be a distribution in [0 , 1] n ×{-1 , 1 } with separation bound c and density r , and D tr ∼ D N . Then, there are w ∈ R n and b ∈ R that satisfy:

(1): O ( nN 3 r/c ) ≥ wx + b ≥ 1 for all x ∈ [0 , 1] n ;

(2): | wx -wz | ≥ 4 for all ( x, 1) , ( z, -1) ∈ D tr ;

(3): P ( x,y ) ∼D ( ∃ ( z, y z ) ∈ D tr , | wx -wz | ≤ 3) < 0 . 01 .

Proof. Since distribution D is definition on [0 , 1] n , we have c ≤ 1 and r ≥ 1 .

Because the density function of D is r , we have P ( x,y ) ∼D ( x ∈ B 2 ( z, r 1 )) ≤ rV ( B 2 ( z, r 1 )) < r (2 r 1 ) n = 1 400 N 2 for all z ∈ R n , where r 1 = 1 2(400 rN 2 ) 1 /n . It is easy to see that r 1 ≤ 1 because r ≥ 1 .

Then, we have the following two results:

Result one: Let u ∈ R n be uniformly randomly sampled from the hypersphere S n -1 . Then we have P ( |⟨ u, ( x -z ) ⟩| ≥ c 4 N 2 √ 8 nπ , ∀ ( x, -1) , ( z, 1) ∈ D tr ) > 0 . 5 .The proof is similar to that of lemma B.1.

Result Two: Let u ∈ R n be uniformly randomly sampled from the hypersphere S n -1 . Then P u ( P ( x,y ) ∼D ( ∃ ( x i , y i ) ∈ D tr , |⟨ u, ( x -x i ) ⟩| < r 800 N 2 √ 8 nπ ) < 0 . 01) > 0 . 5 .

Firstly, by lemma B.2, and take T = 800 N 2 , we can get that: for any given v ∈ R n , if u ∈ R n be uniformly randomly sampled from the hypersphere S n -1 , then P ( |⟨ u, v ⟩| < || v || 2 800 N 2 √ 8 nπ ) < 1 400 N 2 . Thus, by such inequality, the density of D and the definition of r 1 , we have that:

On the other hand, we have

So, we have P u ( P ( x,y ) ∼D ( |⟨ u, ( x -v ) ⟩| < r 800 N 2 √ 8 nπ ) ≥ 0 . 01 /N ) < 1 200 N 2 / (0 . 01 /N ) = 1 / (2 N ) . Name this inequality as (*).

On the other hand, we have

Then, if a u ∈ R n satisfies P ( x,y ) ∼D ( ∃ ( x i , y i ) ∈ D tr , |⟨ u, ( x -x i ) ⟩| < r 800 N 2 √ 8 nπ ) ≥ 0 . 01 , then we have P ( x,y ) ∼D ( | u ( x -x i ) | < r 800 N 2 √ 8 nπ ) ≥ 0 . 01 /N for some ( x i , y i ) ∈ D tr .

So taking v as x i in inequality (*) and using the above result, we have

So we get the result. This is what we want.

Construct w,b and verify their property

Consider the fact: if A ( u ) , B ( u ) are two events about random variable u , and P u ( A ( u ) = True ) > 0 . 5 , P u ( B ( u ) = True ) > 0 . 5 , then there is a u , which makes events A ( u ) and B ( u ) occurring simultaneously. By the above fact and Results one and two, we have that there exist || u || 2 = 1

√ ∈

Now, let w = max { 2400 √ nN 2 r 1 , 16 √ nN 2 c } u and b = || w || 2 √ n +1 , then we show that w and b are what we want:

Firstly, because D is defined in [0 , 1] n ×{-1 , 1 } , we have || x || 2 ≤ √ n , resulting in and wx + b ≥ b -|| w || 2 √ n ≥ 1 .

On the other hand, using c ≤ 1 and r 1 ≤ 1 , we have | wx | ≤ || w || 2 √ n ≤ O ( nN 2 r 1 c ) , so wx + b ≤ | wx | + b ≤ O ( nN 3 r 1 /n /c ) .

(2): We have | w ( x -z ) | ≥ 4 for all ( x, 1) , ( z, -1) ∈ D tr .

It is easy to see that | w ( x -z ) | ≥ | 16 √ nN 2 c u ( x -z ) | = 16 √ nN 2 c | u ( x -z ) | . Because | u ( x -z ) | ≥ c 4 √ nN 2 , so | w ( x -z ) | = 16 √ nN 2 c | u ( x -z ) | ≥ 16 √ nN 2 c c 4 √ nN 2 = 4 .

(3): we have P ( x,y ) ∼D ( ∃ ( z, y z ) ∈ D tr , | wx -wz | ≤ 3) < 0 . 01 .

Because | w ( x -z ) | ≥ 2400 √ nN 2 r 1 | u ( x -z ) | ≥ | u ( x -z ) | , and consider that P ( x,y ) ∼D ( ∃ ( z, y z ) ∈ D tr , | u ( x -z ) | < r 1 800 N 2 √ 8 nπ ) < 0 . 01 , we get the result. So, w and b are what we want. and the lemma is proved.

E.2 Data Projection

The purpose of this part is to map the compressed data to appropriate values. Let w ∈ R n and b ∈ R be given, and D tr = { ( x i , y i ) } N i =1 . Without losing generality, we assume that wx i < wx i +1 .

In this section, we show that, after compressing the data into 1-dimension, we can use a network F to map wx i + b to v [ i [ √ N ] ] , where { v j } [ N [ √ N ] ] j =0 are the given values. Furthermore, F should also satisfy F ( wx + b ) ∈ { v j } [ N [ √ N ] ]+1 j =0 for all x ∈ [0 , 1] n except for a small portion.

This network has O ( √ N ) parameters, as shown below.

Lemma E.3. Let w ∈ R n and b ∈ R be given, { v j } [ N [ √ N ] ] j =0 ⊂ R and 1 > ϵ > 0 be given.

Let D tr = { ( x i , y i ) } N i =1 and D tr ∼ D N where D is a distribution, and assume that wx i + b < wx i +1 + b .

Then a network F with width O ( √ N ) , depth 2 , and at most O ( √ N ) parameters, can satisfy that: (1): F ( wx i + b ) = v [ i √ N ] for all i ∈ [ N ] ;

Proof. Let q i = ( wx i +1 + b ) -( wx i + b ) and q = min i { q i } . Then we consider the set of points S i = { wx i + b + qϵ 2 N ∗ j } [ N/ϵ ]+1 j =1 , for any i . We have that:

Consider that | S i | ≥ N/ϵ , so for any i , there is a s i ∈ S i , makes that P ( x,y ) ∼D ( wx + b ∈ ( s i -qϵ 2 N / 2 , s i + qϵ 2 N / 2)) ≤ ϵ N .

And it is easy to see that S i satisfies the following result: if z ∈ S i , then:

So we have ( s i -qϵ 2 N / 2 , s i + qϵ 2 N / 2) ∈ ( wx i + b, wx i +1 + b ) , Name this inequality as ( ∗ ) .

Let k = [ N [ √ N ] ] and t ( i ) = argmax j ∈ [ N ] { [ j/ √ N ] = i } . Now, we define such a network:

This network has width 2 k , depth 2 and O ( N ) parameters. We can verify that such networks satisfy (1) and (2).

√

Verify (1): For a given i ∈ [ N ] , let c ( i ) = [ i √ N ] . Then, when j < c ( i ) , we have t ( j ) < i , so s t ( j ) + qϵ 2 N / 2 ≤ wx t ( j )+1 + b ≤ wx i + b (this has been shown in ( ∗ ) ), resulting in: v j -v j -1 qϵ 2 N (Relu( wx i + b -s t ( j ) + qϵ 2 N / 2) -Relu( wx i + b -s t ( j ) -qϵ 2 N / 2) = v j -v j -1 . When j ≥ c ( i ) , similar to before, we have s t ( j ) -qϵ 2 N / 2 > wx i + b , resulting in v j -v j -1 qϵ 2 N (Relu( wx i + b -s t ( j ) + qϵ 2 N / 2) -Relu( wx i + b -s t ( j ) -qϵ 2 N / 2) = 0 . So F ( x i ) = v 0 +( v 1 -v 0 ) + · · · +( v c ( i ) -v c ( i ) -1 ) = v c ( i ) , this is what we want.

Verify (2): At first, we show that for any x ∈ [0 , 1] n satisying wx + b / ∈ ∪ k i =1 ( s i -qϵ 2 N / 2 , s i + qϵ 2 N / 2) , we have F ( x ) ∈ { v i } .

This is because: for any x satisfies wx + b / ∈ ∪ k i =1 ( s i -qϵ 2 N / 2 , s i + qϵ 2 N / 2) , we have F ( wx + b ) = v 0 +( v 1 -v 0 ) + · · · +( v k -v k -1 ) = v k , where k satisfies s t ( k ) < wx + b and k is the maximum. The proof is similar as above.

Second, we show that the probability of such x is at least 1 -ϵ .

E.3 Label determination

This is the same as in section B.3.

E.4 The proof of Theorem 5.3

Three steps are required: data compression, data projection, label determination. The specific proof is as follows.

̸

Proof. Assume that D tr = { x i } N i =1 , without loss of generality, let x i = x j . Now, we show that there is a memorization network F of D tr with O ( √ N ) parameters but with poor generalization.

Part One, data compression. The first part is to compress the data in D tr into R , let w,b satisfy (1),(2),(3) in lemma E.2. Then, the first part of F is f 1 ( x ) = Relu( wx + b ) .

On the other hand, not just samples in D tr , all the data in R n have been compressed into R by f 1 ( x ) . By (3) in lemma E.2, we have P ( x,y ) ∼D ( ∃ ( z, y z ) ∈ D tr , | wx -wz | ≤ 3) < 0 . 01 , resulting in, we have P ( x,y ) ∼D ( | wx -wz | > 3 for ∀ ( z, y z ) ∈ D tr ) > 0 . 99 . By the probability theory, we have

Without losing generality, we assume that P ( x,y ) ∼D ( ∀ ( z, y z ) ∈ D tr , | wx -wz | > 3 , y = 1) > 0 . 99 / 2 , which represents the following fact. Define S = { x : x has label 1 and | wx -wz | > 3 for ∀ ( z, y z ) ∈ D tr } . Then the probability of points in S is at least 0 . 99 / 2 . In the following proof, in order to make the network having bad generalization, we will make the network giving these points (the points in S ) incorrect labels.

Part two, data projection.

Let c i = f 1 ( x i ) / Without losing generality, we will assume c i ≤ c i +1 .

Now, assume that we have N 0 samples in D tr with label 1, and { i j } N 0 j =1 ⊂ [ N ] such that x i j has label 1, and i j < i j +1 . Let t ( i ) = argmax j ∈ [ N ] { [ j/ √ N 0 ] = i } and v k = [ c i t ( k -1)+1 ][ c i t ( k -1)+2 ] . . . [ c i t ( k ) ] .

In this part, the second part of F ( x ) , named as f 2 ( x ) , need to satisfy f 2 ( c i j ) = ( v [ j √ N ] , c i j ) . Furthermore, we also hope that P ( x,y ) ∼D ( f 2 ( f 1 ( x ))[1] ∈ { v i } ) ≥ 0 . 999 , where f 2 ( f 1 ( x ))[ i ] is the i -th weight of f 2 ( f 1 ( x )) , and P ( x,y ) ∼D ( f 2 ( f 1 ( x ))[2]) = f 1 ( x ) .

By lemma B.3, a network with O ( √ N ) parameters and depth 2 is enough to calculate v [ j √ N ] by c i j , and the output in { v i } has probability 0.999. Retaining c i just need one node. So f 2 need O ( √ N ) parameters.

Part Three, Label determination. In this part, we will use the v k mentioned in part two to output the label of inputs. The third part, named as f 3 ( v, c ) , should satisfy that for f 3 ( v k , c ) , where v k = [ c i t ( k -1)+1 ][ c i t ( k -1)+2 ] . . . [ c i t ( k ) ] as mentioned above, if | c -c i q | < 1 for some q ∈ [ t ( k -1) + 1 , t ( k )] , then f 3 ( v k , c ) > 0 . 1 , and f 3 ( v k , c ) = 0 when | c -c i q | ≥ 1 . 1 for all q ∈ [ t ( k -1) + 1 , t ( k )] .

This network need O ( √ N 0 ln( N 0 nr/c )) parameters, by (1) in lemma E.2 and lemma B.5.

Construction of F and verify it:

Let F ( x ) = f 3 ( f 2 ( f 1 ( x ))) -0 . 05 . We show that F is what we want.

(1): By parts one, two, three, and the fact N 0 ≤ N , it is easy to see that F has at most O ( n + √ N ln( Nnr/c )) parameters.

(2): F ( x ) is a memorization of D tr . For any ( x, y ) ∈ D tr , two cases are consided.

(1.1, if y = 1 ): using the symbols in Part two, because y = 1 , so x = x i k for some k . As mentioned in part two, f 2 ( f 1 ( x )) will output ( v i [ k √ N ] , f 1 ( x )) . Then, by part three, because | f 1 ( x ) -[ f 1 ( x )] | < 1 ,

0

so we have f 3 ( f 2 ( f 1 ( x ))) -0 . 05 ≥ 0 . 1 -0 . 05 > 0 .

(1.2 if y = -1 ): By (2) in lemma E.2, for ∀ ( z, 1) ∈ D tr , we know that | f 1 ( x ) -[ f 1 ( z )] | ≥ | f 1 ( x ) -f 1 ( z ) | - | f 1 ( z ) -[ f 1 ( z )] | ≥ 4 -1 = 3 . So, by part three, we have f 3 ( f 2 ( f 1 ( x ))) = 0 -0 . 05 < 0 .

(3): A D ( F ) < 0 . 51 . We show that, almost all x ∈ S ( S is mentioned in part one) will be given wrong label.

For x ∈ S , we have | wx -wx i | ≥ 3 , so | wx + b -[ wx i + b ] | ≥ 2 for all ( x i , y i ) ∈ D tr . Then for any v i , by part three and the definition of v i , we have f 3 ( v i , wx + b ) = 0 when x ∈ S . So, when f 2 ( f 1 ( x ))[1] ∈ { v i } and x ∈ S , we have f 3 ( f 2 ( f 1 ( x ))) -0 . 05 = 0 -0 . 05 < 0 .

Consider that for any x ∈ S , the label of x is 1 in distribution D . So when x ∈ S satisfies f 2 ( f 1 ( x ))[1] ∈ { v i } , we find that f ( x ) gives the wrong label to x . Since P ( x ∈ S ) ≥ 0 . 99 / 2 and P ( f 2 ( f 1 ( x ))[1] ∈ { v i } ) > 0 . 999 , we have P ( x ∈ S, f 2 ( f 1 ( x ))[1] ∈ { v i } ) ≥ 0 . 99 / 2 -0 . 001 > 0 . 49 .

̸

By the above result, we have that, with probability at least 0 . 49 , Sgn ( f ( x )) = y , so A D ( f ) < 0 . 51 . So, we prove the theorem.

F Proof of Theorem 6.1

We first give three simple lemmas.

Lemma F.1. We can find 2 [ n ⌈ c 2 ⌉ ] points in [0 , 1] n , and the distance between any two points shall not be less than c .

Proof. Let t = [ n ⌈ c 2 ⌉ ] . We just need to consider following points in [0 , 1] n :

For any given i 1 , i 2 , i 3 , . . . , i t ∈ { 0 , 1 } , let x i 1 ,i 2 ,i 3 ,...,i t be the vector in [0 , 1] n satisfying: for any j ∈ [ t ] , the ( j -1) ⌈ c 2 ⌉ +1 to j ⌈ c 2 ⌉ weights of x i 1 ,i 2 ,i 3 ,...,i t is i j ; other weights are 0.

̸

We will show that, if { i 1 , i 2 , i 3 , . . . , i t } = { j 1 , j 2 , j 3 , . . . , j t } , then it holds || x i 1 ,i 2 ,i 3 ,...,i t -x j 1 ,j 2 ,j 3 ,...,j t || 2 ≥ c . Without losing generality, let i 1 = j 1 . Then the first ⌈ c 2 ⌉ weights of x i 1 ,i 2 ,i 3 ,...,i t and x j 1 ,j 2 ,j 3 ,...,j t are different: one is all 1, and the other is all 0. So, the distance between such two points is at least √ ⌈ c 2 ⌉ ≥ c .

̸

Then { x i 1 ,i 2 ,i 3 ,...,i t } i j ∈ [0 , 1] is the 2 t point we want, so we prove the lemma.

Lemma F.2. If ϵ, δ ∈ (0 , 1) and k, x ∈ Z + satisfy that: x ≤ k (1 -2 ϵ -δ ) , then 2 x ( ∑ [ kϵ ] j =0 ( k -x j ) ) < 2 k (1 -δ ) .

Proof. We have

The first inequality sign uses ∑ m j =0 ( n m ) ≤ m 2 n /n where m ≤ n/ 2 , and by x ≤ k (1 -2 ϵ -δ ) , so [ kϵ ] ≤ ( k -x ) / 2 . The third inequality sign uses the fact x ≤ k (1 -2 ϵ -δ ) .

Lemma F.3. If k, v ∈ R + such that kv > 3 , and a = [ kv ] and 3 ≤ b ≤ √ k ln( √ k ) , then a ≥ ( b/ ln( b )) 2 v/ 2 .

Proof. If √ k ≤ b/ ln( b ) , then b ≤ √ k ln( √ k ) < √ k ln( b ) ≤ b , which is impossible. So b ≤ √ k ln( √ k ) , and then √ k ≥ b/ ln( b ) . Resulting in a ≥ kv -1 ≥ kv/ 2 ≥ ( b/ ln( b )) 2 v/ 2 .

Now, we prove Theorem 6.1

Proof. By Theorem 4.1, we know that there is a v 1 > 1 , when √ N ≥ n , for any distribution D ∈ D ( n, c ) and D tr ∼ D N , D tr has a memorization with v 1 √ N ln( Nn/c ) parameters. We will show that Theorem 6.1 is true for v = 1 32 v 2 1 .

Assume Theorem 6.1 is wrong, then there exists a memorization algorithm L such that for any n ∈ Z + , c, ϵ, δ ∈ (0 , 1) , if D ∈ D ( n, c ) and N ≥ 1 32 v 2 1 ∗ N 2 D ln 2 ( N D ) (1 -2 ϵ -δ ) , we have

We will derive contradictions based on this L .

Part 1: Find some points and values.

We can find k, n, c, δ, ϵ satisfying

(1): we have n, k ∈ Z + and 12 v 1 ≤ n ≤ √ k . Let c = 1 , and we can find k points in [0 , 1] n and the distance between any pair of these points is greater than c ;

By lemma F.1, to make (1) valid, we just need n 2 < k ≤ 2 n , and (2) is easy to satisfy.

Part 2: Construct some distribution

Let { u i } k i =1 satisfy u i ∈ [0 , 1] n and || u i -u j || 2 ≥ c . By (1) mentioned in (1) in Part 1, such { u i } k i =1 must exist. Now, we consider the following types of distribution D :

It is obvious that, by || u i -u j || 2 ≥ c , such a distribution exists. Let S be the set that contains all such distributions. We will show that for D ∈ S , it holds N D ≤ v 1 √ k ln( kn/c ) .

By Theorem 4.1 and definition of v 1 , we know that for any distribution D ∈ S , let y i be the label of u i in distribution D ∈ S . Then there is a memorization F of { ( u i , y i ) } k i =1 with at most v 1 √ k ln( kn/c ) parameters. Then by (c1), the above result implies A D ( F ( x )) = 1 , so we know that N D ≤ v 1 √ k ln( kn/c ) for any D ∈ S . Moreover, by k ≥ n ≥ 3 , c = 1 and it is easy to see that N D ≥ n . We thus have 3 ≤ N D ≤ 4 v 1 √ k ln( √ k ) .

Part 3: A definition.

Moreover, for D ∈ S , we define S ( D ) as the following set:

Z ∈ S ( D ) if and only if Z ∈ [ k ] q is a vector satisfying: Define D ( Z ) as D ( Z ) = { ( u z i , y z i ) } q i =1 , then A D ( L ( D ( Z ))) ≥ 1 -ϵ , where z i is the i -th weight of Z and y z i is the label of u z i in distribution D .

It is easy to see that, if we i.i.d select q samples in distribution D to form a dataset D tr , then

(1): By c 2 , with probability 1, D tr only contains the samples ( u j , y j ) where j ∈ [ k ] ;

(2): Let D tr has the form shown in (1). Then every time a sample is selected, it is in { ( u i , y i ) } k i =1 . Now we construct a vector in [ k ] q as follows: the index of i -th selected samples as the i -th component of the vector. Then each selection situation corresponds to a vector in [ k ] q which is constructed as before. Then by the definition of S ( D ) , we have A D ( L ( D tr )) ≥ 1 -ϵ if and only if the corresponding vector of D tr is in S ( D ) .

By the above result and the by the assumption of L at the beginning of the proof, so that for any D ∈ S we have t

Part 4: Prove the Theorem.

Let S s be a subset of S , and S s = {D i 1 ,i 2 ,...,i k } i j ∈{-1 , 1 } ,j ∈ [ k ] ⊂ S , where distribution D i 1 ,i 2 ,...,i k satisfies the label of u j is i j , where j ∈ [ k ] .

We will show that there exists at least one D ⊂ S s , such that | S ( D ) | < (1 -δ ) k q , which is contrary to equation 4. To prove that, we just need to prove that ∑ D∈ S s | S ( D ) | < (1 -δ )2 k k q , use | S s | = 2 k here.

To prove that, for any vector Z ∈ [ k ] q , we estimate how many D ∈ S s which makes Z to be included in S ( D ) .

Part 4.1, situation of a given vector Z and a given distribution D .

For a Z = ( z i ) q i =1 and D such that Z ∈ S ( D ) , let len( Z ) = { c ∈ [ k ] : ∃ i, c = z i } . We consider the distributions in S s that satisfy the following condition: for i ∈ len( Z ) , the label of u i is equal to the label of u i in D .

Obviously, we have 2 k -| len( Z ) | distributions that can satisfy the above condition in S s . Let such distributions make up a set S ss ( D , Z ) . Now, we estimate how many distributions D s in S ss ( D , Z ) satisfy Z ∈ S ( D s ) .

For any distribution G ∈ S s , let y ( G ) i be the label of u i in distribution G , and define the dataset D tr = { ( u z i , y ( D ) z i ) } q i =1 . Then Z ∈ S ( D s ) if and only if: for at least k -[ kϵ ] of i ∈ [ k ] , L ( D tr ) gives the label y ( D s ) i to u i .

Firstly, consider that when i ∈ len( Z ) . For any D s ∈ S ss ( D , Z ) , we have y ( D s ) i = y ( D ) i and L ( D tr ) must give the label y ( D ) i to u i , so when i ∈ len( Z ) , L ( D tr ) gives the label y ( D s ) i to u i .

Then, consider i / ∈ len( Z ) . Because Z is a given vector, so if Z ∈ S ( D s ) , the label y ( D s ) i where i / ∈ len( Z ) are at most [ kϵ ] different from the label of u i given by L ( D tr ) .

So, by the above two results, this kind of D s is at most ∑ [ kϵ ] i =0 ( k -| len( Z ) | i ) . So, we have ∑ [ kϵ ] i =0 ( k -| len( Z ) | i ) number of distributions D s in S ss ( D , Z ) satisfy Z ∈ S ( D s ) .

Part 4.2, for any vector Z and distribution D .

Firstly, for a given Z , we have at most 2 | len ( Z ) | different S ss ( D , Z ) for D ∈ D S .

Because when D 1 and D 2 satisfy y ( D 1 ) i = y ( D 2 ) i for any i ∈ len( Z ) , we have D ss ( D 1 , Z ) = D ss ( D 2 , Z ) , and 2 | len( Z ) | situations of label of u i where i ∈ len( Z ) , so there exist at most 2 | len( Z ) | different S ss ( D , Z ) .

By part 4.1, for a S ss ( D , Z ) , at most ∑ [ kϵ ] i =0 ( k -| len( Z ) | i ) of D s ∈ S ss ( D , Z ) satsify Z ∈ S ( D s ) . So by the above result and consider that D s = ∪ D∈D s S ss ( D , Z ) , at most 2 | len( Z ) | ∑ [ kϵ ] i =0 ( k -| len( Z ) | i ) number of D s ∈ S s such that Z ∈ S ( D s ) .

And there exist k q different Z , so ∑ D∈ S s | S ( D ) | ≤ ∑ Z 2 | len( Z ) | ∑ [ kϵ ] i =0 ( k -| len( Z ) | i ) ≤ ∑ Z 2 k (1 -δ ) = k q 2 k (1 -δ ) . For the last inequality, we use 2 | len( Z ) | ∑ [ kϵ ] i =0 ( k -| len( Z ) | i ) < 2 k (1 -δ ) , which can be shown by | len( Z ) | ≤ q and lemma F.2.

This is what we want. we proved the theorem.

We now prove Corollary 6.4.

Proof. Using lemma F.1, we can find 2 [ n ⌈ c 2 ⌉ ] points in [0 , 1] n and the distance between any two points shall not be less than c . So we take a ϵ, δ such that 1 -2 ϵ -δ > 0 , n = 3[12 v 1 / (1 -2 ϵ -δ )] + 3 , c = 1 and k = 2 [ n ⌈ c 2 ⌉ ] in the (1) in the part 1 of the proof of Theorem 6.1, then similar as the proof of Theorem 6.1, and we get this corollary.

G Proof of Theorem 6.5

G.1 The Existence

Firstly, it is easy to show that there exists a memorization algorithm which satisfies L ( D tr ) ≤ N D when D tr ∼ D N with probability 1. We just consider the following memorization algorithm:

For a given dataset D , let L ( D ) be the memorization of D with minimum parameters, as shown in Theorem 4.1. Then para ( L ( D )) ≤ O ( √ | D | ) .

And if D is i.i.d selected from distribution D , where D ∈ D ( n, c ) , then by the definition of L and N D in Theorem 4.3, we have para ( L ( D )) ≤ N D with probability 1. So L is what we want.

G.2 The Sample Complexity of Generalization

To prove (1) in the theorem, we need three lemmas.

Lemma G.1 ([44]) . Let H be a hypothesis space with VCdim h and D is distribution of data, if N ≥ h , then with probability 1 -δ of D tr ∼ D N , we have

for any F ∈ H . Here, E D ( F ) = E ( x,y ) ∼D [ I ( F ( x ) = y )] , E D tr ( F ) = ∑ ( x,y ∈D tr ) [ I ( F ( x ) = y )] and I ( x ) = 1 if x is true or I ( x ) = 0 .

Moreover, when h ≥ 1 , we have

Lemma G.2. If e ≤ ba/c , then we have a ln( bu ) ≤ cu when u ≥ 2 a ln( ba/c ) /c .

Proof. Firstly, we have a ln( bu ) cu = ln( ba/c ( cu/a )) cu/a , and we just need to show ln( ba/c ( cu/a )) cu/a ≤ 1 .

Then, we show that there are 2 ln( ba/c ) ≤ ba/c . Just consider the function g ( x ) = x -2 ln x , by g ′ ( x ) = 1 -2 /x , so g ′ ( x ) ≥ 0 when x ≥ 2 , so g ( ba/c ) ≥ g ( e ) = e -2 > 0 , this is what we want.

Now we consider the function f ( x ) = ln(( ba/c ) x ) /x , by the above result, we have that 1 ≤ 2 ln( ba/c ) ≤ ba/c , we have that

And consider that f ′ ( x ) = 1 -ln(( ba/c ) x ) x 2 ≤ 0 when x ≥ 1 , so, when x ≥ 2 ln( ba/c ) , we have f ( x ) ≤ f (2 ln( ba/c )) ≤ 1 , which means that when cu/a ≥ 2 ln( ba/c ) , it holds ln( ba/c ( cu/a )) cu/a ≤ 1 . The lemma is proved.

Lemma G.3 ([6]) . Let H m be the hypothesis space composed of the networks with at most m parameters. Then the VCdim of H m is not more than qm 2 ln( m ) , where q is a constant not dependent on m .

Then we can prove (1) in the theorem.

Proof. Let D tr ∼ D N . Because the algorithm satisfies the condition in theorem, then L ( D tr ) ∈ H N D , where H N D is defined in lemma G.3. By lemma G.3, the VCdim of H D tr is not more than qN 2 D ln( N D ) for some q ≥ 1 . Using lemma G.1 to this fact, we have N ≥ 64 qeN 2 ln( N )

16 qN 2 D ln( N D ) ln( D D δϵ 2 ) ϵ 2 . Take these values in lemma G.1, and considering that the memorization algorithm L must satisfy that E D tr ( L ( D tr )) = 1 , using lemma G.2 (just take a = 8 qN 2 D ln( N D ) , b = 8 e/δ and c = ϵ 2 in lemma G.2), we have

which implies 1 -ϵ ≤ E D ( L ( D tr )) . The theorem is proved.

G.3 More Lemmas

We need three more lemmas to prove Theorem 6.5.

Lemma G.4. Let D ⊂ [0 , 1] n ×{-1 , 1 } . Then D has a memorization with width 1 if and only if D is linearly separable.

Proof. If D is linearly separable, then it obviously has a memorization with width 1.

If D has a memorization with width 1, we show that D is linearly separable. Let F be the memorization network of D with width 1, and F 1 the first layer of F .

Part 1: We show that it is impossible to find any ( x 1 , 1) , ( x 2 , -1) , ( x 3 , 1) ∈ D such that F 1 ( x 1 ) < F 1 ( x 2 ) < F 1 ( x 3 ) . If we can, then contradiction will be obtained.

Assume ( x 1 , 1) , ( x 2 , -1) , ( x 3 , 1) ∈ D such that F 1 ( x 1 ) < F 1 ( x 2 ) < F 1 ( x 3 ) .

It is easy to see that, for any linear function wx + b and u ≤ v ≤ k , we have wu + b ≤ wv + b ≤ wk + b or wu + b ≥ wv + b ≥ wk + b , which implies Relu( wu + b ) ≤ Relu( wv + b ) ≤ Relu( wk + b ) or Relu( wu + b ) ≥ Relu( wv + b ) ≥ Relu( wk + b ) .

Because ( x 1 , 1) , ( x 2 , -1) , ( x 3 , 1) ∈ D satisfy that F 1 ( x 1 ) < F 1 ( x 2 ) < F 1 ( x 3 ) , and each layer of F is a linear function composite Relu , so after each layer, the order of F 1 ( x 1 ) , F 1 ( x 2 ) , F 1 ( x 3 ) is not changed or contrary. So there must be F ( x 1 ) ≤ F ( x 2 ) ≤ F ( x 3 ) or F ( x 1 ) ≥ F ( x 2 ) ≥ F ( x 3 ) . Then F cannot classify x 1 , x 2 , x 3 correctly, which contradicts to the fact that F is a memorization of D .

Part 2: Weshow that, it is impossible to find any ( x 1 , -1) , ( x 2 , 1) , ( x 3 , -1) ∈ D such that F 1 ( x 1 ) < F 1 ( x 2 ) < F 1 ( x 3 ) .

This is similar to Part 1.

By parts 1 and 2, without losing generalization, we know that for any ( x 1 , 1) , ( x 2 , -1) ∈ D , it holds F 1 ( x 1 ) > F 1 ( x 2 ) . Since F 1 is a linear function composite Relu , D is linear separable.

Lemma G.5. Let D = { ( x i , y i ) } ⊂ [0 , 1] ×{-1 , 1 } . Then D has a memorization with width 2 and depth 2 if and only if at least one of the following conditions is valid.

(c1): There is a closed interval I such that: if ( x, 1) ∈ D then x ∈ I and if ( x, -1) ∈ D then x / ∈ I .

(c2): There is a closed interval I such that: if ( x, 1) ∈ D then x / ∈ I and if ( x, -1) ∈ D then x ∈ I .

Proof. Part 1: We show that if condition (c1) is valid, then D has a memorization with width 2 and depth 2. It is similar for (c2) to be valid.

Let I = [ a, b ] . If for all ( x, -1) ∈ D , we have x < a , then D is linear separable, and the result is valid. If for all ( x, -1) ∈ D , we have x > b , then D is linear separable, and the result is valid. Now we consider the situation where x > a for some ( x, -1) ∈ D and x < b for some ( x, -1) ∈ D .

Let x -1 = max ( x, -1) ∈ D { x < a } . Then for F 1 ( x ) = x -( x -1 + a ) / 2 , it is easy to verify that F 1 ( x ) > a -x -1 2 for all x ≥ a and F 1 ( x ) < 0 for all ( x 0 , -1) ∈ D such that x 0 < a .

Let x 1 = min ( x, -1) ∈ D { x > b } . Then for F 2 ( x ) = x -( x 1 + b ) / 2 , it is easy to verify that F 2 ( x ) < 0 for all x ≤ b and F 2 ( x ) > ( x 1 -b ) / 2 for all ( x 0 , -1) ∈ D such that x 0 > b .

Let the network F be defined by F = Relu( F 1 ( x )) -T Relu( F 2 ( x )) -t , where T = 8 x 1 -b is a positive real number, and t = a -x -1 4 > 0 .

Now we prove F is what we want. It is easy to see that, F is a depth 2 width 2 network. When x ∈ [ a, b ] , then F 1 ( x ) > a -x -1 2 and F 2 ( x ) ≤ 0 , so F ( x ) > 0 . For ( x, -1) ∈ D such that x < a , we have F 1 ( x ) < 0 and F 2 ( x ) < 0 , so F ( x ) < 0 ; for ( x, -1) ∈ D such that x > b , we have F 1 ( x ) < 1 and F 2 ( x ) > x 1 -b 4 , so F ( x ) ≤ 1 -2 -a -x -1 4 < 0 , this is what we want.

Part 2: If D has a memorization with width 2 and depth 2, then we show that D satisfies conditions (c1) or (c2).

If D is linear separable, (c1) and (c2) are valid. If not, without losing generality, assume that ( x 1 , 1) , ( x 2 , -1) , ( x 3 , 1) ∈ D such that x 1 < x 2 < x 3 (for the situation that ( x 1 , -1) , ( x 2 , 1) , ( x 3 , -1) ∈ D such that x 1 < x 2 < x 3 , the proof is similar). Then we show that if ( x, -1) ∈ D , we have x 1 < x < x 3 . Assume ( x 0 , -1) ∈ D such that x 0 < x 1 , then we have that x 0 < x 1 < x 2 < x 3 , then we can deduce the contradiction.

Let F = a Relu( F 1 ( x )) + b Relu( F 2 ( x )) + c be the memorization network of D , where F i ( x ) is a linear function. Let u, v ∈ R such that F 1 ( u ) = F 2 ( v ) = 0 , without loss generality, let u ≤ v .

Then we know that F is linear in such three regions: ( -∞ , u ] , [ u, v ] and [ v, ∞ ) . We call the three regions as linear regions of F . We prove the following three results at first.

(1): The slope of F on ( -∞ , u ] is positive.

Firstly, we show that x 0 ∈ ( -∞ , u ] . If not, since ( x 0 , -1) , ( x 1 , 1) , ( x 2 , -1) are not linear separable, and ( x 1 , 1) , ( x 2 , -1) , ( x 3 , 1) are not linear separable, we have ( x 0 , -1) , ( x 1 , 1) ∈ [ u, v ] and ( x 2 , -1) , ( x 3 , 1) ∈ [ v, ∞ ) . Then, because x 1 > x 0 and F ( x 1 ) > F ( x 0 ) , and F is linear in [ u, v ] , we have that F ( v ) ≥ F ( x 1 ) > 0 . Now we consider the points ( v, 1) , ( x 2 , -1) , ( x 3 , 1) . It is easy to see that F memorizes such three points and they are in the linear region of F , so ( v, 1) , ( x 2 , -1) , ( x 3 , 1) is linear separable, which is impossible because v ≤ x 2 ≤ x 3 and resulting in contradiction, so x 0 ∈ ( -∞ , u ] .

If the slope of F on ( -∞ , u ] is not positive, since u ≥ x 0 , we have F ( u ) < 0 . Now we consider the points ( u, -1) , ( x 1 , 1) , ( x 2 , -1) , ( x 3 , 1) . Just similar to above to get the contradiction. So the slope of F on ( -∞ , u ] is positive.

(2): The slope of F on [ v, ∞ ) is positive. Similar to (1).

(3): The slope of F on ( -∞ , u ] is negative. If not, F must be a non-decreasing function, which is impossible.

Using (1),(2),(3), we can get a contradiction, which means that there is a ( x 0 , -1) ∈ D such that x 0 < x 1 is not possible.

Consider that, in a linear region of F , if the activation states of F 1 and F 2 are both not activated, then on such linear regions, the slope of F is 0. But due to (1),(2),(3), all linear regions have non-zeros slope of F , so on each linear regions, at least one of F 1 and F 2 is activated. So, the activation states of F 1 and F 2 at ( -∞ , u ] , [ u, v ] and [ v, ∞ ) is ( -, +) , (+ , +) , (+ , -) (+ means activated, - means not activated).

Then the slope of F on [ u, v ] is equal to the sum of the slopes of F on ( -∞ , u ] and the slope of F on [ v, ∞ ) . But by (1),(2),(3), that means a negative number is equal to the sum of two positive numbers, which is impossible. So we get a contradiction.

So if ( x 0 , -1) ∈ D , we have x 0 > x 1 . Similar to before, we have x 0 < x 3 . So we get the result.

By the above result, all the samples ( x 0 , -1) ∈ D satisfies x ∈ ( x 1 , x 3 ) , so there is a close interval in ( x 1 , x 3 ) such that: if ( x 0 , -1) ∈ D , then x 0 is in such interval, then (c2) is vald, and we prove the lemma.

G.4 The algorithm is no-efficient.

Now we prove (2) of theorem 6.5, that is, all such algorithm is not efficient if P = NP . We need the reversible 6-SAT problem defined in definition 6.6.

̸

Proof. Wewill show that, if there is an efficient memorization algorithm which satisfies the conditions of the theorem (has at most N D parameters with probability 1), then we can solve the reversible 6-SAT in polynomial time, which implies P = NP .

Firstly, for the 6-SAT problem, we write it as the following form.

Let φ = ∧ i m =1 φ i ( n, m ) be a 6-SAT for n variables, where φ i ( n, m ) = ∨ 6 j =1 ˜ x i,j and ˜ x i,j is either x s or ¬ x s for some s ∈ [ n ] (see Definition 6.6). Then, we define some vectors in R n based on φ i ( n, m ) .

For i ∈ [ m ] , define Q φ i ∈ R n as follows: Q φ i [ j ] = 1 if x j occurs in φ i ( n, m ) ; Q φ i [ j ] = -1 if ¬ x j occurs in φ i ( n, m ) ; Q φ i [ j ] = 0 otherwise. Q φ i [ j ] is the j -th entry in Q φ i , then six entries in Q φ i are 1 or -1 and all other entries are zero.

Now, we define a binary classification dataset D ( φ ) = { ( x i , y i ) } m +4 n i =1 ⊂ [0 , 1] n × [2] as follows.

For i ∈ [ n ] , x i = 1 i / 3 + 1 . 1 1 / 3 , y i = 1 .

For i ∈ { n +1 , n +2 , . . . , 2 n } , x i = 1 . 1 1 i -n / 3 + 1 . 1 1 / 3 , y i = -1 .

For i ∈ { 4 n +1 , 4 n +2 , . . . , 4 n + m } , x i = 1 / 12 Q φ i -4 n +1 . 1 1 / 3 , y i = 1 .

Here, 1 i is the vector whose i -th weight is 1 and other weights are 0, 1 is the vector whose weights are all 1.

̸

Let L be an efficient memorization algorithm which satisfies the condition in the theorem. Then we prove the following result: If n ≥ 4 and φ is a reversible 6-SAT problem, then para ( L ( D ( φ ))) = n +8 if and only if φ has a solution, which means P = NP and leads to that L does not exist when P = NP . The proof is divided into two parts.

Part 1: If φ is a reversible 6-SAT problem that has a solution, then para ( L ( D ( φ ))) = n +8 .

To prove this part, we only need to prove that para ( L ( D ( φ ))) ≥ n +8 and para ( L ( D ( φ ))) ≤ n +8 .

Firstly, we show that { ( x 1 , 1) , ( x n +1 , -1) , ( x 2 n +1 , 1) , ( x 3 n +1 , -1) } ⊂ D ( φ ) are not linearly separable. This is because { x 1 , x n +1 , x 2 n +1 , x 3 n +1 } is a linear transformation of { 1 1 , 1 . 1 1 1 , -1 1 , -1 . 1 1 1 } , so { ( x 1 , 1) , ( x n +1 , -1) , ( x 2 n +1 , 1) , ( x 3 n +1 , -1) } ⊂ D ( φ ) are not linearly separable if and only if { ( 1 1 , 1) , (1 . 1 1 1 , -1) , ( -1 1 , 1) , ( -1 . 1 1 1 , -1) } are not linearly separable, by the definition of 1 1 , easy to see that { ( 1 1 , 1) , (1 . 1 1 1 , -1) , ( -1 1 , 1) , ( -1 . 1 1 1 , -1) } are not linearly separable, so we get the result.

By the above result, a subset of D ( φ ) is not linearly separable, so we have that D ( φ ) is not linearly separable. So, by lemma G.4, L ( D ( φ )) must have width more than 1. For a network with width at least 2 , when it has depth 2, it has at least 2 n +5 parameters; when it has depth 3, it has at least n +8 parameters; when it has depth more than 3, it has at least n +10 parameters. So when n ≥ 4 , we have para ( L ( D ( φ ))) ≥ n +8 .

Part 1.2: If φ is a reversible 6-SAT problem that has a solution, then para ( L ( D ( φ ))) ≤ n +8 .

We define a distribution D at first. D is defined on D ( φ ) , and each point has the same probability. It is easy to see that, D ∈ D ( n, 1 / 30) .

Since when N ≥ m +4 n , we have P D tr ∼D N ( D tr = D ( φ )) > 0 , so by the definition of N D and the fact L satisfies the conditions in the theorem, we have para ( L ( D ( φ ))) ≤ N D . Moreover, because D is defined on D ( φ ) , we will construct a network with n +8 parameters to memorize D ( φ ) to show that N D ≤ n +8 , which implies para ( L ( D ( φ ))) ≤ N D ≤ n +8 because L satisfies the condition in the theorem.

This network has three layers, the first layer has width 1; the second layer has width 2; the third output layer has width 1.

Let s = ( s 1 , s 2 , . . . , s n ) ∈ {-1 , 1 } n be a solution of φ . Then the first layer is F 1 ( x ) = Relu(3 s ( x -1 . 1 1 / 3) + 3) . Then we have the following results:

(2): 2 ≤ |F 1 ( x ) | ≤ 4 for all ( x, 1) ∈ D ( φ ) .

(1) is very easy to validate. We just prove (2).

For i ∈ [ n ] and i ∈ { 2 n +1 , . . . , 3 n } , because s ∈ {-1 , 1 } n , so 3 s ( x -1 . 1 1 / 3) = 1 or 3 s ( x -1 . 1 1 / 3) = -1 , which implies 2 ≤ |F 1 ( x i ) | ≤ 4 .

For i ∈ { 4 n +1 , . . . , 4 n + m } , x i -1 . 1 1 / 3 has only six components that are not 0. Because s is the solution of φ , which indicates that at least one of the six non-zero components of x i -1 . 1 1 / 3 has the same positive or negative shape as the corresponding component of s . Consider that such six non-zero components of x i -1 . 1 1 / 3 are in {-1 / 12 , 1 / 12 } , so 3 s ( x i -1 . 1 1 / 3) ≥ 1 / 4 -5 / 4 = -1 .

Moreover, because φ is a reversible problem, so φ i ( n, m ) and φ i ( n, m ) are both in the φ , which indicates that the positive and negative forms of the six non-zero components of x i -1 . 1 1 / 3 cannot be exactly the same as the positive and negative forms of the corresponding components in s , or there must be φ i ( n, m ) = 0 , which contradicts to s is the solution of φ . So, we have 3 s ( x i -1 . 1 1 / 3) ≤ 5 / 4 -1 / 4 = 1 .

Then we have that, for i ∈ { 4 n +1 , . . . , 4 n + m } , it holds 3 s ( x i -1 . 1 1 / 3) ∈ [ -1 , 1] , resulting in 2 ≤ |F 1 ( x i ) | ≤ 4 . We proved (2).

By (1) and (2), and using lemma G.5, there is a network F 2 : R → R with width 2, depth 2 and 7 parameters that can classify the { ( F 1 ( x i ) , y i ) } 4 n + m i =1 , so F 2 ◦ F 1 is the network we want.

By such a network, we have that N D ≤ n +8 , and then, we have para ( L ( D ( φ ))) ≤ N D ≤ n +8 . We proved the result.

Part Two: If φ is a reversible 6-SAT problem and para ( L ( D ( φ ))) = n +8 , then φ has a solution.

If L ( D ( φ )) has width 2 of the first layer, then para ( L ( D ( φ ))) ≥ 2 n + 5 > n + 8 , so when para ( L ( D ( φ ))) = n +8 , the first layer has width 1.

Write L ( D ( φ )) = F 2 ( F 1 ( x )) , and write F 1 as F 1 ( x ) = Relu(3 s ( x -1 . 1 1 / 3) + b ) , and let s = ( s 1 , s 2 , . . . , s n ) .

We will prove that Sgn ( s ) = ( Sgn ( s 1 ) , Sgn ( s 2 ) , Sgn ( s 3 ) , . . . , Sgn ( s n )) is a solution of φ . The proof is given in two parts.

̸

Part 2.1 we have 1 . 1 | s i | ≥ | s j | for any i, j ∈ [ n ] . Firstly, we have s i = 0 for any i ∈ [ n ] . Because if s i = 0 , it holds F 1 ( x i ) = F 1 ( x n + i ) , which implies that L ( D ( φ )) gives the same label to x i and x n + i , but x i and x n + i have the different labels in dataset D ( φ ) , so it contradicts L ( D ( φ )) is the memorization of D ( φ ) .

Without losing generality, let | s 1 | ≥ | s 2 | ≥ · · · ≥ | s n | . Then we just need to prove that 1 . 1 | s n | ≥ | s 1 | .

Because D ( φ ) is not linear separable, so by lemma G.4, L ( D ( φ )) has width more than 1. Because F 1 has width 1, so F 2 has width 2 and 7 parameters, resulting in that F 2 is a network with width 2 and depth 2. And F 2 can classify such six points: { ( F 1 ( x i ) , y i ) } i ∈{ 1 ,n +1 , 2 n +1 , 3 n +1 , 2 n, 4 n } .

If s 1 > 0 , taking the values of x 1 , x n +1 , x 2 n +1 , x 3 n +1 in F 1 , we have 1 . 1 s 1 + b = F 1 ( x n +1 ) ≥ s 1 + b = F 1 ( x 1 ) ≥ -s 1 + b = F 1 ( x 2 n +1 ) ≥ -1 . 1 s 1 + b = F 1 ( x 3 n +1 ) , which implies F 1 ( x n +1 ) ≥

F 1 ( x 1 ) ≥ F 1 ( x 2 n +1 ) ≥ F 1 ( x 3 n +1 ) ; if s 1 < 0 . Similarly as before, we have F 1 ( x n +1 ) ≤ F 1 ( x 1 ) ≤ F 1 ( x 2 n +1 ) ≤ F 1 ( x 3 n +1 ) . So, F 1 ( x 1 ) and F 1 ( x 2 n +1 ) are always in the interval from F 1 ( x n +1 ) to F 1 ( x 3 n +1 ) .

Consider that x n +1 and x 3 n +1 have label -1 , x 1 and x 2 n +1 have label 1 , so by Lemma G.5, if { ( F 1 ( x i ) , y i ) } i ∈{ 1 ,n +1 , 2 n +1 , 3 n +1 , 2 n, 4 n } can be memorized by a depth 2 width 2 network, then F 1 ( x 2 n ) and F 1 ( x 4 n ) must be not in the interval from F 1 ( x 1 ) to F 1 ( x 2 n +1 ) , or we cannot find a interval satisfies the conditions of lemma G.5.

Since max {F 1 ( x 2 n ) , F 1 ( x 4 n ) } = 1 . 1 | s n | + b , to ensure that F 1 ( x 2 n ) and F 1 ( x 4 n ) are not in the interval from F 1 ( x 1 ) to F 1 ( x 2 n +1 ) , we have max {F 1 ( x 2 n ) , F 1 ( x 4 n ) } = 1 . 1 | s n | + b ≥ max {F 1 ( x 1 ) , F 1 ( x 2 n +1 ) } = | s 1 | + b or max {F 1 ( x 2 n ) , F 1 ( x 4 n ) } = 1 . 1 | s n | + b ≤ min {F 1 ( x 1 ) , F 1 ( x 2 n +1 ) } = -| s 1 | + b . The second case is impossible, so we have 1 . 1 | s n | ≥ | s 1 | . This is what we want in this part.

Part 2.2 We show that Sgn ( s ) is the solution of φ . Assume that Sgn ( s ) is not the solution of φ . There is a i ∈ { 4 n + 1 , . . . , 4 n + m } , such that the positive and negative forms of the six non-zero components of x i are exactly the same as the positive and negative forms of the corresponding components in s . Then sx i + b ≥ 6 / 4 | s n | + b ≥ 6 / 4 . 4 | s 1 | + b ≥ 1 . 1 | s 1 | + b . So, by max {F 1 ( x 1+ n ) , F 1 ( x 3 n +1 ) } = 1 . 1 | s 1 | + b and min {F 1 ( x 1+ n ) , F 1 ( x 3 n +1 ) } = -1 . 1 | s 1 | + b , we know that F 1 ( x i ) is not in the interval from F 1 ( x 1+ n ) to F 1 ( x 3 n +1 ) .

Then similar to part 2.1, consider the point { ( F 1 ( x i ) , y i ) } i ∈{ 1 ,n +1 , 2 n +1 , 3 n +1 ,i } , we have that F 1 ( x 1 ) and F 1 ( x 2 n +1 ) are always in the interval from F 1 ( x n +1 ) to F 1 ( x 3 n +1 ) , but F 1 ( x i ) is not in the interval from F 1 ( x 1+ n ) to F 1 ( x 3 n +1 ) . By lemma G.5 and the fact that the label of F 1 ( x n +1 ) and F 1 ( x 3 n +1 ) is different from that of other three samples, we cannot find an interval satisfying the condition in lemma G.5, so F 2 ( x ) cannot classify such five points: { ( F 1 ( x i ) , y i ) } i =1 ,n +1 , 2 n +1 , 3 n +1 ,i . This is contradictory, as L ( D ( φ )) is the memorization of D ( φ ) . So, the assumption is wrong, we prove the theorem.

H Proof of Theorem 7.3

H.1 Proof of Proposition 7.7

Proof. It suffices to prove that we can find an S c ( D ) ⊂ { ( x, y ) ∥ ( x, y ) ∼ D} such that for any ( x, y ) ∼ D , we have x ∈ ∪ ( z,w ) ∈ S c ( D ) B (( z, w )) .

Let S c = { ( i 1 c/ (6 . 2 n ) , i 2 c/ (6 . 2 n ) , . . . , i n c/ (6 . 2 n )) ∥ i j ∈ { 0 , 1 , . . . , [6 . 2 n/c ] + 1 }} , and define S c ( D ) as: for any ( i 1 c/ (6 . 2 n ) , i 2 c/ (6 . 2 n ) , . . . , i n c/ (6 . 2 n )) ∈ S c , randomly take a ( x, y ) ∼ D satisfying || x -( i 1 c/ (6 . 2 n ) , i 2 c/ (6 . 2 n ) , . . . , i n c/ (6 . 2 n )) || ∞ ≤ c/ (6 . 2 n ) (if we have such a x ), and put ( x, y ) into S c ( D ) .

Then, we have that, for any ( x, y ) ∼ D , there is a point z ∈ S c such that || z -x || ∞ ≤ c/ (6 . 2 n ) , and there is a ( x z , y z ) ∈ S c ( D ) such that || z -x z || ∞ ≤ c/ (6 . 2 n ) , so || x z -x || ∞ ≤ c/ (3 . 1 n ) , which implies || x -x z || ∞ ≤ c/ 3 . 1 .

Since the radius of B (( z, w )) is more than c/ 3 . 1 , for any ( x, y ) ∼ D , we have x ∈ ∪ ( z,w ) ∈ S c ( D ) B (( z, w )) , we prove the lemma.

H.2 Main idea

For a given dataset D tr ⊂ [0 , 1] n ×{-1 , 1 } , we use the following two steps to construct a memorization network:

(c1): Find suitable convex sets { C i } in [0 , 1] n , ensuring that: each sample in D tr is in at least one of these convex sets. Furthermore, if x, z ∈ C i and ( x, y x ) , ( z, y z ) ∈ D tr , then y x = y z , and define y ( C i ) = y x .

(c2): Construct a network F satisfying that for any x ∈ C i , Sgn ( F ( x )) = y ( C i ) . Such a network must be a memorization of D tr , because each sample in D tr is in at least one of { C i } , so if x ∈ C i and ( x, y x ) ∈ D tr , then Sgn ( F ( x )) = y ( C i ) = y x , which is the network we want.

H.3 Finding convex sets

For a given dataset D tr ⊂ [0 , 1] n ×{-1 , 1 } , let D tr = { ( x i , y i ) } N i =1 , and for i ∈ [ n ] , the convex sets C i are constructed as follows:

̸

Now, we have the following lemma, which implies that C i satisfies conditions (c1) mentioned in above.

Lemma H.1. If C i are constructed as above, then

(3): C i is a convex set.

Proof. Firstly, we show that x i ∈ C i . For any i, j ∈ [ N ] , taking x i into S i,j ( x ) , we have S i,j ( x i ) = 0 . 49 || x i -x j || 2 2 > 0 , so x i ∈ { x ∈ [0 , 1] n ∥ S i,j ( x ) ≥ 0 } . Thus x i ∈ ∩ j ∈ [ N ] ,y i = y j { x ∈ [0 , 1] n ∥ S i,j ( x ) ≥ 0 } = C i .

̸

̸

Then, we show that: if y j = y i , then x j / ∈ C i , which implies (2) of lemma is valid.

̸

Finally, we show C i is a convex set. Because for any i, j ∈ [ N ] , { x ∈ [0 , 1] n ∥ S i,j ( x ) ≥ 0 } is a convex set, and the combination of convex sets is also convex set, so C i is a convex set.

H.4 Construct the Network

We show how to construct a network F , such that Sgn ( F ( x )) = y ( C i ) for any x ∈ C i , where C i is defined in section H.3.

For a given dataset D tr = { ( x i , y i ) } N i =1 , we construct a network F mem which has three layers as following.

̸

(1): Let r = 0 . 01 ∗ min i,j ∈ [ N ] ,y i = y j || x i -x j || 2 2 . For any i, j ∈ [ N ] , S i,j defined in section H.3, let u i ( x ) = ∑ j ∈ [ N ] ,y j = y i Relu( -S i,j ( x )) -r . It is easy to see that u i is a depth 2 network.

̸

(2): The first two layers are F 1 : R n → R N . Let F 1 ( x )[ i ] be the i -th output of F 1 ( x ) , then let F 1 ( x )[ i ] equal to Relu( -u i ( x )) . It is easy to see that, F 1 ( x ) requires O ( N 2 n ) parameters.

Now, we prove that Sgn ( F mem ( x )) = y ( C i ) for any x ∈ C i . We need the following lemma.

̸

Lemma H.2. For any x ∈ C i , we have u i ( x ) < 0 and u j ( x ) > 0 when y i = y j .

Proof. Assume that x ∈ C i . We prove the following two properties, and hence the lemma.

̸

By the definition of C i , we have S i,j ( x ) ≥ 0 for all j ∈ [ N ] staisfying y i = y j , so u i ( x ) = ∑ j ∈ [ N ] ,y j = y i Relu( -S i,j ( x )) -r = ∑ j ∈ [ N ] ,y j = y i 0 -r = -r < 0 .

̸

̸

For any j such that y i = y j , we show S j,i ( x ) ≤ -0 . 02 || x i -x j || 2 2 at first. Because x ∈ C i , so S i,j ( x ) ≥ 0 , that is ( x i -x j )( x -(0 . 51 ∗ x i +0 . 49 ∗ x j )) ≥ 0 , so

̸

̸

Thus S j,i ( x ) ≤ -0 . 02 || x i -x j || 2 2 . Then, by the above result, taking the value of r in it, we have u j ( x ) ≥ Relu( -S i,j ( x )) -r ≥ 0 . 02 || x i -x j || 2 2 -r > 0 .

By the above lemma, we can prove the result.

Lemma H.3. we have Sgn ( F mem ( x )) = y i for any x ∈ C i .

̸

Proof. Let x ∈ C i . By lemma H.2, we have F 1 ( x )[ i ] > 0 , and F 1 ( x )[ j ] = 0 when j satisfies y j = y i , so F ( x ) = ∑ j ∈ [ N ] y j F 1 ( x )[ j ] = y i ∑ j ∈ [ N ] ,y j = y i F 1 ( x )[ j ] , by F 1 ( x )[ i ] > 0 , and we thus have Sgn ( F ( x )) = y i .

H.5 Effective and Generalization Guarantee

In this section, we prove that the above algorithm is an effective memorization algorithm with guaranteed generalization. We give a lemma.

Lemma H.4. For any a, b, c ∈ R n such that || b -a || 2 ≥ 3 . 1 || a -c || 2 , let V be the plane ( b -c )( x -(0 . 51 c +0 . 49 b )) . Then the distance of a to the plane V is greater than || b -a || / 3 . 1 .

Proof. Let || a -b || 2 = L ab , || a -c || 2 = L ac , || c -b || 2 = L bc . Let the angle ∠ abc = θ . Then the distance between a and the plane V is L ab cos θ -0 . 51 L bc .

Using cosine theorem, we have cos θ = L 2 bc + L 2 ab -L 2 ac 2 L bc L ab , so we just need to prove that L 2 bc + L 2 ab -L 2 ac 2 L bc -0 . 51 L bc ≥ L ab / 3 . 1 , that is 0 . 5 L 2 ab -0 . 5 L 2 ac -L ab L bc / 3 . 1 L 2 bc ≥ 0 . 01 . It is easy to see that such value is inversely proportional to L ac and L bc . By L ac ≤ L ab / 3 . 1 and L bc ≤ L ac + L ab ≤ 4 . 1 L ab / 3 . 1 , we have 0 . 5 L 2 ab -0 . 5 L 2 ac -L ab L bc / 3 . 1 L 2 bc ≥ 0 . 5 -0 . 5 / (3 . 1) 2 -4 . 1 / (3 . 1) 2 (4 . 1 / 3 . 1) 2 > 0 . 01 . The lemma is proved.

We now show that the algorithm is effective and has generalization guarantee.

Proof. Let F mem be the memorization network of D tr constructed by the above algorithm.

Effective. We show that F mem is a memorization of D tr can be constructed in polynomial time.

It is easy to see that, u i has width at most N , and each value of parameters can be calculated by D tr in polynomial time. So F 1 defined in (1) in section H.4 can be calculated in polynomial time. It is easy to see that the F 2 defined in (1) in section H.4 can be calculated in polynomial time. This, F can be calculated in polynomial time.

Generalization Guarantee. Let S = { ( v i , y v i ) } S D i =1 be the nearby set defined in Definition 7.1. Then, we show the result in two parts.

Part One , we show that: for a ( x i , y i ) ∈ D tr , if x i ∈ B (( v j , y v j )) for a j ∈ [ S D ] , then Sgn ( F ( x )) = y i for any x ∈ B (( v j , y v j )) .

̸

Firstly, we show that it holds B (( v j , y v j )) ∈ C i . For any k ∈ [ N ] such that y k = y i , we have || v j -x k || 2 ≥ 3 . 1 r ≥ 3 . 1 || v j -x i || , where r is the radius of B (( v j , y v j )) so by lemma H.4, the distance from v j to S ik ( x ) is greater than r , which means that the points in B (( v j , y v j )) are on the same side of the plane S ik ( x ) , by x i ∈ B (( v j , y v j )) and S ik ( x i ) > 0 as said in lemma H.1. Thus, for any x ∈ B (( v j , y v j )) , we have S ik ( x ) ≥ 0 . By C i = ∩ j ∈ [ N ] ,y i = y j { x ∈ [0 , 1] n ∥ S i,j ( x ) ≥ 0 } , we know that B (( v j , y v j )) ∈ C i .

̸

By the above result, if x ∈ B (( v j , y v j )) , then x ∈ C i ; so by lemma H.3, we have Sgn ( F ( x )) = y i for all x ∈ B (( v j , y v j )) .

Part Two , we show that if D tr ∼ D N and N ≥ S D /ϵ ln( S D /δ ) , then P D tr ∼D N ( A D ( F mem ) ≥ 1 -ϵ ) ≥ 1 -δ .

Let Q i = P ( x,y ) ∼D ( x ∈ B (( v i , y v i ))) , then without losing generality, we assume that Q 1 ≤ Q 2 ≤ · · · ≤ Q S D . Then, for the dataset D tr = { ( x i , y i ) } N i =1 , let Z ( D tr ) = { j ∈ [ S D ] ∥∃ i ∈ [ N ] , x i ∈ B (( v j , y v j )) } . The proof is given in three parts.

If i ∈ Z ( D tr ) , then by the definition of Z ( D tr ) , we know that there is a j ∈ [ N ] such that x j ∈ B (( v i , y v i )) , so by part one, we have Sgn ( F mem ( x )) = y j for any x ∈ B (( v i , y v i )) .

Moreover, for any ( x, y ) ∼ D and x ∈ B (( v i , y v i )) , by lemma H.1 and B (( v i , y v i )) ∈ C j which has been shown in part one, we know that y = y j .

So Sgn ( F mem ( x )) = y j = y for any ( x, y ) ∼ D and x ∈ B (( v i , y v i )) , which means that F mem gives the correct label to all x ∈ B (( v i , y v i )) when i ∈ Z ( D tr , S ) . So A D ( F mem ) ≥ ∑ i ∈ Z ( D tr ,S ) Q i ≥ 1 -∑ i/ ∈ Z ( D tr ,S ) Q i .

̸

Let Cc i = {D tr ∥D tr ∼ D N , i / ∈ Z ( D tr ) and j ∈ Z ( D tr ) for ∀ j > i } , easy to see that Cc j ∩ Cc i = ∅ when i = j and ∑ N i =0 P D tr ∼D N ( D tr ∈ Cc i ) = 1 . It is easy to see that, P D tr ∼D N ( D tr ∈ Cc i ) ≤ (1 -Q i ) N when i ≥ 1 .

Firstly we have that, if some i ∈ [ S D ] makes that Q i < ϵ/i , then for any D tr ∈ Cc j where j ≤ i , we have ∑ k/ ∈ Z ( D tr ) Q k ≤ ∑ j k =1 Q k ≤ jQ j ≤ iQ i < ϵ .

So that, we consider two situations.

Situation 1: There is a i ∈ [ S D ] such that Q i < ϵ/i .

Let N 0 be the biggest number in [ S D ] such that Q N 0 < ϵ/N 0 . Then we have that:

Hence, we have

The last step is to take N ≥ S D /ϵ ln( S D /δ ) in. So, taking the above result in equation 5, we have

which is what we want.

Situation 2: There is no i ∈ [ S D ] such that Q i < ϵ/i .

Then, we have

So with probability 1 -δ , we have D tr ∈ Cc 0 . When D tr ∈ Cc 0 , we have Z ( D tr ) = [ S D ] , so that ∑ i/ ∈ Z ( D tr ) Q i = 0 . Hence, P D tr ∼D N ( ∑ i/ ∈ Z ( D tr ) Q i ≤ ϵ ) ≥ 1 -δ .

I Experiments

We try to verify Theorem 7.3 on MNIST and CIFAR10 [33].

I.1 Experiment on MNIST

For MNIST, we tested all binary classification problems with different label compositions. For each pair of labels, we use 500 corresponding samples with each label in the original dataset to form a new dataset D tr , and then construct memorization network for D tr by Theorem 7.3. For each binary classification problem, Table 1 shows the accuracy on the samples with such two labels in testset.

From Table 1, we can see that the algorithm shown in the theorem 7.3 has good generalization ability for mnist, almost all result is higher than 90% .

I.2 Experiment on CIFAR10

For CIFAR10, we test all binary classification problems with different label combinations. For each pair of labels, we use 3000 corresponding samples with each label in the original dataset to form a new dataset D tr , and then construct memorization network for D tr by Theorem 7.3. For each binary classification problem, Table 2 shows the accuracy on the samples with such two labels in testset.

From Table 2, we can see that, most of the accuracies are above 70 % , but for certain pairs, the results may be poor, such as cat and dog (category 3 and category 5).

Our memorization algorithm cannot exceed the training methods empirically. Training, as a method that has been developed for a long time, is undoubtedly effective. For each pair of labels, we use 3000 corresponding samples with each label in the original dataset to form a training set D tr , and train Resnet18 [28] on D tr (with 20 epochs, learning rate 0.1, use crossentropy as loss function, device is GPU NVIDIA GeForce RTX 3090), the accuracy of the obtained network is shown in Table 3.

Comparing Tables 2 and 3, it can be seen that the training results are significantly better.

I.3 Compare with other memorization algorithm

Three memorization network construction methods are considered in this section: (M1): Our algorithm in theorem 7.3; (M2): Method in [49]; (M3): Method in [55].

In particular, we do experiments on the classification of such five pairs of numbers in MNIST: 1 and 7, 2 and 3, 4 and 9, 5 and 6, 8 and 9, to compare methods M1, M2, M3. The main basis for selecting such pairs of labels is the similarity of the numbers. For any pair of numbers, we label the smaller number as -1 and the larger number as 1. Other settings follow section I.1, and the result is given in Table 4. We can see that our method performs much better in all cases.

From table 4, our method gets the best accuracy. When constructing a memorization network, the methods (M2), (M3) compress data into one dimension, such action will break the feature of the image, so they cannot get a good generalization.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our paper supports the claims made in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed limitations in Section 8.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided the full set of assumptions in every theorem and made a complete proof in Appendix A to appendix H.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided reproductive details in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided our codes in the supplemental matrial.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided experimental details in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our main contribution is in terms of theory.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided them in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [NA]

Justification: Our main contribution is in terms of theory.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: None of this.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If we have negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use open-source dataset and models in our paper, and have cited the original paper of these dataset and models as presented in Appendix I.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the generalization capabilities of memorization networks, specifically networks that achieve optimal memorization capacity (i.e. O(sqrt(N)) parameters for memorizing N samples). The authors present a memorization algorithm that is based on the construction of Vardi et al. Next, they show that memorization with fixed width or small number of parameters cannot generalize well. They also provide upper and lower bounds for the communication complexity of learning with a memorization algorithm, while also providing an efficient memorization algorithm.\n\nSTRENGTHS:\n- I think this research direction is novel and interesting. It puts a spotlight on memorization results, which may be efficient in terms of expressiveness but their generalization capabilities are unclear.\n\n- The presentation of the paper is good and it is easy to follow, while also providing sufficient proof intuitions.\n\n- Section 5 is particularly interesting since it shows the limitations of efficient memorization results, and specifically that memorization networks cannot generalize well.\n\n- Section 6 is also very extensive, and provides tight sample complexity bounds on memorization learning, that connects the size of the memorization network to the number of required samples for learning.\n\nWEAKNESSES:\nI think this paper is very nice overall, however, there are some issues with specific results which I would be happy to see the author’s response about:\n- Proposition 3.8 - where is the proof? Also, it is not clear to me why the constructions in [54,48] are probabilistic. \n\n- There is something unclear about the proof of Theorem 5.3. The theorem statement is that for every distribution D and any test set D_tr there exists a memorization network that doesn’t generalize. However, in the proof, the authors begin with a given set D_tr, and then define D based on this training set, i.e. D is defined by some set S which in itself depends on D_tr. I think the statement should be that there exists a distribution D such that for all training set D_tr  there exists a memorization network that doesn’t generalize. Note that the current proof doesn’t show it, because D is constructed such that it depends on D_tr.\n\n- The comparison between interpolation and memorization learning is unclear. Specifically, in what cases do the results on interpolation learning already provide generalization bounds for memorization learning? \n\nSome minor comments:\n\n- Results and comparisons on implicit bias toward margin maximization of neural networks do not appear (e.g. https://arxiv.org/abs/1906.05890, https://arxiv.org/abs/2006.06657). They also study a regime where the optimization converged to 100% accuracy on the dataset and provide generalization guarantees based on margin arguments.\n\n- In the main contribution, point 3, it is better to write lower bound is \\Omega rather than O.\n\nQUESTIONS:\n- How large can S_D be? It seems that for any continuous distribution on [0,1]^n, S_D is already exponential in n, hence having such a large sample complexity makes this bound vacuous. Can the authors provide some examples where S_D and N_D are not very far apart?\n\n- What is the size (num of parameters, width, depth) of the network defined in Theorem 7.3?\n\nLIMITATIONS:\nThe authors discuss the limitations of their results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-19"}

comments: SUMMARY:
This work studies the generalization properties of neural network memorization algorithms. Several results are proved: (1) Construction of a memorization network for a sampled dataset, with an optimal number of parameters. (2) There exists a constant such that for all datasets sampled from a distribution, there exists a memorization network with at most the constant number of parameters. (3) Sample complexity lower and upper bounds for memorization algorithms. (4) An efficient memorization algorithm with a generalization guarantee.

STRENGTHS:
Solid mathematical paper with several novel results in DL theory. Mostly clearly written.

WEAKNESSES:
-	The significance of the theoretical results are not very clear because no estimates of $N_\mathcal{D}$ and $S_\mathcal{D}$ are given.

-	Writing can be improved, e.g.: “is also a challenge problem” line 408, “which is more challenging compare to linear model” line 44, “and shows that even if there is enough data” line 63, “In other word” line 87.

QUESTIONS:
Can estimates of $N_\mathcal{D}$ and $S_\mathcal{D}$ be provided even for simple toy examples (e.g., linearly separable data)?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a first, thorough theoretical understanding of the generalization ability of the memorized neural network, including the existence of the memorized neural network, the matching upper bound and lower bound on the sample complexity, the computational infeasibility of attaining optimal sample complexity, and a computation-efficient but sub-optimal remedy.\n\nSTRENGTHS:\nOverall, the paper is well-written, the results are novel and comprehensive, and it makes a clear contribution to the theoretical community.\n\nWEAKNESSES:\nThere are some comments, shown in the Questions below, but I recommend accepting this paper.\n\nQUESTIONS:\nMajor Comments:\n1. I think the hypothesis class D(n,c) is simple to be learned. I guess, for fixed (c, n), if the density in the support of X is bounded from below and above, then the minimax (worst D) error rate in D is the parametric rate? Hence this is a relatively easy class of function to be analyzed, and there is almost no noise, which may weaken the results of this paper.\n2. The existence result in Theorem 4.3 is nice. But it should be clear whether the memorization algorithm L is dependent on D, or at least c. It will be good if such an algorithm can obtain memorization agnostic to the knowledge of c.\n3. The upper bound and lower bound does not exactly match in its current form in terms of epsilon. I guess the lower bound is not tight, because it does not rule out the possibility that a fixed number of samples can have exact recovery (eps=delta=0), this is relatively loose. Can it be improved?\n\n\n\nMinor comments:\n1. I think the introduction part can make a clear, or informal definition of what is generalization under the well-separated classification task given it lists all the theoretical results.\n\nLIMITATIONS:\nSee the Questions above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The authors study the memorization capacity of ReLU networks and its generalization theory for i.i.d. datasets over the compact domain, under a binary classification setting.\n- They propose two different memorization capacity upper bound of ReLU networks. One bound depends on the size $N$ of the training dataset ($O(\\sqrt{N}\\ln{N})$) and the other bound is independent of $N$. Notably, the second bound implies that even with an arbitrarily large dataset sampled i.i.d., a constant number $N_{\\mathcal{D}}$ of the parameters is enough to memorize the dataset, where $N_{\\mathcal{D}}$ depends only on the true data distribution $\\mathcal{D}$.\n- They show that memorization is not enough for generalization with concrete examples. First, a generalizable memorization network must have a width of at least $n$ (the data dimension). Second, for almost every data distribution, there exists a non-generalizable memorization network with $O(\\sqrt{N}\\ln{N})$ parameters ($N$: size of training dataset).\n- They study the sample complexity for $(\\\\epsilon,\\\\delta)$-PAC-generalizable memorization networks. First, they provide a lower bound of sample complexity for general memorization networks. Second, they provide an upper bound of sample complexity for particular memorization networks with at most $N_{\\mathcal{D}}$ parameters. Both bounds are tight in terms of $N_{\\mathcal{D}}$ up to logarithmic factors.\n- Lastly, they study the algorithm constructing a PAC-generalizable memorization network in poly$(N)$ time.\n\nSTRENGTHS:\n- S1. As far as I know, this is the first work studying the generalizability of the memorization ReLU network, under the binary classification task.\n- S2. In terms of ReLU networks, they discover two novel complexity terms $N_{\\mathcal{D}}$ (smallest number of parameters to memorize i.i.d. dataset (of any size) from $\\mathcal{D}$) and $S_{\\mathcal{D}}$ (”efficient memorization sample complexity”; minimum size of the nearby set of  $\\mathcal{D}$) which depend only on the true data distribution.\n- S3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n- W1. Compactness of the input domain\n    - Line 47 of the introduction says the paper will consider the open domain $\\mathbb{R}^n$ of input. However, only the compact input domain $[0,1]^n$ is considered throughout the paper. It is unclear whether the results on the compact domain can be easily extended to the  $\\mathbb{R}^n$ case. In the approximation literature, it is important whether the input domain is compact or not. Thus, the authors should mention whether the extension to a larger domain $\\mathbb{R}^n$ is easy. Or, it would be better to replace $\\mathbb{R}^n$ with a compact input domain in the introduction.\n- W2. Randomness in Theorem 4.1\n    - Theorem 4.1 contains a probabilistic argument. However, it is unclear where the randomness comes from, with the theorem statement alone.\n- W3. Efficiency of memorization algorithm in Theorem 4.3\n    - It would be meaningful to mention whether the memorization algorithm in Theorem 4.3 is efficient.\n- W4. Parameter complexity of efficient memorization algorithm in Theorem 7.3\n    - It would be meaningful to clarify how many parameters the network constructed by the efficient memorization algorithm in Theorem 7.3 should have.\n- W6. Minor comments on notations/typos (There are so many typos…)\n    - Title: “Generalizablity” → “Generalizability”\n    - Line 60: “…, (2) of Theorem 1.1…”\n    - Please use $\\overline{O}$ for upper bounds, $\\overline{\\Omega}$ for lower bounds, and $\\overline{\\Theta}$ for tight bounds (up to logarithmic factors).\n    - Section 3.1: “$X_0 = x$” and “$n_0 = n$” must be explicitly mentioned.\n    - As far as i know, $\\mathbb{R}\\_{+}$ and $\\mathbb{Z}\\_{+}$ usually include zero. To exclude zero, $\\mathbb{R}\\_{++}$ (or $\\mathbb{R}\\_{>0}$) and $\\mathbb{Z}\\_{++}$ (or $\\mathbb{Z}\\_{>0}$) are better to use.\n    - Line 250: “generaizable” → “generalizable”\n    - Line 274: “The density of distribution $\\mathcal{D}$”\n    - Line 279: “$A_{\\mathcal{D}} (\\mathcal{F}) \\le 0.51$”\n    - Line 291: “generazable” → “generalizable”\n    - Line 309: “generalizability,.” → “generalizability.”\n    - Line 321: “$N_D$” → “$N_{\\mathcal{D}}$” (calligraphic D)\n    - Line 325: “bpund” → “bound”\n    - Line 337: “$\\mathcal{D}_{\\rm tr} \\sim \\mathcal{D}^N$\n    - Lines 340 and 472: “Proof Idea.” is not in a bold font.\n    - Line 367: What does it mean by “a subset of $(x,y)\\sim \\mathcal{D}$”?\n    - Line 382: Is “$S_{\\mathcal{D}} \\ge N_{\\mathcal{D}}^2$” really true? I guess it should be “$S_{\\mathcal{D}} \\gtrsim N_{\\mathcal{D}}^2$”, ignoring logarithmic factors.\n    - Line 619: “Uisng” → “Using”\n    - Step Three in the proof of Theorem 4.3 (Appendix C): Please make clear which theorem of [61] you refer to.\n    - Line 1052: “Tree” → “Three”\n    - Section F: a very non-standard notation is used here: “$C^m_n$” means a binomial coefficient “$\\binom{n}{m}$”! This makes the proof really hard to read. Please consider changing the notation.\n    - Part 2 in the proof of Theorem 6.1: I guess $12v_1 \\le N_{\\mathcal{D}} \\le 4v_1 \\sqrt{k} \\ln(\\sqrt{k})$ is required for the later use, rather than $3 \\le N_{\\mathcal{D}} \\le 4v_1 \\sqrt{k} \\ln(\\sqrt{k})$. This can be fixed with minor changes.\n    - Lines 1148-1149: Make clear that $S(\\mathcal{D}) \\subset [k]^q$.\n    - Equation (4) and the rest of the proof of Theorem 6.1: I think $q^k$ should be replaced with $k^q$. See lines 1166, 1167, 1195, and 1196.\n    - Line 1189: “Because” → “This is because”\n    - Lines 1189-1192: several “$S_{ss}(\\mathcal{D},Z)$”s are written in wrong symbols.\n    - Line 1213: “Here, $E\\_{\\mathcal{D}}(\\mathcal{F}) = \\mathbb{E}\\_{(x,y)\\sim \\mathcal{D}} [I(\\operatorname{Sgn}(\\mathcal{F}(x))=y)]$,  $E\\_{\\mathcal{D}\\_{\\rm tr}}(\\mathcal{F}) = \\frac{1}{N} \\sum_{(x,y) \\in \\mathcal{D}\\_{\\rm tr}} I(\\operatorname{Sgn}(\\mathcal{F}(x))=y)$ ”.\n    - Line 1224: “$n$” → “$m$”\n    - Line 1249: “similarto” → “similar to”\n    - and many more…\n\nQUESTIONS:\n- Q1. Architectural constraint: ReLU network\n    - A clear limitation of this work is that it is limited to a fully-connected ReLU network.\n    - Can you imagine how the change in activation function will affect the memorization capacity, $N_{\\mathcal{D}}$, and $S_{\\mathcal{D}}$?\n- Q2. Sample complexity bound gap of memorization algorithms in Section 6\n    - Currently, there is an $\\epsilon^{-2}$ gap between the upper and lower sample complexity for the generalization of memorization networks. I guess this is quite huge. Do you think this gap can be reduced?\n- Q3. Proof of Corollary 6.4\n    - The proof is too short. How did you get to the statement of Corollary 6.4 by plugging $k=2^{[\\tfrac{n}{\\lceil c^2 \\rceil}]}$? It seems unclear to me.\n- Q4. Algorithm in Theorem 7.3 as an initialization\n    - Empirically, what if you run SGD (or other standard optimizers) with logistic loss starting from the network parameters constructed by the algorithm of Theorem 7.3? Will it be better than the result of SGD from random initialization?\n- Q5. Generalizability of memorization v.s. non-memorization networks\n    - I know this question is somewhat out of topic, but can you discuss the generalizability of memorization and non-memorization networks? Can a non-memorization network generalize better than some memorization networks? If so, when can it happen?\n\nLIMITATIONS:\nThe authors have discussed the limitation in Section 8 (Conclusion). The major limitation is that it is unclear how to compute the numerical complexities $N_{\\mathcal{D}}$, $S_{\\mathcal{D}}$.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}