Title: Computational Complexity of Learning Neural Networks: Smoothness and Degeneracy
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning neural networks

Computational complexity

Hardness of learning

Smoothed analysis

Degenerate weights
Abstract: Understanding when neural networks can be learned efficiently	is a fundamental question in learning theory.	Existing hardness results suggest that assumptions on both the input distribution and the network's weights are necessary for obtaining efficient algorithms. Moreover, it was previously shown that depth-$2$ networks can be efficiently learned under the assumptions that the input distribution is Gaussian, and the weight matrix is non-degenerate. In this work, we study whether such assumptions may suffice for learning deeper networks and prove negative results. We show that learning depth-$3$ ReLU networks under the Gaussian input distribution is hard even in the smoothed-analysis framework, where a random noise is added to the network's parameters. It implies that learning depth-$3$ ReLU networks under the Gaussian distribution is hard even if the weight matrices are non-degenerate. Moreover, we consider depth-$2$ networks, and show hardness of learning in the smoothed-analysis framework, where both the network parameters and the input distribution are smoothed. Our hardness results are under a well-studied assumption on the existence of local pseudorandom generators.

Computational Complexity of Learning Neural Networks: Smoothness and Degeneracy

Abstract

Understanding when neural networks can be learned efficiently is a fundamental question in learning theory. Existing hardness results suggest that assumptions on both the input distribution and the network's weights are necessary for obtaining efficient algorithms. Moreover, it was previously shown that depth2 networks can be efficiently learned under the assumptions that the input distribution is Gaussian, and the weight matrix is non-degenerate. In this work, we study whether such assumptions may suffice for learning deeper networks and prove negative results. We show that learning depth3 ReLU networks under the Gaussian input distribution is hard even in the smoothed-analysis framework, where a random noise is added to the network's parameters. It implies that learning depth3 ReLU networks under the Gaussian distribution is hard even if the weight matrices are non-degenerate. Moreover, we consider depth2 networks, and show hardness of learning in the smoothed-analysis framework, where both the network parameters and the input distribution are smoothed. Our hardness results are under a wellstudied assumption on the existence of local pseudorandom generators.

1 Introduction

The computational complexity of learning neural networks has been extensively studied in recent years, and there has been much effort in obtaining both upper bounds and hardness results. Nevertheless, it is still unclear when neural networks can be learned in polynomial time, namely, under what assumptions provably efficient algorithms exist.

Existing results imply hardness already for learning depth2 ReLU networks in the standard PAC learning framework (e.g., [31, 14]). Thus, without any assumptions on the input distribution or the network's weights, efficient learning algorithms might not be achievable. Even when assuming that the input distribution is Gaussian, strong hardness results were obtained for depth3 ReLU networks [16, 11], suggesting that assumptions merely on the input distribution might not suffice. Also, a hardness result by Daniely and Vardi [15] shows that strong assumptions merely on the network's weights (without restricting the input distribution) might not suffice even for efficiently learning depth2 networks. The aforementioned hardness results hold already for improper learning , namely, where the learning algorithm is allowed to return a hypothesis that does not belong to the considered hypothesis class. Thus, a combination of assumptions on the input distribution and the network's weights seems to be necessary for obtaining efficient algorithms.

Several polynomial-time algorithms for learning depth2 neural networks have been obtained, under assumptions on the input distribution and the network's weights [6, 28, 43, 20, 7, 21]. In these works, it is assumed that the weight matrices are non-degenerate. That is, they either assume that the condition number of the weight matrix is bounded, or some similar non-degeneracy assumption. Specifically, Awasthi et al. [6] gave an efficient algorithm for learning depth2 (one-hidden-layer) ReLU networks, that may include bias terms in the hidden neurons, under the assumption that the input distribution is Gaussian, and the weight matrix is non-degenerate. The non-degeneracy assumption holds w.h.p. if we add a small random noise to any weight matrix, and hence their result implies efficient learning of depth2 ReLU networks under the Gaussian distribution in the smoothed-analysis framework.

The positive results on depth2 networks suggest the following question:

Is there an efficient algorithm for learning ReLU networks of depth larger than 2 under the Gaussian distribution, where the weight matrices are non-degenerate, or in the smoothed-analysis framework where the network's parameters are smoothed?

In this work, we give a negative answer to this question, already for depth3 networks 1 . We show that learning depth3 ReLU networks under the Gaussian distribution is hard even in the smoothedanalysis framework, where a random noise is added to the network's parameters. As a corollary, we show that learning depth3 ReLU networks under the Gaussian distribution is hard even if the weight matrices are non-degenerate. Our hardness results are under a well-studied cryptographic assumption on the existence of local pseudorandom generators (PRG) with polynomial stretch.

Motivated by the existing positive results on smoothed-analysis in depth2 networks, we also study whether learning depth2 networks with smoothed parameters can be done under weak assumptions on the input distribution. Specifically, we consider the following question:

Is there an efficient algorithm for learning depth2 ReLU networks in the smoothedanalysis framework, where both the network's parameters and the input distribution are smoothed?

We give a negative answer to this question, by showing hardness of learning depth2 ReLU networks where a random noise is added to the network's parameters, and the input distribution on R d is obtained by smoothing an i.i.d. Bernoulli distribution on { 0 , 1 } d . This hardness result is also under the assumption on the existence of local PRGs.

Related work

Hardness of learning neural networks. Hardness of improperly learning depth2 neural networks follows from hardness of improperly learning DNFs or intersections of halfspaces, since these classes can be expressed by depth2 networks. Klivans and Sherstov [31] showed, assuming the hardness of the shortest vector problem, that learning intersections of halfspaces is hard. Hardness of learning DNF formulas is implied by Applebaum et al. [4] under a combination of two assumptions: the first is related to the planted dense subgraph problem in hypergraphs, and the second is related to local PRGs. Daniely and Shalev-Shwartz [14] showed hardness of learning DNFs under a common assumption, namely, that refuting a random K -SAT formula is hard. All of the above results are distribution-free, namely, they do not imply hardness of learning neural networks under some specific distribution.

Applebaum and Raykov [3] showed, under an assumption on a specific candidate for Goldreich's PRG (i.e., based on a predicate called XOR-MAJ ), that learning depth3 Boolean circuits under the uniform distribution on the hypercube is hard. Daniely and Vardi [16] proved distribution-specific hardness of learning Boolean circuits of depth2 (namely, DNFs) and depth3 , under the assumpion on the existence of local PRGs that we also use in this work. For DNF formulas, they showed hardness of learning under a distribution where each component is drawn i.i.d. from a Bernoulli distribution (which is non-uniform). For depth3 Boolean circuits, they showed hardness of learning under the uniform distribution on the hypercube. Since the Boolean circuits can be expressed by ReLU networks of the same depth, these results readily translate to distribution-specific hardness of learning neural networks. Chen et al. [11] showed hardness of learning depth2 neural networks under the uniform distribution on the hypercube, based on an assumption on the hardness of the

Learning with Rounding (LWR) problem. Note that the input distributions in the above results are supported on the hypercube, and they do not immediately imply hardness of learning neural networks under continuous distributions.

When considering the computational complexity of learning neural networks, perhaps the most natural choice of an input distribution is the standard Gaussian distribution. Daniely and Vardi [16] established hardness of learning depth3 networks under this distribution, based on the assumpion on the existence of local PRGs. Chen et al. [11] also showed hardness of learning depth3 networks under the Gaussian distribution, but their result holds already for networks that do not have an activation function in the output neuron, and it is based on the LWR assumption. They also showed hardness of learning constant depth ReLU networks from label queries (i.e., where the learner has the ability to query the value of the target network at any desired input) under the Gaussian distribution, based either on the decisional Diffie-Hellman or the Learning with Errors assumptions.

The above results suggest that assumptions on the input distribution might not suffice for achieving an efficient algorithm for learning depth3 neural networks. A natural question is whether assumptions on the network weights may suffice. Daniely and Vardi [15] showed (under the assumption that refuting a random K -SAT formula is hard) that distribution-free learning of depth2 neural networks is hard already if the weights are drawn from some 'natural' distribution or satisfy some 'natural' properties. Thus, if we do not impose any assumptions on the input distribution, then even very strong assumptions on the network's weights do not suffice for efficient learning.

Several works in recent years have shown hardness of distribution-specific learning shallow neural networks using gradient-methods or statistical query (SQ) algorithms [38, 39, 41, 24, 18, 11]. It is worth noting that while the SQ framework captures some variants of the gradient-descent algorithm, it does not capture, for example, stochastic gradient-descent (SGD), which examines training points individually (see a discussion in [24]).

We emphasize that none of the above distribution-specific hardness results for neural networks (either for improper learning or SQ learning) holds in the smoothed analysis framework or for non-degenerate weights.

Learning neural networks in polynomial time. Awasthi et al. [6] gave a polynomial-time algorithm for learning depth2 (one-hidden-layer) ReLU networks, under the assumption that the input distribution is Gaussian, and the weight matrix of the target network is non-degenerate. Their algorithm is based on tensor decomposition, and it can handle bias terms in the hidden layer. Their result also implies that depth2 ReLU networks with Gaussian inputs can be learned efficiently under the smoothed-analysis framework. Our work implies that such a result might not be possible in depth3 networks (with activation in the output neuron). Prior to [6], several works gave polynomial time algorithms for learning depth2 neural networks where the input distribution is Gaussian and the weight matrix is non-degenerate [28, 43, 20, 21, 7], but these works either do not handle the presence of bias terms or do not handle the ReLU activation. Some of the aforementioned works consider networks with multiple outputs, and allow certain non-Gaussian input distributions. Provable guarantees for learning neural networks in super-polynomial time were given in [18, 12, 17, 22, 41, 42, 23].

2 Preliminaries

Notations. We use bold-face letters to denote vectors, e.g., x = ( x 1 , . . . , x d ) . For a vector x and a sequence S = ( i 1 , . . . , i k ) of k indices, we let x S = ( x i 1 , . . . , x i k ) , i.e., the restriction of x to the indices S . We denote by ✶ [ · ] the indicator function, for example ✶ [ t ≥ 5] equals 1 if t ≥ 5 and 0 otherwise. For an integer d ≥ 1 we denote [ d ] = { 1 , . . . , d } . We denote by N (0 , σ 2 ) the normal distribution with mean 0 and variance σ 2 , and by N ( 0 , Σ) the multivariate normal distribution with mean 0 and covariance matrix Σ . The identity matrix of size d is denoted by I d . For x ∈ R d we denote by ∥ x ∥ the Euclidean norm. We use poly( a 1 , . . . , a n ) to denote a polynomial in a 1 , . . . , a n .

Local pseudorandom generators. An ( n, m, k ) -hypergraph is a hypergraph over n vertices [ n ] with m hyperedges S 1 , . . . , S m , each of cardinality k . Each hyperedge S = ( i 1 , . . . , i k ) is ordered, and all the k members of a hyperedge are distinct. We let G n,m,k be the distribution over such hypergraphs in which a hypergraph is chosen by picking each hyperedge uniformly and independently at random among all the possible n · ( n -1) · . . . · ( n -k +1) ordered hyperedges. Let P : { 0 , 1 } k →

{ 0 , 1 } be a predicate, and let G be a ( n, m, k ) -hypergraph. We call Goldreich's pseudorandom generator (PRG) [25] the function f P,G : { 0 , 1 } n → { 0 , 1 } m such that for x ∈ { 0 , 1 } n , we have f P,G ( x ) = ( P ( x S 1 ) , . . . , P ( x S m )) . The integer k is called the locality of the PRG. If k is a constant then the PRG and the predicate P are called local . We say that the PRG has polynomial stretch if m = n s for some constant s > 1 . We let F P,n,m denote the collection of functions f P,G where G is an ( n, m, k ) -hypergraph. We sample a function from F P,n,m by choosing a random hypergraph G from G n,m,k .

We denote by G R ← - G n,m,k the operation of sampling a hypergraph G from G n,m,k , and by x R ← -{ 0 , 1 } n the operation of sampling x from the uniform distribution on { 0 , 1 } n . We say that F P,n,m is ε -pseudorandom generator ( ε -PRG) if for every polynomial-time probabilistic algorithm A the distinguishing advantage

is at most ε . Thus, the distinguisher A is given a random hypergraph G and a string y ∈ { 0 , 1 } m , and its goal is to distinguish between the case where y is chosen at random, and the case where y is a random image of f P,G .

Our assumption is that local PRGs with polynomial stretch and constant distinguishing advantage exist:

Assumption 2.1. For every constant s > 1 , there exists a constant k and a predicate P : { 0 , 1 } k → { 0 , 1 } , such that F P,n,n s is 1 3 -PRG.

Weremark that the same assumption was used by Daniely and Vardi [16] to show hardness-of-learning results. Local PRGs have been extensively studied in the last two decades. In particular, local PRGs with polynomial stretch have shown to have remarkable applications, such as secure-computation with constant computational overhead [27, 5], and general-purpose obfuscation based on constant degree multilinear maps (cf. [33, 34]). Significant evidence for Assumption 2.1 was shown in Applebaum [1]. Moreover, a concrete candidate for a local PRG, based on the XOR-MAJ predicate, was shown to be secure against all known attacks [2, 13, 35, 3]. See [16] for further discussion on the assumption, and on prior work regarding the relation between Goldreich's PRG and hardness of learning.

Neural networks. We consider feedforward ReLU networks. Starting from an input x ∈ R d , each layer in the network is of the form z ↦→ σ ( W i z + b i ) , where σ ( a ) = [ a ] + = max { 0 , a } is the ReLU activation which applies to vectors entry-wise, W i is the weight matrix, and b i are the bias terms. The weights vector of the j -th neuron in the i -th layer is the j -th row of W i , and its outgoing-weights vector is the j -th column of W i +1 . We define the depth of the network as the number of layers. Unless stated otherwise, the output neuron also has a ReLU activation function. Note that a depthk network with activation in the output neuron has k non-linear layers. A neuron which is not an input or output neuron is called a hidden neuron . We sometimes consider neural networks with multiple outputs. The parameters of the neural network is the set of its weight matrices and bias vectors. We often view the parameters as a vector θ ∈ R p obtained by concatenating these matrices and vectors. For B ≥ 0 , we say that the parameters are B -bounded if the absolute values of all weights and biases are at most B .

Learning neural networks and the smoothed-analysis framework. We first define neural networks learning under the standard PAC framework:

Definition 2.1 (Distribution-specific PAC learning) . Learning depthk neural networks under an input distribution D on R d is defined by the following framework:

An adversary chooses a set of B -bounded parameters θ ∈ R p for a depthk neural network N θ : R d → R , as well as some ϵ > 0 .

Consider an examples oracle, such that each example ( x , y ) ∈ R d × R is drawn i.i.d. with x ∼ D and y = N θ ( x ) . Then, given access to the examples oracle, the goal of the learning algorithm L is to return with probability at least 3 4 a hypothesis h : R d → R such that E x ∼D [ ( h ( x ) -N θ ( x )) 2 ] ≤ ϵ . We say that L is efficient if it runs in time poly( d, p, B, 1 /ϵ ) .

We consider learning in the smoothed-analysis framework [40], which is a popular paradigm for analyzing non-worst-case computational complexity [36]. The smoothed-analysis framework has been successfully applied to many learning problems (e.g., [6, 21, 30, 8, 9, 26, 19, 29, 10]). In the smoothed-analysis setting, the target network is not purely controlled by an adversary. Instead, the adversary can first generate an arbitrary network, and the parameters for this network (i.e., the weight matrices and bias terms) will be randomly perturbed to yield a perturbed network. The algorithm only needs to work with high probability on the perturbed network. This limits the power of the adversary and prevents it from creating highly degenerate cases. Formally, we consider the following framework (we note that a similar model was considered in [6, 21]):

Definition 2.2 (Learning with smoothed parameters) . Learning depthk neural networks with smoothed parameters under an input distribution D is defined as follows:

An adversary chooses a set of B -bounded parameters θ ∈ R p for a depthk neural network N θ : R d → R , as well as some τ, ϵ > 0 .

A perturbed set of parameters ˆ θ is obtained by a random perturbation to θ , namely, ˆ θ = θ + ξ for ξ ∼ N ( 0 , τ 2 I p ) .

Consider an examples oracle, such that each example ( x , y ) ∈ R d × R is drawn i.i.d. with x ∼ D and y = N ˆ θ ( x ) . Then, given access to the examples oracle, the goal of the learning algorithm L is to return with probability at least 3 4 (over the random perturbation ξ and the internal randomness of L ) a hypothesis h : R d → R such that E x ∼D [ ( h ( x ) -N ˆ θ ( x ) ) 2 ] ≤ ϵ . We say that L is efficient if it runs in time poly( d, p, B, 1 /ϵ, 1 /τ ) .

Finally, we also consider a setting where both the parameters and the input distribution are smoothed:

Definition 2.3 (Learning with smoothed parameters and inputs) . Learning depthk neural networks with smoothed parameters and inputs under an input distribution D is defined as follows:

An adversary chooses a set of B -bounded parameters θ ∈ R p for a depthk neural network N θ : R d → R , as well as some τ, ω, ϵ > 0 .

A perturbed set of parameters ˆ θ is obtained by a random perturbation to θ , namely, ˆ θ = θ + ξ for ξ ∼ N ( 0 , τ 2 I p ) . Moreover, a smoothed input distribution ˆ D is obtained from D , such that ˆ x ∼ ˆ D is chosen by drawing x ∼ D and adding a random perturbation from N ( 0 , ω 2 I d ) .

Consider an examples oracle, such that each example ( x , y ) ∈ R d × R is drawn i.i.d. with x ∼ ˆ D and y = N ˆ θ ( x ) . Then, given access to the examples oracle, the goal of the learning algorithm L is to return with probability at least 3 4 (over the random perturbation ξ and the internal randomness of L ) a hypothesis h : R d → R such that E x ∼ ˆ D [ ( h ( x ) -N ˆ θ ( x ) ) 2 ] ≤ ϵ . We say that L is efficient if it runs in time poly( d, p, B, 1 /ϵ, 1 /τ, 1 /ω ) .

We emphasize that all of the above definitions consider learning in the distribution-specific setting. Thus, the learning algorithm may depend on the specific input distribution D .

3 Results

As we discussed in the introduction, there exist efficient algorithms for learning depth2 (one-hiddenlayer) ReLU networks with smoothed parameters under the Gaussian distribution. We now show that such a result may not be achieved for depth3 networks:

Theorem 3.1. Under Assumption 2.1, there is no efficient algorithm that learns depth3 networks with smoothed parameters (in the sense of Definition 2.2) under the standard Gaussian distribution.

We prove the theorem in Section 4. Next, we conclude that learning depth3 neural networks under the Gaussian distribution on R d is hard in the standard PAC framework even if all weight matrices are non-degenerate, namely, when the minimal singular values of the weight matrices are lower bounded by 1 / poly( d ) . As we discussed in the introduction, in one-hidden-layer networks with similar assumptions there exist efficient learning algorithms.

Corollary 3.1. Under Assumption 2.1, there is no efficient algorithm that learns depth3 networks (in the sense of Definition 2.1) under the standard Gaussian distribution on R d , even if the smallest singular value of each weight matrix is at least 1 / poly( d ) .

The proof of the above corollary follows from Theorem 3.1, using the fact that by adding a small random perturbation to the weight matrices, we get w.h.p. non-degenerate matrices. Hence, an efficient algorithm that learns non-degenerate networks suffices for obtaining an efficient algorithm that learns under the smoothed analysis framework. See Appendix B for the formal proof.

The above hardness results consider depth3 networks that include activation in the output neuron. Thus, the networks have three non-linear layers. We remark that these results readily imply hardness also for depth4 networks without activation in the output, and hardness for depthk networks for any k > 3 . We also note that the hardness results hold already for networks where all hidden layers are of the same width, e.g., where all layers are of width d .

Theorem 3.1 gives a strong limitation on learning depth3 networks in the smoothed-analysis framework. Motivated by existing positive results on smoothed-analysis in depth2 networks, we now study whether learning depth2 networks with smoothed parameters can be done under weak assumptions on the input distribution. Specifically, we consider the case where both the parameters and the input distribution are smoothed. We show that efficiently learning depth2 networks may not be possible with smoothed parameters where the input distribution on R d is obtained by smoothing an i.i.d. Bernoulli distribution on { 0 , 1 } d .

Theorem 3.2. Under Assumption 2.1, there is no efficient algorithm that learns depth2 networks with smoothed parameters and inputs (in the sense of Definition 2.3), under the distribution D on { 0 , 1 } d where each coordinate is drawn i.i.d. from a Bernoulli distribution which takes the value 0 with probability 1 √ d .

We prove the theorem in Appendix C. The proof follows from similar ideas to the proof of Theorem 3.1, which we discuss in the next section.

4 Proof of Theorem 3.1

The proof builds on a technique from Daniely and Vardi [16]. It follows by showing that an efficient algorithm for learning depth3 neural networks with smoothed parameters under the Gaussian distribution can be used for breaking a local PRG. Intuitively, the main challenge in our proof in comparison to [16] is that our reduction must handle the random noise which is added to the parameters. Specifically, [16] define a certain examples oracle, and show that the examples returned by the oracle are realizable by some neural network which depends on the unknown x ∈ { 0 , 1 } n used by the PRG. Since the network depends on this unknown x , some of the parameters of the network are unknown, and hence it is non-trivial how to define an examples oracle which is realizable by a perturbed network. Moreover, we need to handle this random perturbation without increasing the network's depth. For these reasons, our reduction is significantly different from [16].

We now provide the formal proof. The proof relies on several lemmas which we prove in Appendix A. For a sufficiently large n , let D be the standard Gaussian distribution on R n 2 . Assume that there is a poly( n ) -time algorithm L that learns depth3 neural networks with at most n 2 hidden neurons and parameter magnitudes bounded by n 3 , with smoothed parameters, under the distribution D , with ϵ = 1 n , and τ = 1 / poly( n ) that we will specify later. Let m ( n ) ≤ poly( n ) be the sample complexity of L , namely, L uses a sample of size at most m ( n ) and returns w.p. at least 3 4 a hypothesis h with E z ∼D [ ( h ( z ) -N ˆ θ ( z ) ) 2 ] ≤ ϵ = 1 n , where N ˆ θ is the perturbed network. Let s > 1 be a constant such that n s ≥ m ( n ) + n 3 for every sufficiently large n . By Assumption 2.1, there exists a constant k and a predicate P : { 0 , 1 } k → { 0 , 1 } , such that F P,n,n s is 1 3 -PRG. We will show an efficient algorithm A with distinguishing advantage greater than 1 3 and thus reach a contradiction.

Throughout this proof, we will use the following notations. For a hyperedge S = ( i 1 , . . . , i k ) we denote by z S ∈ { 0 , 1 } kn the following encoding of S : the vector z S is a concatenation of k vectors in { 0 , 1 } n , such that the j -th vector has 0 in the i j -th coordinate and 1 elsewhere. Thus, z S consists of k sizen slices, each encoding a member of S . For z ∈ { 0 , 1 } kn , i ∈ [ k ] and j ∈ [ n ] , we denote z i,j = z ( i -1) n + j . That is, z i,j is the j -th component in the i -th slice in z . For x ∈ { 0 , 1 } n , let

P x : { 0 , 1 } kn → { 0 , 1 } be such that for every hyperedge S we have P x ( z S ) = P ( x S ) . Let c be such that Pr t ∼N (0 , 1) [ t ≤ c ] = 1 n . Let µ be the density of N (0 , 1) , let µ -( t ) = n · ✶ [ t ≤ c ] · µ ( t ) , and let µ + = n n -1 · ✶ [ t ≥ c ] · µ ( t ) . Note that µ -, µ + are the densities of the restriction of µ to the intervals t ≤ c and t ≥ c respectively. Let Ψ : R kn →{ 0 , 1 } kn be a mapping such that for every z ′ ∈ R kn and i ∈ [ kn ] we have Ψ( z ′ ) i = ✶ [ z ′ i ≥ c ] . For ˜ z ∈ R n 2 we denote ˜ z [ kn ] = (˜ z 1 , . . . , ˜ z kn ) , namely, the first kn components of ˜ z (assuming n 2 ≥ kn ).

4.1 Defining the target network for L

Since our goal is to use the algorithm L for breaking PRGs, in this subsection we define a neural network ˜ N : R n 2 → R that we will later use as a target network for L . The network ˜ N contains the subnetworks N 1 , N 2 , N 3 which we define below.

Let N 1 be a depth2 neural network with input dimension kn , at most n log( n ) hidden neurons, at most log( n ) output neurons (with activations in the output neurons), and parameter magnitudes bounded by n 3 (all bounds are for a sufficiently large n ), which satisfies the following. We denote the set of output neurons of N 1 by E 1 . Let z ′ ∈ R kn be an input to N 1 such that Ψ( z ′ ) = z S for some hyperedge S , and assume that for every i ∈ [ kn ] we have z ′ i ̸∈ ( c, c + 1 n 2 ) . Fix some x ∈ { 0 , 1 } n . Then, for S with P x ( z S ) = 0 the inputs to all output neurons E 1 are at most -1 , and for S with P x ( z S ) = 1 there exists a neuron in E 1 with input at least 2 . Recall that our definition of a neuron's input includes the addition of the bias term. The construction of the network N 1 is given in Lemma A.3. Intuitively, the network N 1 consists of a layer that transforms w.h.p. the input z ′ to Ψ( z ′ ) = z S , followed by a layer that satisfies the following: Building on a lemma from [16] which shows that P x ( z S ) can be computed by a DNF formula, we define a layer where each output neuron corresponds to a term in the DNF formula, such that if the term evaluates to 0 then the input to the neuron is at most -1 , and otherwise it is at least 2 . Note that the network N 1 depends on x . However, only the second layer depends on x , and thus given an input we may compute the first layer even without knowing x . Let N ′ 1 : R kn → R be a depth3 network with no activation in the output neuron, obtained from N 1 by summing the outputs from all neurons E 1 .

Let N 2 be a depth2 neural network with input dimension kn , at most n log( n ) hidden neurons, at most 2 n output neurons, and parameter magnitudes bounded by n 3 (for a sufficiently large n ), which satisfies the following. We denote the set of output neurons of N 2 by E 2 . Let z ′ ∈ R kn be an input to N 2 such that for every i ∈ [ kn ] we have z ′ i ̸∈ ( c, c + 1 n 2 ) . If Ψ( z ′ ) is an encoding of a hyperedge then the inputs to all output neurons E 2 are at most -1 , and otherwise there exists a neuron in E 2 with input at least 2 . The construction of the network N 2 is given in Lemma A.5. Intuitively, each neuron in E 2 is responsible for checking whether Ψ( z ′ ) violates some requirement that must hold in an encoding of a hyperedge. Let N ′ 2 : R kn → R be a depth3 network with no activation in the output neuron, obtained from N 2 by summing the outputs from all neurons E 2 .

Let N 3 be a depth2 neural network with input dimension kn , at most n log( n ) hidden neurons, kn ≤ n log( n ) output neurons, and parameter magnitudes bounded by n 3 (for a sufficiently large n ), which satisfies the following. We denote the set of output neurons of N 3 by E 3 . Let z ′ ∈ R kn be an input to N 3 . If there exists i ∈ [ kn ] such that z ′ i ∈ ( c, c + 1 n 2 ) then there exists a neuron in E 3 with input at least 2 . Moreover, if for all i ∈ [ kn ] we have z ′ i ̸∈ ( c -1 n 2 , c + 2 n 2 ) then the inputs to all neurons in E 3 are at most -1 . The construction of the network N 3 is straightforward and given in Lemma A.6. Let N ′ 3 : R kn → R be a depth3 neural network with no activation function in the output neuron, obtained from N 3 by summing the outputs from all neurons E 3 .

Let N ′ : R kn → R be a depth3 network obtained from N ′ 1 , N ′ 2 , N ′ 3 as follows. For z ′ ∈ R kn we have N ′ ( z ′ ) = [1 -N ′ 1 ( z ′ ) -N ′ 2 ( z ′ ) -N ′ 3 ( z ′ )] + . The network N ′ has at most n 2 neurons, and parameter magnitudes bounded by n 3 (all bounds are for a sufficiently large n ). Finally, let ˜ N : R n 2 → R be a depth3 neural network such that ˜ N (˜ z ) = N ′ ( ˜ z [ kn ] ) .

4.2 Defining the noise magnitude τ and analyzing the perturbed network

In order to use the algorithm L w.r.t. some neural network with parameters θ , we need to implement an examples oracle, such that the examples are labeled according to a neural network with parameters θ + ξ , where ξ is a random perturbation. Specifically, we use L with an examples oracle where the

labels correspond to a network ˆ N : R n 2 → R , obtained from ˜ N (w.r.t. an appropriate x ∈ { 0 , 1 } n in the construction of N 1 ) by adding a small perturbation to the parameters. The perturbation is such that we add i.i.d. noise to each parameter in ˜ N , where the noise is distributed according to N (0 , τ 2 ) , and τ = 1 / poly( n ) is small enough such that the following holds. Let f θ : R n 2 → R be any depth3 neural network parameterized by θ ∈ R r for some r > 0 with at most n 2 neurons, and parameter magnitudes bounded by n 3 (note that r is polynomial in n ). Then w.p. at least 1 -1 n over ξ ∼ N ( 0 , τ 2 I r ) , we have | ξ i | ≤ 1 10 for all i ∈ [ r ] , and the network f θ + ξ is such that for every input ˜ z ∈ R n 2 with ∥ ˜ z ∥ ≤ 2 n and every neuron we have: Let a, b be the inputs to the neuron in the computations f θ (˜ z ) and f θ + ξ (˜ z ) (respectively), then | a -b | ≤ 1 2 . Thus, τ is sufficiently small, such that w.h.p. adding i.i.d. noise N (0 , τ 2 ) to each parameter does not change the inputs to the neurons by more than 1 2 . Note that such an inverse-polynomial τ exists, since when the network size, parameter magnitudes, and input size are bounded by some poly( n ) , then the input to each neuron in f θ (˜ z ) is poly( n ) -Lipschitz as a function of θ , and thus it suffices to choose τ that implies w.p. at least 1 -1 n that ∥ ξ ∥ ≤ 1 q ( n ) for a sufficiently large polynomial q ( n ) (see Lemma A.7 for details).

Let ˜ θ ∈ R p be the parameters of the network ˜ N . Recall that the parameters vector ˜ θ is the concatenation of all weight matrices and bias terms. Let ˆ θ ∈ R p be the parameters of ˆ N , namely, ˆ θ = ˜ θ + ξ where ξ ∼ N ( 0 , τ 2 I p ) . By our choice of τ and the construction of the networks N 1 , N 2 , N 3 , w.p. at least 1 -1 n over ξ , for every ˜ z with ∥ ˜ z ∥ ≤ 2 n , the inputs to the neurons E 1 , E 2 , E 3 in the computation ˆ N (˜ z ) satisfy the following properties, where we denote z ′ = ˜ z [ kn ] :

(P1) If Ψ( z ′ ) = z S for some hyperedge S , and for every i ∈ [ kn ] we have z ′ i ̸∈ ( c, c + 1 n 2 ) , then the inputs to E 1 satisfy: If P x ( z S ) = 0 then the inputs to all neurons in E 1 are at most -1 2 , and if P x ( z S ) = 1 then there exists a neuron in E 1 with input at least 3 2 .

(P2) If for every i ∈ [ kn ] we have z ′ i ̸∈ ( c, c + 1 n 2 ) , then the inputs to E 2 satisfy: If Ψ( z ′ ) is an encoding of a hyperedge then the inputs to all neurons E 2 are at most -1 2 , and otherwise there exists a neuron in E 2 with input at least 3 2 .

(P3) The inputs to E 3 satisfy: If there exists i ∈ [ kn ] such that z ′ i ∈ ( c, c + 1 n 2 ) then there exists a neuron in E 3 with input at least 3 2 , and if for all i ∈ [ kn ] we have z ′ i ̸∈ ( c -1 n 2 , c + 2 n 2 ) then the inputs to all neurons in E 3 are at most -1 2 .

4.3 Stating the algorithm A

Given a sequence ( S 1 , y 1 ) , . . . , ( S n s , y n s ) , where S 1 , . . . , S n s are i.i.d. random hyperedges, the algorithm A needs to distinguish whether y = ( y 1 , . . . , y n s ) is random or that we have y = ( P ( x S 1 ) , . . . , P ( x S n s )) = ( P x ( z S 1 ) , . . . , P x ( z S n s )) for a random x ∈ { 0 , 1 } n . We let S = (( z S 1 , y 1 ) , . . . , ( z S n s , y n s )) .

1

We use the efficient algorithm L in order to obtain distinguishing advantage greater than 3 as follows. Let ξ be a random perturbation, and let ˆ N be the perturbed network as defined above, w.r.t. the unknown x ∈ { 0 , 1 } n . Note that given a perturbation ξ , only the weights in the second layer of the subnetwork N 1 in ˆ N are unknown, since all other parameters do not depend on x . The algorithm A runs L with the following examples oracle. In the i -th call, the oracle first draws z ∈ { 0 , 1 } kn such that each component is drawn i.i.d. from a Bernoulli distribution which takes the value 0 w.p. 1 n . If z is an encoding of a hyperedge then the oracle replaces z with z S i . Then, the oracle chooses z ′ ∈ R kn such that for each component j , if z j = 1 then z ′ j is drawn from µ + , and otherwise z ′ j is drawn from µ -. Let ˜ z ∈ R n 2 be such that ˜ z [ kn ] = z ′ , and the other n 2 -kn components of ˜ z are drawn i.i.d. from N (0 , 1) . Note that the vector ˜ z has the distribution D , due to the definitions of the densities µ + and µ -, and since replacing an encoding of a random hyperedge by an encoding of another random hyperedge does not change the distribution of z . Let ˆ b ∈ R be the bias term of the output neuron of ˆ N . The oracle returns (˜ z , ˜ y ) , where the labels ˜ y are chosen as follows:

If Ψ( z ′ ) is not an encoding of a hyperedge, then ˜ y = 0 .

If Ψ( z ′ ) is an encoding of a hyperedge:

-If z ′ does not have components in the interval ( c -1 n 2 , c + 2 n 2 ) , then if y i = 0 we set ˜ y = ˆ b , and if y i = 1 we set ˜ y = 0 .

-If z ′ has a component in the interval ( c, c + 1 n 2 ) , then ˜ y = 0 .

-If z ′ does not have components in the interval ( c, c + 1 n 2 ) , but has a component in the interval ( c -1 n 2 , c + 2 n 2 ) , then the label ˜ y is determined as follows: If y i = 1 then ˜ y = 0 . If y i = 0 , we have: Let ˆ N 3 be the network ˆ N after omitting the neurons E 1 , E 2 and their incoming and outgoing weights. Then, we set ˜ y = [ ˆ b -ˆ N 3 (˜ z )] + . Note that since only the second layer of N 1 depends on x , then we can compute ˆ N 3 (˜ z ) without knowing x .

Let h be the hypothesis returned by L . Recall that L uses at most m ( n ) examples, and hence S contains at least n 3 examples that L cannot view. We denote the indices of these examples by I = { m ( n ) + 1 , . . . , m ( n ) + n 3 } , and the examples by S I = { ( z S i , y i ) } i ∈ I . By n 3 additional calls to the oracle, the algorithm A obtains the examples ˜ S I = { (˜ z i , ˜ y i ) } i ∈ I that correspond to S I . Let h ′ be a hypothesis such that for all ˜ z ∈ R n 2 we have h ′ (˜ z ) = max { 0 , min { ˆ b, h (˜ z ) }} , thus, for ˆ b ≥ 0 the hypothesis h ′ is obtained from h by clipping the output to the interval [0 , ˆ b ] . Let ℓ I ( h ′ ) = 1 | I | ∑ i ∈ I ( h ′ (˜ z i ) -˜ y i ) 2 . Now, if ℓ I ( h ′ ) ≤ 2 n , then A returns 1 , and otherwise it returns 0 . We remark that the decision of our algorithm is based on h ′ (rather than h ) since we need the outputs to be bounded, in order to allow using Hoeffding's inequality in our analysis, which we discuss in the next subsection.

4.4 Analyzing the algorithm A

Note that the algorithm A runs in poly( n ) time. We now show that if S is pseudorandom then A returns 1 w.p. greater than 2 3 , and if S is random then A returns 1 w.p. less than 1 3 .

We start with the case where S is pseudorandom. In Lemma A.8, we prove that if S is pseudorandom then w.h.p. (over ξ ∼ N ( 0 , τ 2 I p ) and the i.i.d. inputs ˜ z i ∼ D ) the examples (˜ z 1 , ˜ y 1 ) , . . . , (˜ z m ( n )+ n 3 , ˜ y m ( n )+ n 3 ) returned by the oracle are realized by ˆ N . Thus, ˜ y i = ˆ N (˜ z i ) for all i . As we show in the lemma, this claim follows by considering the definition of the oracle for the different cases of (˜ z i ) [ kn ] , and using Properties (P1)-(P3) to show that ˆ N behaves similarly.

Recall that the algorithm L is such that w.p. at least 3 4 (over ξ ∼ N ( 0 , τ 2 I p ) , the i.i.d. inputs ˜ z i ∼ D , and its internal randomness), given a sizem ( n ) dataset labeled by ˆ N , it returns a hypothesis h such that E ˜ z ∼D [ ( h (˜ z ) -ˆ N (˜ z )) 2 ] ≤ 1 n . By the definition of h ′ and the construction of ˆ N , if h has small error then h ′ also has small error, namely, we have E ˜ z ∼D [ ( h ′ (˜ z ) -ˆ N (˜ z )) 2 ] ≤ 1 n . In Lemma A.9 we use the above arguments and Hoeffding's inequality over ˜ S I , and prove that w.p. greater than 2 3 we have ℓ I ( h ′ ) ≤ 2 n .

Next, we consider the case where S is random. Let ˜ Z ⊆ R n 2 be such that ˜ z ∈ ˜ Z iff ˜ z [ kn ] does not have components in the interval ( c -1 n 2 , c + 2 n 2 ) , and Ψ(˜ z [ kn ] ) = z S for a hyperedge S . If S is random, then by the definition of our examples oracle, for every i ∈ [ m ( n ) + n 3 ] such that ˜ z i ∈ ˜ Z , we have ˜ y i = ˆ b w.p. 1 2 and ˜ y i = 0 otherwise. Also, by the definition of the oracle, ˜ y i is independent of S i and independent of the choice of the vector ˜ z i that corresponds to z S i . Hence, for such ˜ z i ∈ ˜ Z with i ∈ I , any hypothesis cannot predict the label ˜ y i , and the expected loss for the example is at least ( ˆ b 2 ) 2 . Moreover, in Lemma A.11 we show that Pr [ ˜ z i ∈ ˜ Z ] ≥ 1 2 log( n ) for a sufficiently large n . In Lemma A.12 we use these arguments to prove a lower bound on E ˜ S I [ ℓ I ( h ′ )] , and by Hoeffding's inequality over ˜ S I we conclude that w.p. greater than 2 3 we have ℓ I ( h ′ ) > 2 n .

Overall, if S is pseudorandom then w.p. greater than 2 3 the algorithm A returns 1 , and if S is random then w.p. greater than 2 3 it returns 0 . Thus, the distinguishing advantage is greater than 1 3 .

5 Discussion

Understanding the computational complexity of learning neural networks is a central question in learning theory. Our results imply that the assumptions which allow for efficient learning in onehidden-layer networks might not suffice in deeper networks. Also, in depth2 networks we show that it is not sufficient to assume that both the parameters and the inputs are smoothed. We hope that our hardness results will help focus on assumptions that may allow for efficient learning.

We emphasize that our hardness results are for neural networks that include the ReLU activation also in the output neuron. In contrast, the positive results on learning depth2 networks that we discussed in the introduction do not include activation in the output neuron. Therefore, as far as we are aware, there is still a small gap between the upper bounds and our hardness results: (1) Under the assumption that the input is Gaussian and the weights are non-degenerate, the cases of depth2 networks with activation in the output neuron and of depth3 networks without activation in the output are not settled; (2) In the setting where both the parameters and the input distribution are smoothed, the case of depth2 networks without activation in the output is not settled. These gaps are an intriguing subject for further research.

Acknowledgments and Disclosure of Funding

This work was done as part of the NSF-Simons Sponsored Collaboration on the Theoretical Foundations of Deep Learning.

References

B. Applebaum. Pseudorandom generators with long stretch and low locality from random local one-way functions. SIAM Journal on Computing , 42(5):2008-2037, 2013.

B. Applebaum and S. Lovett. Algebraic attacks against random local functions and their countermeasures. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 1087-1100, 2016.

B. Applebaum and P. Raykov. Fast pseudorandom functions based on expander graphs. In Theory of Cryptography Conference , pages 27-56. Springer, 2016.

B. Applebaum, B. Barak, and A. Wigderson. Public-key cryptography from different assumptions. In Proceedings of the forty-second ACM symposium on Theory of computing , pages 171-180, 2010.

B. Applebaum, I. Damgård, Y. Ishai, M. Nielsen, and L. Zichron. Secure arithmetic computation with constant computational overhead. In Annual International Cryptology Conference , pages 223-254. Springer, 2017.

P. Awasthi, A. Tang, and A. Vijayaraghavan. Efficient algorithms for learning depth-2 neural networks with general relu activations. Advances in Neural Information Processing Systems , 34:13485-13496, 2021.

A. Bakshi, R. Jayaram, and D. P. Woodruff. Learning two layer rectified neural networks in polynomial time. In Conference on Learning Theory , pages 195-268. PMLR, 2019.

A. Bhaskara, M. Charikar, A. Moitra, and A. Vijayaraghavan. Smoothed analysis of tensor decompositions. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing , pages 594-603, 2014.

A. Bhaskara, A. Chen, A. Perreault, and A. Vijayaraghavan. Smoothed analysis in unsupervised learning via decoupling. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS) , pages 582-610. IEEE, 2019.

A. Brutzkus, A. Daniely, and E. Malach. Id3 learns juntas for smoothed product distributions. In Conference on Learning Theory , pages 902-915. PMLR, 2020.

S. Chen, A. Gollakota, A. R. Klivans, and R. Meka. Hardness of noise-free learning for two-hidden-layer neural networks. arXiv preprint arXiv:2202.05258 , 2022.

S. Chen, A. R. Klivans, and R. Meka. Learning deep relu networks is fixed-parameter tractable. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 696-707. IEEE, 2022.

G. Couteau, A. Dupin, P. Méaux, M. Rossi, and Y. Rotella. On the concrete security of goldreich's pseudorandom generator. In International Conference on the Theory and Application of Cryptology and Information Security , pages 96-124. Springer, 2018.

A. Daniely and S. Shalev-Shwartz. Complexity theoretic limitations on learning dnf's. In Conference on Learning Theory , pages 815-830, 2016.

A. Daniely and G. Vardi. Hardness of learning neural networks with natural weights. arXiv preprint arXiv:2006.03177 , 2020.

A. Daniely and G. Vardi. From local pseudorandom generators to hardness of learning. In Conference on Learning Theory , pages 1358-1394. PMLR, 2021.

I. Diakonikolas and D. M. Kane. Small covers for near-zero sets of polynomials and learning latent variable models. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages 184-195. IEEE, 2020.

I. Diakonikolas, D. M. Kane, V. Kontonis, and N. Zarifis. Algorithms and sq lower bounds for pac learning one-hidden-layer relu networks. arXiv preprint arXiv:2006.12476 , 2020.

R. Ge, Q. Huang, and S. M. Kakade. Learning mixtures of gaussians in high dimensions. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , pages 761-770, 2015.

R. Ge, J. D. Lee, and T. Ma. Learning one-hidden-layer neural networks with landscape design. arXiv preprint arXiv:1711.00501 , 2017.

R. Ge, R. Kuditipudi, Z. Li, and X. Wang. Learning two-layer neural networks with symmetric inputs. arXiv preprint arXiv:1810.06793 , 2018.

S. Goel and A. R. Klivans. Learning neural networks with two nonlinear layers in polynomial time. In Conference on Learning Theory , pages 1470-1499. PMLR, 2019.

S. Goel, V. Kanade, A. Klivans, and J. Thaler. Reliably learning the relu in polynomial time. In Conference on Learning Theory , pages 1004-1042. PMLR, 2017.

S. Goel, A. Gollakota, Z. Jin, S. Karmalkar, and A. Klivans. Superpolynomial lower bounds for learning one-layer neural networks using gradient descent. arXiv preprint arXiv:2006.12011 , 2020.

O. Goldreich. Candidate one-way functions based on expander graphs. IACR Cryptol. ePrint Arch. , 2000:63, 2000.

N. Haghtalab, T. Roughgarden, and A. Shetty. Smoothed analysis of online and differentially private learning. Advances in Neural Information Processing Systems , 33:9203-9215, 2020.

Y. Ishai, E. Kushilevitz, R. Ostrovsky, and A. Sahai. Cryptography with constant computational overhead. In Proceedings of the fortieth annual ACM symposium on Theory of computing , pages 433-442, 2008.

M. Janzamin, H. Sedghi, and A. Anandkumar. Beating the perils of non-convexity: Guaranteed training of neural networks using tensor methods. arXiv preprint arXiv:1506.08473 , 2015.

A. T. Kalai and S.-H. Teng. Decision trees are pac-learnable from most product distributions: a smoothed analysis. arXiv preprint arXiv:0812.0933 , 2008.

A. T. Kalai, A. Samorodnitsky, and S.-H. Teng. Learning and smoothed analysis. In 2009 50th Annual IEEE Symposium on Foundations of Computer Science , pages 395-404. IEEE, 2009.

A. R. Klivans and A. A. Sherstov. Cryptographic hardness for learning intersections of halfspaces. In FOCS , 2006.

B. Laurent and P. Massart. Adaptive estimation of a quadratic functional by model selection. Annals of Statistics , pages 1302-1338, 2000.

H. Lin. Indistinguishability obfuscation from constant-degree graded encoding schemes. In Annual International Conference on the Theory and Applications of Cryptographic Techniques , pages 28-57. Springer, 2016.

H. Lin and V. Vaikuntanathan. Indistinguishability obfuscation from ddh-like assumptions on constant-degree graded encodings. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS) , pages 11-20. IEEE, 2016.

P. Méaux, C. Carlet, A. Journault, and F.-X. Standaert. Improved filter permutators: Combining symmetric encryption design, boolean functions, low complexity cryptography, and homomorphic encryption, for private delegation of computations. IACR Cryptol. ePrint Arch. , 2019:483, 2019.

T. Roughgarden. Beyond the worst-case analysis of algorithms . Cambridge University Press, 2021.

A. Sankar, D. A. Spielman, and S.-H. Teng. Smoothed analysis of the condition numbers and growth factors of matrices. SIAM Journal on Matrix Analysis and Applications , 28(2):446-476, 2006.

O. Shamir. Distribution-specific hardness of learning neural networks. The Journal of Machine Learning Research , 19(1):1135-1163, 2018.

L. Song, S. Vempala, J. Wilmes, and B. Xie. On the complexity of learning neural networks. In Advances in neural information processing systems , pages 5514-5522, 2017.

D. A. Spielman and S.-H. Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. Journal of the ACM (JACM) , 51(3):385-463, 2004.

S. Vempala and J. Wilmes. Gradient descent for one-hidden-layer neural networks: Polynomial convergence and sq lower bounds. In Conference on Learning Theory , pages 3115-3117. PMLR, 2019.

Y. Zhang, J. D. Lee, and M. I. Jordan. l1-regularized neural networks are improperly learnable in polynomial time. In International Conference on Machine Learning , pages 993-1001. PMLR, 2016.

K. Zhong, Z. Song, P. Jain, P. L. Bartlett, and I. S. Dhillon. Recovery guarantees for one-hiddenlayer neural networks. In International conference on machine learning , pages 4140-4149. PMLR, 2017.

A Missing lemmas for the proof of Theorem 3.1

Lemma A.1 (Daniely and Vardi [16]) . For every predicate P : { 0 , 1 } k →{ 0 , 1 } and x ∈ { 0 , 1 } n , there is a DNF formula ψ over { 0 , 1 } kn with at most 2 k terms, such that for every hyperedge S we have P x ( z S ) = ψ ( z S ) . Moreover, each term in ψ is a conjunction of positive literals.

Proof. The following proof is from Daniely and Vardi [16], and we give it here for completeness.

We denote by B ⊆ { 0 , 1 } k the set of satisfying assignments of P . Note that the size of B is at most 2 k . Consider the following DNF formula over { 0 , 1 } kn :

̸

̸

For a hyperedge S = ( i , . . . , i ) , we have

̸

Lemma A.2. Let x ∈ { 0 , 1 } n . There exists an affine layer with at most 2 k outputs, weights bounded by a constant and bias terms bounded by n log( n ) (for a sufficiently large n ), such that given an input z S ∈ { 0 , 1 } kn for some hyperedge S , it satisfies the following: For S with P x ( z S ) = 0 all outputs are at most -1 , and for S with P x ( z S ) = 1 there exists an output greater or equal to 2 .

Proof. By Lemma A.1, there exists a DNF formula φ x over { 0 , 1 } kn with at most 2 k terms, such that φ x ( z S ) = P x ( z S ) . Thus, if P x ( z S ) = 0 then all terms in φ x are not satisfied for the input z S , and if P x ( z S ) = 1 then there is at least one term in φ x which is satisfied for the input z S . Therefore, it suffices to construct an affine layer such that for an input z S , the j -th output will be at most -1 if the j -th term of φ x is not satisfied, and at least 2 otherwise. Each term C j in φ x is a conjunction of positive literals. Let I j ⊆ [ kn ] be the indices of these literals. The j -th output of the affine layer will be

Note that if the conjunction C j holds, then this expression is exactly 3 | I j | -3 | I j | + 2 = 2 , and otherwise it is at most 3( | I j | -1) -3 | I j | +2 = -1 . Finally, note that all weights are bounded by 3 and all bias terms are bounded by n log( n ) (for large enough n ).

Lemma A.3. Let x ∈ { 0 , 1 } n . There exists a depth2 neural network N 1 with input dimension kn , 2 kn hidden neurons, at most 2 k output neurons, and parameter magnitudes bounded by n 3 (for a sufficiently large n ), which satisfies the following. We denote the set of output neurons of N 1 by E 1 . Let z ′ ∈ R kn be such that Ψ( z ′ ) = z S for some hyperedge S , and assume that for every i ∈ [ kn ] we have z ′ i ̸∈ ( c, c + 1 n 2 ) . Then, for S with P x ( z S ) = 0 the inputs to all neurons E 1 are at most -1 , and for S with P x ( z S ) = 1 there exists a neuron in E 1 with input at least 2 . Moreover, only the second layer of N 1 depends on x .

Proof. First, we construct a depth2 neural network N Ψ : R kn → [0 , 1] kn with a single layer of nonlinearity, such that for every z ′ ∈ R kn with z ′ i ̸∈ ( c, c + 1 n 2 ) for every i ∈ [ kn ] , we have N Ψ ( z ′ ) = Ψ( z ′ ) . The network N Ψ has 2 kn hidden neurons, and computes N Ψ ( z ′ ) = ( f ( z ′ 1 ) , . . . , f ( z ′ kn )) , where f : R → [0 , 1] is such that

.

̸

Note that if t ≤ c then f ( t ) = 0 , if t ≥ c + 1 n 2 then f ( t ) = 1 , and if c < t < c + 1 n 2 then f ( t ) ∈ (0 , 1) . Also, note that all weights and bias terms can be bounded by n 2 (for large enough n ). Moreover, the network N Ψ does not depend on x .

Let z ′ ∈ R kn such that Ψ( z ′ ) = z S for some hyperedge S , and assume that for every i ∈ [ kn ] we have z ′ i ̸∈ ( c, c + 1 n 2 ) . For such z ′ , we have N Ψ ( z ′ ) = Ψ( z ′ ) = z S . Hence, it suffices to show that we can construct an affine layer with at most 2 k outputs, weights bounded by a constant and bias terms bounded by n 3 , such that given an input z S it satisfies the following: For S with P x ( z S ) = 0 all outputs are at most -1 , and for S with P x ( z S ) = 1 there exists an output greater or equal to 2 . We construct such an affine layer in Lemma A.2.

Lemma A.4. There exists an affine layer with 2 k + n outputs, weights bounded by a constant and bias terms bounded by n log( n ) (for a sufficiently large n ), such that given an input z ∈ { 0 , 1 } kn , if it is an encoding of a hyperedge then all outputs are at most -1 , and otherwise there exists an output greater or equal to 2 .

Proof. Note that z ∈ { 0 , 1 } kn is not an encoding of a hyperedge iff at least one of the following holds:

At least one of the k sizen slices in z contains 0 more than once.

At least one of the k sizen slices in z does not contain 0 .

There are two sizen slices in z that encode the same index.

We define the outputs of our affine layer as follows. First, we have k outputs that correspond to (1). In order to check whether slice i ∈ [ k ] contains 0 more than once, the output will be 3 n -4 -( ∑ j ∈ [ n ] 3 z i,j ) . Second, we have k outputs that correspond to (2): in order to check whether slice i ∈ [ k ] does not contain 0 , the output will be ( ∑ j ∈ [ n ] 3 z i,j ) -3 n + 2 . Finally, we have n outputs that correspond to (3): in order to check whether there are two slices that encode the same index j ∈ [ n ] , the output will be 3 k -4 -( ∑ i ∈ [ k ] 3 z i,j ) . Note that all weights are bounded by 3 and all bias terms are bounded by n log( n ) for large enought n .

Lemma A.5. There exists a depth2 neural network N 2 with input dimension kn , at most 2 kn hidden neurons, 2 k + n output neurons, and parameter magnitudes bounded by n 3 (for a sufficiently large n ), which satisfies the following. We denote the set of output neurons of N 2 by E 2 . Let z ′ ∈ R kn be such that for every i ∈ [ kn ] we have z ′ i ̸∈ ( c, c + 1 n 2 ) . If Ψ( z ′ ) is an encoding of a hyperedge then the inputs to all neurons E 2 are at most -1 , and otherwise there exists a neuron in E 2 with input at least 2 .

Proof. Let N Ψ : R kn → [0 , 1] kn be the depth2 neural network from the proof of Lemma A.3, with a single layer of non-linearity with 2 kn hidden neurons, and parameter magnitudes bounded by n 2 , such that for every z ′ ∈ R kn with z ′ i ̸∈ ( c, c + 1 n 2 ) for every i ∈ [ kn ] , we have N Ψ ( z ′ ) = Ψ( z ′ ) .

Let z ′ ∈ R kn be such that for every i ∈ [ kn ] we have z ′ i ̸∈ ( c, c + 1 n 2 ) . For such z ′ we have N Ψ ( z ′ ) = Ψ( z ′ ) . Hence, it suffices to show that we can construct an affine layer with 2 k + n outputs, weights bounded by a constant and bias terms bounded by n 3 , such that given an input z ∈ { 0 , 1 } kn , if it is an encoding of a hyperedge then all outputs are at most -1 , and otherwise there exists an output greater or equal to 2 . We construct such an affine layer in Lemma A.4.

Lemma A.6. There exists a depth2 neural network N 3 with input dimension kn , at most n log( n ) hidden neurons, kn ≤ n log( n ) output neurons, and parameter magnitudes bounded by n 3 (for a sufficiently large n ), which satisfies the following. We denote the set of output neurons of N 3 by E 3 . Let z ′ ∈ R kn . If there exists i ∈ [ kn ] such that z ′ i ∈ ( c, c + 1 n 2 ) then there exists a neuron in E 3 with input at least 2 . If for all i ∈ [ kn ] we have z ′ i ̸∈ ( c -1 n 2 , c + 2 n 2 ) then the inputs to all neurons in E 3 are at most -1 .

Proof. It suffices to construct a univariate depth2 network f : R → R with one non-linear layer and a constant number of hidden neurons, such that for every input z ′ i ∈ ( c, c + 1 n 2 ) we have f ( z ′ i ) = 2 , and for every z ′ i ̸∈ ( c -1 n 2 , c + 2 n 2 ) we have f ( z ′ i ) = -1 .

We construct f as follows:

Note that all weights and bias terms are bounded by n 3 (for large enough n ).

Lemma A.7. Let q = poly( n ) and r = poly( n ) . Then, there exists τ = 1 poly( n ) such that for a sufficiently large n , with probability at least 1 -exp( -n/ 2) a vector ξ ∼ N ( 0 , τ 2 I r ) satisfies ∥ ξ ∥ ≤ 1 q .

Proof. Let τ = 1 q √ 2 rn . Every component ξ i in ξ has the distribution N (0 , τ 2 ) . By a standard tail bound of the Gaussian distribution, we have for every i ∈ [ r ] and t ≥ 0 that Pr[ ξ i ≥ t ] ≤ 2 exp ( -t 2 2 τ 2 ) . Hence, for t = 1 q √ r , we get

By the union bound, with probability at least 1 -r · 2 e -n , we have

Thus, for a sufficiently large n , with probability at least 1 -exp( -n/ 2) we have ∥ ξ ∥ ≤ 1 q .

Lemma A.8. If S is pseudorandom then with probability at least 39 40 (over ξ ∼ N ( 0 , τ 2 I p ) and the i.i.d. inputs ˜ z i ∼ D ) the examples (˜ z 1 , ˜ y 1 ) , . . . , (˜ z m ( n )+ n 3 , ˜ y m ( n )+ n 3 ) returned by the oracle are realized by ˆ N .

Proof. By our choice of τ , with probability at least 1 -1 n over ξ ∼ N ( 0 , τ 2 I p ) , we have | ξ j | ≤ 1 10 for all j ∈ [ p ] , and for every ˜ z with ∥ ˜ z ∥ ≤ 2 n the inputs to the neurons E 1 , E 2 , E 3 in the computation ˆ N (˜ z ) satisfy Properties (P1) through (P3). We first show that with probability at least 1 -1 n all examples ˜ z 1 , . . . , ˜ z m ( n )+ n 3 satisfy ∥ ˜ z i ∥ ≤ 2 n . Hence, with probability at least 1 -2 n , Properties (P1) through (P3) hold for the computations ˆ N (˜ z i ) for all i ∈ [ m ( n ) + n 3 ] .

Note that ∥ ˜ z i ∥ 2 has the Chi-squared distribution. Since ˜ z i is of dimension n 2 , a concentration bound by Laurent and Massart [32, Lemma 1] implies that for all t > 0 we have

Plugging-in t = n 2 4 , we get

Thus, we have Pr[ ∥ ˜ z i ∥ ≥ 2 n ] ≤ exp ( -n 2 4 ) . By the union bound, with probability at least

(for a sufficiently large n ), all examples (˜ z i , ˜ y i ) satisfy ∥ ˜ z i ∥ ≤ 2 n .

Thus, we showed that with probability at least 1 -2 n ≥ 39 40 (for a sufficiently large n ), we have | ξ j | ≤ 1 10 for all j ∈ [ p ] , and Properties (P1) through (P3) hold for the computations ˆ N (˜ z i ) for all i ∈ [ m ( n ) + n 3 ] . It remains to show that if these properties hold, then the examples (˜ z 1 , ˜ y 1 ) , . . . , (˜ z m ( n )+ n 3 , ˜ y m ( n )+ n 3 ) are realized by ˆ N .

Let i ∈ [ m ( n ) + n 3 ] . For brevity, we denote ˜ z = ˜ z i , ˜ y = ˜ y i , and z ′ = ˜ z [ kn ] . Since | ξ j | ≤ 1 10 for all j ∈ [ p ] , and all incoming weights to the output neuron in ˜ N are -1 , then in ˆ N all incoming weights to the output neuron are in [ -11 10 , -9 10 ] , and the bias term in the output neuron, denoted by ˆ b , is in [ 9 10 , 11 10 ] . Consider the following cases:

If Ψ( z ′ ) is not an encoding of a hyperedge then ˜ y = 0 , and ˆ N (˜ z ) satisfies:

If z ′ does not have components in ( c, c + 1 n ) , then there exists a neuron in E 2 with output at least 3 2 (by Property (P2)).

If z ′ has a component in ( c, c + 1 n ) , then there exists a neuron in E 3 with output at least 3 2 (by Property (P3)).

In both cases, since all incoming weights to the output neuron in ˆ N are in [ -11 10 , -9 10 ] , and ˆ b ∈ [ 9 10 , 11 10 ] , then the input to the output neuron (including the bias term) is at most 11 10 -3 2 · 9 10 < 0 , and thus its output is 0 .

If Ψ( z ′ ) is an encoding of a hyperedge S , then by the definition of the examples oracle we have S = S i . Hence:

-If z ′ does not have components in ( c -1 n 2 , c + 2 n 2 ) , then:

If y i = 0 then the oracle sets ˜ y = ˆ b . Since S is pseudorandom, we have P x ( z S ) = P x ( z S i ) = y i = 0 . Hence, in the computation ˆ N (˜ z ) the inputs to all neurons in E 1 , E 2 , E 3 are at most -1 2 (by Properties (P1), (P2) and (P3)), and thus their outputs are 0 . Therefore, ˆ N (˜ z ) = ˆ b .

If y i = 1 then the oracle sets ˜ y = 0 . Since S is pseudorandom, we have P x ( z S ) = P x ( z S i ) = y i = 1 . Hence, in the computation ˆ N (˜ z ) there exists a neuron in E 1 with output at least 3 2 (by Property (P1)). Since all incoming weights to the output neuron in ˆ N are in [ -11 10 , -9 10 ] , and ˆ b ∈ [ 9 10 , 11 10 ] , then the input to output neuron (including the bias term) is at most 11 10 -3 2 · 9 10 < 0 , and thus its output is 0 .

-If z ′ has a component in ( c, c + 1 n 2 ) , then ˜ y = 0 . Also, in the computation ˆ N (˜ z ) there exists a neuron in E 3 with output at least 3 2 (by Property (P3)). Since all incoming weights to the output neuron in ˆ N are in [ -11 10 , -9 10 ] , and ˆ b ∈ [ 9 10 , 11 10 ] , then the input to output neuron (including the bias term) is at most 11 10 -3 2 · 9 10 < 0 , and thus its output is 0 .

-If z ′ does not have components in the interval ( c, c + 1 n 2 ) , but has a component in the interval ( c -1 n 2 , c + 2 n 2 ) , then:

If y i = 1 the oracle sets ˜ y = 0 . Since S is pseudorandom, we have P x ( z S ) = P x ( z S i ) = y i = 1 . Hence, in the computation ˆ N (˜ z ) there exists a neuron in E 1 with output at least 3 2 (by Property (P1)). Since all incoming weights to the output neuron in ˆ N are in [ -11 10 , -9 10 ] , and ˆ b ∈ [ 9 10 , 11 10 ] , then the input to output neuron (including the bias term) is at most 11 10 -3 2 · 9 10 < 0 , and thus its output is 0 .

If y i = 0 the oracle sets ˜ y = [ ˆ b -ˆ N 3 (˜ z )] + . Since S is pseudorandom, we have P x ( z S ) = P x ( z S i ) = y i = 0 . Therefore, in the computation ˆ N (˜ z ) all neurons in E 1 , E 2 have output 0 (by Properties (P1) and (P2)), and hence their contribution to the output of ˆ N is 0 . Thus, by the definition of ˆ N 3 , we have ˆ N (˜ z ) = [ ˆ b -ˆ N 3 (˜ z )] + .

Lemma A.9. If S is pseudorandom, then for a sufficiently large n , with probability greater than 2 3 we have

Proof. By Lemma A.8, if S is pseudorandom then with probability at least 39 40 (over ξ ∼ N ( 0 , τ 2 I p ) and the i.i.d. inputs ˜ z i ∼ D ) the examples (˜ z 1 , ˜ y 1 ) , . . . , (˜ z m ( n ) , ˜ y m ( n ) ) returned by the oracle are realized by ˆ N . Recall that the algorithm L is such that with probability at least 3 4 (over ξ ∼ N ( 0 , τ 2 I p ) , the i.i.d. inputs ˜ z i ∼ D , and possibly its internal randomness), given a sizem ( n ) dataset labeled by ˆ N , it returns a hypothesis h such that E ˜ z ∼D [ ( h (˜ z ) -ˆ N (˜ z )) 2 ] ≤ 1 n . Hence, with probability at least 3 4 -1 40 the algorithm L returns such a good hypothesis h , given m ( n ) examples labeled by our examples oracle. Indeed, note that L can return a bad hypothesis only if the random choices are either bad for L (when used with realizable examples) or bad for the realizability of the examples returned by our oracle. By the definition of h ′ and the construction of ˆ N , if h has small error then h ′ also has small error, namely,

Let ˆ ℓ I ( h ′ ) = 1 | I | ∑ i ∈ I ( h ′ (˜ z i ) -ˆ N (˜ z i )) 2 . Recall that by our choice of τ we have Pr[ ˆ b > 11 10 ] ≤ 1 n . Since, ( h ′ (˜ z ) -ˆ N (˜ z )) 2 ∈ [0 , ˆ b 2 ] for all ˜ z ∈ R n 2 , by Hoeffding's inequality, we have for a sufficiently large n that

̸

̸

Overall, by the union bound we have with probability at least 1 -( 1 4 + 1 40 + 1 40 + 1 40 ) > 2 3 for sufficiently large n that:

E ˜ S I ˆ ℓ I ( h ′ ) = E ˜ z ∼D [ ( h ′ (˜ z ) -ˆ N (˜ z )) 2 ] ≤ 1 n .

∣ ∣ ∣ ˆ ℓ I ( h ′ ) -E ˜ S I ˆ ℓ I ( h ′ ) ∣ ∣ ∣ ≤ 1 n .

ℓ I ( h ′ ) -ˆ ℓ I ( h ′ ) = 0 .

Combining the above, we get that if S is pseudorandom, then with probability greater than 2 3 we have

Lemma A.10. Let z ∈ { 0 , 1 } kn be a random vector whose components are drawn i.i.d. from a Bernoulli distribution, which takes the value 0 with probability 1 n . Then, for a sufficiently large n , the vector z is an encoding of a hyperedge with probability at least 1 log( n ) .

Proof. The vector z represents a hyperedge iff in each of the k sizen slices in z there is exactly one 0 -bit and each two of the k slices in z encode different indices. Hence,

Since for every x ∈ (0 , 1) we have e -x < 1 -x 2 , then for a sufficiently large n the above is at least

Lemma A.11. Let ˜ z ∈ R n 2 be the vector returned by the oracle. We have

Proof. Let z ′ = ˜ z [ kn ] . We have

We now bound the terms in the above RHS. First, since z ′ has the Gaussian distribution, then its components are drawn i.i.d. from a density function bounded by 1 2 π . Hence, for a sufficiently large n we have

Let z = Ψ( z ′ ) . Note that z is a random vector whose components are drawn i.i.d. from a Bernoulli distribution, where the probability to get 0 is 1 n . By Lemma A.10, z is an encoding of a hyperedge with probability at least 1 log( n ) . Combining it with Eq. (1) and (2), , we get for a sufficiently large n that

as required.

Lemma A.12. If S is random, then for a sufficiently large n with probability larger than 2 3 we have

Proof. Let ˜ Z ⊆ R n 2 be such that ˜ z ∈ ˜ Z iff ˜ z [ kn ] does not have components in the interval ( c -1 n 2 , c + 2 n 2 ) , and Ψ(˜ z [ kn ] ) = z S for a hyperedge S . If S is random, then by the definition of our examples oracle, for every i ∈ [ m ( n ) + n 3 ] such that ˜ z i ∈ ˜ Z , we have ˜ y i = ˆ b with probability 1 2 and ˜ y i = 0 otherwise. Also, by the definition of the oracle, ˜ y i is independent of S i and independent of the choice of the vector ˜ z i that corresponds to z S i . If ˆ b ≥ 9 10 then for a sufficiently large n the

hypothesis h ′ satisfies for each random example (˜ z i , ˜ y i ) ∈ ˜ S I the following

In Lemma A.11, we show that Pr ˜ z i [ ˜ z i ∈ ˜ Z ] ≥ 1 2 log( n ) . Hence,

Thus, if ˆ b ≥ 9 10 then we have

Therefore, for large n we have

Since, ( h ′ (˜ z ) -˜ y ) 2 ∈ [0 , ˆ b 2 ] for all ˜ z , ˜ y returned by the examples oracle, and the examples ˜ z i for i ∈ I are i.i.d., then by Hoeffding's inequality, we have for a sufficiently large n that

Hence, for large enough n , with probability at least 1 -1 8 -1 8 = 3 4 > 2 3 we have both E ˜ S I [ ℓ I ( h ′ )] ≥ 1 20 log( n ) and ∣ ∣ ℓ I ( h ′ ) -E ˜ S I ℓ I ( h ′ ) ∣ ∣ ≤ 1 n , and thus

B Proof of Corollary 3.1

By the proof of Theorem 3.1, under Assumption 2.1, there is no poly( d ) -time algorithm L s that satisfies the following: Let θ ∈ R p be B -bounded parameters of a depth3 network N θ : R d → R , and let τ, ϵ > 0 . Assume that p, B, 1 /ϵ, 1 /τ ≤ poly( d ) , and that the widths of the hidden layers in N θ are d (i.e., the weight matrices are square). Let ξ ∈ N ( 0 , τ 2 I p ) and let ˆ θ = θ + ξ . Then, with probability at least 3 4 -1 1000 , given access to an examples oracle for N ˆ θ , the algorithm L s returns a hypothesis h with E x [ ( h ( x ) -N ˆ θ ) 2 ] ≤ ϵ .

Note that in the above, the requirements from L s are somewhat weaker than in our original definition of learning with smoothed parameters. Indeed, we assume that the widths of the hidden layers are d and the required success probability is only 3 4 -1 1000 (rather than 3 4 ). We now explain why the hardness result holds already under these conditions:

Note that if we change the assumption on the learning algorithm in proof of Theorem 3.1 such that it succeeds with probability at least 3 4 -1 1000 (rather than 3 4 ), then in the case where S is pseudorandom we get that the algorithm A returns 1 with probability at least 1 -( 1 4 + 1 1000 + 1 40 + 1 40 + 1 40 ) (see the proof of Lemma A.9), which is still greater than 2 3 . Also, the analysis of the case where S is random does not change, and thus in this case A returns 0 with probability greater than 2 3 . Consequently, we still get distinguishing advantage greater than 1 3 .

Regarding the requirement on the widths, we note that in the proof of Theorem 3.1 the layers satisfy the following. The input dimension is d = n 2 , the width of the first hidden layer is at most 3 n log( n ) ≤ d , and the width of the second hidden layer is at most log( n ) + 2 n + n log( n ) ≤ d (all bounds are for a sufficiently large d ). In order to get a network where all layers are of width d , we add new neurons to the hidden layers, with incoming weights 0 , outgoing weights 0 , and bias terms -1 . Then, for an appropriate choice of τ = 1 / poly( n ) , even in the perturbed network the outputs of these new neurons will be 0 w.h.p. for every input ˜ z 1 , . . . , ˜ z m ( n )+ n 3 , and thus they will not affect the network's output. Thus, using the same argument as in the proof of Theorem 3.1, we conclude that the hardness results holds already for network with square weight matrices.

Suppose that there exists an efficient algorithm L p that learns in the standard PAC framework depth3 neural networks where the minimal singular value of each weight matrix is lower bounded by 1 /q ( d ) for any polynomial q ( d ) . We will use L p to obtain an efficient algorithm L s that learns depth3 networks with smoothed parameters as described above, and thus reach a contradiction.

Let θ ∈ R p be B -bounded parameters of a depth3 network N θ : R d → R , and let τ, ϵ > 0 . Assume that p, B, 1 /ϵ, 1 /τ ≤ poly( d ) , and that the widths of the hidden layers in N θ are d . For random ξ ∼ N ( 0 , τ 2 I p ) and ˆ θ = θ + ξ , the algorithm L s has access to examples labeled by N ˆ θ . Using Lemma B.1 below with t = τ d and the union bound over the two weight matrices in N θ , we get that with probability at least 1 -2 · 2 . 35 √ d ≥ 1 -1 1000 (for large enough d ), the minimal singular values of all weight matrices in ˆ θ are at least τ d ≥ 1 q ( d ) for some sufficiently large polynomial q ( d ) . Our algorithm L s will simply run L p . Given that the minimal singular values of the weight matrices are at least 1 q ( d ) , the algorithm L p runs in time poly( d ) and returns with probability at least 3 4 a hypothesis h with E x [ ( h ( x ) -N ˆ θ ( x )) 2 ] ≤ ϵ . Overall, the algorithm L s runs in poly( d ) time, and with probability at least 3 4 -1 1000 (over both ξ and the internal randomness) returns a hypothesis h with loss at most ϵ .

Lemma B.1 (Sankar et al. [37], Theorem 3.3) . Let W be an arbitrary square matrix in R d × d , and let P ∈ R d × d be a random matrix, where each entry is drawn i.i.d. from N (0 , τ 2 ) for some τ > 0 . Let σ d be the minimal singular value of the matrix W + P . Then, for every t > 0 we have

C Proof of Theorem 3.2

The proof follows similar ideas to the proof of Theorem 3.1. The main difference is that we need to handle here a smoothed discrete input distribution rather than the standard Gaussian distribution.

For a sufficiently large n , let D be a distribution on { 0 , 1 } n 2 , where each component is drawn i.i.d. from a Bernoulli distribution which takes the value 0 with probability 1 n . Assume that there is a poly( n ) -time algorithm L that learns depth3 neural networks with at most n 2 hidden neurons and parameter magnitudes bounded by n 3 , with smoothed parameters and inputs, under the distribution D , with ϵ = 1 n and τ, ω = 1 / poly( n ) that we will specify later. Let m ( n ) ≤ poly( n ) be the sample complexity of L , namely, L uses a sample of size at most m ( n ) and returns with probability at least 3 4 a hypothesis h with E z ∼ ˆ D [ ( h ( z ) -N ˆ θ ( z ) ) 2 ] ≤ ϵ = 1 n . Note that ˆ D is the distribution D after smoothing with parameter ω , and the vector ˆ θ is the parameters of the target network after smoothing with parameter τ . Let s > 1 be a constant such that n s ≥ m ( n ) + n 3 for every sufficiently large n . By Assumption 2.1, there exists a constant k and a predicate P : { 0 , 1 } k →{ 0 , 1 } , such that F P,n,n s

is 1 3 -PRG. We will show an efficient algorithm A with distinguishing advantage greater than 1 3 and thus reach a contradiction.

Throughout this proof, we will use some notations from the proof of Theorem 3.1. We repeat it here for convenience. For a hyperedge S = ( i 1 , . . . , i k ) we denote by z S ∈ { 0 , 1 } kn the following encoding of S : the vector z S is a concatenation of k vectors in { 0 , 1 } n , such that the j -th vector has 0 in the i j -th coordinate and 1 elsewhere. Thus, z S consists of k sizen slices, each encoding a member of S . For z ∈ { 0 , 1 } kn , i ∈ [ k ] and j ∈ [ n ] , we denote z i,j = z ( i -1) n + j . That is, z i,j is the j -th component in the i -th slice in z . For x ∈ { 0 , 1 } n , let P x : { 0 , 1 } kn →{ 0 , 1 } be such that for every hyperedge S we have P x ( z S ) = P ( x S ) . For ˜ z ∈ R n 2 we denote ˜ z [ kn ] = (˜ z 1 , . . . , ˜ z kn ) , namely, the first kn components of ˜ z (assuming n 2 ≥ kn ).

C.1 Defining the target network for L

Since our goal is to use the algorithm L for breaking PRGs, in this subsection we define a neural network ˜ N : R n 2 → R that we will later use as a target network for L . The network ˜ N contains the subnetworks N 1 , N 2 that we define below.

Let N 1 be a depth1 neural network (i.e., one layer, with activations in the output neurons) with input dimension kn , at most log( n ) output neurons, and parameter magnitudes bounded by n 3 (all bounds are for a sufficiently large n ), which satisfies the following. We denote the set of output neurons of N 1 by E 1 . Let z ′ ∈ { 0 , 1 } kn be an input to N 1 such that z ′ = z S for some hyperedge S . Thus, even though N 1 takes inputs in R kn , we consider now its behavior for an input z ′ with discrete components in { 0 , 1 } . Fix some x ∈ { 0 , 1 } n . Then, for S with P x ( z S ) = 0 the inputs to all output neurons E 1 are at most -1 , and for S with P x ( z S ) = 1 there exists a neuron in E 1 with input at least 2 . Recall that our definition of a neuron's input includes the addition of the bias term. The construction of the network N 1 is given in Lemma A.2. Note that the network N 1 depends on x . Let N ′ 1 : R kn → R be a depth2 neural network with no activation function in the output neuron, obtained from N 1 by summing the outputs from all neurons E 1 .

Let N 2 be a depth1 neural network (i.e., one layer, with activations in the output neurons) with input dimension kn , at most 2 n output neurons, and parameter magnitudes bounded by n 3 (for a sufficiently large n ), which satisfies the following. We denote the set of output neurons of N 2 by E 2 . Let z ′ ∈ { 0 , 1 } kn be an input to N 2 (note that it has components only in { 0 , 1 } ) . If z ′ is an encoding of a hyperedge then the inputs to all output neurons E 2 are at most -1 , and otherwise there exists a neuron in E 2 with input at least 2 . The construction of the network N 2 is given in Lemma A.4. Let N ′ 2 : R kn → R be a depth2 neural network with no activation function in the output neuron, obtained from N 2 by summing the outputs from all neurons E 2 .

Let N ′ : R kn → R be a depth2 network obtained from N ′ 1 , N ′ 2 as follows. For z ′ ∈ R kn we have N ′ ( z ′ ) = [1 -N ′ 1 ( z ′ ) -N ′ 2 ( z ′ )] + . The network N ′ has at most n 2 neurons, and parameter magnitudes bounded by n 3 (all bounds are for a sufficiently large n ). Finally, let ˜ N : R n 2 → R be a depth2 neural network such that ˜ N (˜ z ) = N ′ ( ˜ z [ kn ] ) .

C.2 Defining the noise magnitudes τ, ω and analyzing the perturbed network under perturbed inputs

In order to use the algorithm L w.r.t. some neural network with parameters θ and a certain input distribution, we need to implement an examples oracle, such that the examples are drawn from a smoothed input distribution, and labeled according to a neural network with parameters θ + ξ , where ξ is a random perturbation. Specifically, we use L with an examples oracle where the input distribution ˆ D is obtained from D by smoothing, and the labels correspond to a network ˆ N : R n 2 → R obtained from ˜ N (w.r.t. an appropriate x ∈ { 0 , 1 } n in the construction of N 1 ) by adding a small perturbation to the parameters. The smoothing magnitudes ω, τ of the inputs and the network's parameters (respectively) are such that the following hold.

We first choose the parameter τ = 1 / poly( n ) as follows. Let f θ : R n 2 → R be any depth2 neural network parameterized by θ ∈ R r for some r > 0 with at most n 2 neurons, and parameter magnitudes bounded by n 3 (note that r is polynomial in n ). Then, τ is such that with probability at least 1 -1 n over ξ ∼ N ( 0 , τ 2 I r ) , we have | ξ i | ≤ 1 10 for all i ∈ [ r ] , and the network f θ + ξ is such that

for every input ˜ z ∈ R n 2 with ∥ ˜ z ∥ ≤ n and every neuron we have: Let a, b be the inputs to the neuron in the computations f θ (˜ z ) and f θ + ξ (˜ z ) (respectively), then | a -b | ≤ 1 4 . Thus, τ is sufficiently small, such that w.h.p. adding i.i.d. noise N (0 , τ 2 ) to each parameter does not change the inputs to the neurons by more than 1 4 . Note that such an inverse-polynomial τ exists, since when the network size, parameter magnitudes, and input size are bounded by some poly( n ) , then the input to each neuron in f θ (˜ z ) is poly( n ) -Lipschitz as a function of θ , and thus it suffices to choose τ that implies with probability at least 1 -1 n that ∥ ξ ∥ ≤ 1 q ( n ) for a sufficiently large polynomial q ( n ) (see Lemma A.7 for details).

Next, we choose the parameter ω = 1 / poly( n ) as follows. Let f θ : R n 2 → R be any depth2 neural network parameterized by θ with at most n 2 neurons, and parameter magnitudes bounded by n 3 + 1 10 . Then, ω is such that for every z ∈ { 0 , 1 } n 2 , with probability at least 1 -exp( -n/ 2) over ζ ∼ N ( 0 , ω 2 I n 2 ) the following holds for every neuron in the f θ : Let a, b be the inputs to the neuron in the computations f θ ( z ) and f θ ( z + ζ ) (respectively), then | a -b | ≤ 1 4 . Thus, ω is sufficiently small, such that w.h.p. adding noise N ( 0 , ω 2 I n 2 ) to the input z does not change the inputs to the neurons by more than 1 4 . Note that such an inverse-polynomial ω exists, since when the network size and parameter magnitudes are bounded by some poly( n ) , then the input to each neuron in f θ ( z ) is poly( n ) -Lipschitz as a function of z , and thus it suffices to choose ω that implies with probability at least 1 -exp( -n/ 2) that ∥ ζ ∥ ≤ 1 q ( n ) for a sufficiently large polynomial q ( n ) (see Lemma A.7 for details).

Let ˜ θ ∈ R p be the parameters of the network ˜ N . Recall that the parameters vector ˜ θ is the concatenation of all weight matrices and bias terms. Let ˆ θ ∈ R p be the parameters of ˆ N , namely, ˆ θ = ˜ θ + ξ where ξ ∼ N ( 0 , τ 2 I p ) . By our choice of τ and the construction of the networks N 1 , N 2 , with probability at least 1 -1 n over ξ , for every z ∈ { 0 , 1 } n 2 the following holds: Let ζ ∼ N ( 0 , ω 2 I n 2 ) and let ˆ z = z + ζ . Then with probability at least 1 -exp( -n/ 2) over ζ the differences between inputs to all neurons in the computations ˆ N (ˆ z ) and ˜ N ( z ) are at most 1 2 . Indeed, w.h.p. for all z ∈ { 0 , 1 } n 2 the computations ˜ N ( z ) and ˆ N ( z ) are roughly similar (up to change of 1 / 4 in the input to each neuron), and w.h.p. the computations ˆ N ( z ) and ˆ N (ˆ z ) are roughly similar (up to change of 1 / 4 in the input to each neuron). Thus, with probability at least 1 -1 n over ξ , the network ˆ N is such that for every z ∈ { 0 , 1 } n 2 , we have with probability at least 1 -exp( -n/ 2) over ζ that the computation ˆ N (ˆ z ) satisfies the following properties, where z ′ := z [ kn ] :

(Q1) If z ′ = z S for some hyperedge S , then the inputs to E 1 satisfy:

If P x ( z S ) = 0 the inputs to all neurons in E 1 are at most -1 2 .

If P x ( z S ) = 1 there exists a neuron in E 1 with input at least 3 2 .

(Q2) The inputs to E 2 satisfy:

If z ′ is an encoding of a hyperedge then the inputs to all neurons E 2 are at most -1 2 .

Otherwise, there exists a neuron in E 2 with input at least 3 2 .

C.3 Stating the algorithm A

Given a sequence ( S 1 , y 1 ) , . . . , ( S n s , y n s ) , where S 1 , . . . , S n s are i.i.d. random hyperedges, the algorithm A needs to distinguish whether y = ( y 1 , . . . , y n s ) is random or that y = ( P ( x S 1 ) , . . . , P ( x S n s )) = ( P x ( z S 1 ) , . . . , P x ( z S n s )) for a random x ∈ { 0 , 1 } n . Let S = (( z S 1 , y 1 ) , . . . , ( z S n s , y n s )) .

We use the efficient algorithm L in order to obtain distinguishing advantage greater than 1 3 as follows. Let ξ be a random perturbation, and let ˆ N be the perturbed network as defined above, w.r.t. the unknown x ∈ { 0 , 1 } n . Note that given a perturbation ξ , only the weights in the second layer of the subnetwork N 1 in ˆ N are unknown, since all other parameters do not depend on x . The algorithm A runs L with the following examples oracle. In the i -th call, the oracle first draws z ′ ∈ { 0 , 1 } kn such that each component is drawn i.i.d. from a Bernoulli distribution which takes the value 0 with probability 1 n . If z ′ is an encoding of a hyperedge then the oracle replaces z ′ with z S i . Let z ∈ { 0 , 1 } n 2 be such that z [ kn ] = z ′ , and the other n 2 -kn components of z are drawn i.i.d. from

a Bernoulli distribution which takes the value 0 with probability 1 n . Note that the vector z has the distribution D , since replacing an encoding of a random hyperedge by an encoding of another random hyperedge does not change the distribution of z ′ . Let ˆ z = z + ζ , where ζ ∼ N ( 0 , ω 2 I n 2 ) . Note that ˆ z has the distribution ˆ D . Let ˆ b ∈ R be the bias term of the output neuron of ˆ N . The oracle returns (ˆ z , ˆ y ) , where the labels ˆ y are chosen as follows:

If z ′ is not an encoding of a hyperedge, then ˆ y = 0 .

If z ′ is an encoding of a hyperedge:

-If y i = 0 we set ˆ y = ˆ b .

-If y i = 1 we set ˆ y = 0 .

Let h be the hypothesis returned by L . Recall that L uses at most m ( n ) examples, and hence S contains at least n 3 examples that L cannot view. We denote the indices of these examples by I = { m ( n ) + 1 , . . . , m ( n ) + n 3 } , and the examples by S I = { ( z S i , y i ) } i ∈ I . By n 3 additional calls to the oracle, the algorithm A obtains the examples ˆ S I = { (ˆ z i , ˆ y i ) } i ∈ I that correspond to S I . Let h ′ be a hypothesis such that for all ˜ z ∈ R n 2 we have h ′ (˜ z ) = max { 0 , min { ˆ b, h (˜ z ) }} , thus, for ˆ b ≥ 0 the hypothesis h ′ is obtained from h by clipping the output to the interval [0 , ˆ b ] . Let ℓ I ( h ′ ) = 1 | I | ∑ i ∈ I ( h ′ (ˆ z i ) -ˆ y i ) 2 . Now, if ℓ I ( h ′ ) ≤ 2 n , then A returns 1 , and otherwise it returns 0 . We remark that the decision of our algorithm is based on h ′ (rather than h ) since we need the outputs to be bounded, in order to allow using Hoeffding's inequality in our analysis, which we discuss in the next subsection.

C.4 Analyzing the algorithm A

Note that the algorithm A runs in poly( n ) time. We now show that if S is pseudorandom then A returns 1 with probability greater than 2 3 , and if S is random then A returns 1 with probability less than 1 3 . To that end, we use similar arguments to the proof of Theorem 3.1.

39

In Lemma C.1, we show that if S is pseudorandom then with probability at least 40 (over ξ ∼ N ( 0 , τ 2 I p ) and ζ i ∼ N ( 0 , ω 2 I n 2 ) for all i ∈ [ m ( n )] ) the examples (ˆ z 1 , ˆ y 1 ) , . . . , (ˆ z m ( n ) , ˆ y m ( n ) ) returned by the oracle are realized by ˆ N . Recall that the algorithm L is such that with probability at least 3 4 (over ξ ∼ N ( 0 , τ 2 I p ) , the i.i.d. inputs ˆ z i ∼ ˆ D , and possibly its internal randomness), given a sizem ( n ) dataset labeled by ˆ N , it returns a hypothesis h such that E ˆ z ∼ ˆ D [ ( h (ˆ z ) -ˆ N (ˆ z )) 2 ] ≤ 1 n . Hence, with probability at least 3 4 -1 40 the algorithm L returns such a good hypothesis h , given m ( n ) examples labeled by our examples oracle. Indeed, note that L can return a bad hypothesis only if the random choices are either bad for L (when used with realizable examples) or bad for the realizability of the examples returned by our oracle. By the definition of h ′ and the construction of ˆ N , if h has small error then h ′ also has small error, namely,

Let ˆ ℓ I ( h ′ ) = 1 | I | ∑ i ∈ I ( h ′ (ˆ z i ) -ˆ N (ˆ z i )) 2 . Recall that by our choice of τ we have Pr[ ˆ b > 11 10 ] ≤ 1 n . Since, ( h ′ (ˆ z ) -ˆ N (ˆ z )) 2 ∈ [0 , ˆ b 2 ] for all ˆ z ∈ R n 2 , by Hoeffding's inequality, we have for a sufficiently large n that

̸

̸

Overall, by the union bound we have with probability at least 1 -( 1 4 + 1 40 + 1 40 + 1 40 ) > 2 3 for sufficiently large n that:

ℓ I ( h ′ ) -ˆ ℓ I ( h ′ ) = 0 .

Combining the above, we get that if S is pseudorandom, then with probability greater than 2 3 we have

We now consider the case where S is random. For an example ˆ z i = z i + ζ i returned by the oracle, we denote z ′ i = ( z i ) [ kn ] ∈ { 0 , 1 } kn . Thus, z ′ i is the input that the oracle used before adding the n 2 -kn additional components and adding noise ζ i . Let Z ′ ⊆ { 0 , 1 } kn be such that z ′ ∈ Z ′ iff z ′ = z S for some hyperedge S . If S is random, then by the definition of our examples oracle, for every i ∈ [ m ( n ) + n 3 ] such that z ′ i ∈ Z ′ , we have ˆ y i = ˆ b with probability 1 2 and ˆ y i = 0 otherwise. Also, by the definition of the oracle, ˆ y i is independent of S i , independent of the n 2 -kn additional components that where added, and independent of the noise ζ i ∼ N ( 0 , ω 2 I n 2 ) that corresponds to ˆ z i .

If ˆ b ≥ 9 10 then for a sufficiently large n the hypothesis h ′ satisfies for each random example (ˆ z i , ˆ y i ) ∈ ˆ S I the following:

In Lemma A.10, we show that for a sufficiently large n we have Pr[ z ′ i ∈ Z ′ ] ≥ 1 log( n ) . Hence,

Thus, if ˆ b ≥ 9 10 then we have

Therefore, for large n we have

Since, ( h ′ (ˆ z ) -ˆ y ) 2 ∈ [0 , ˆ b 2 ] for all ˆ z , ˆ y returned by the examples oracle, and the examples ˆ z i for i ∈ I are i.i.d., then by Hoeffding's inequality, we have for a sufficiently large n that

Hence, for large enough n , with probability at least 1 -1 8 -1 8 = 3 4 > 2 3 we have both E ˆ S I [ ℓ I ( h ′ )] ≥ 1 10 log( n ) and ∣ ∣ ∣ ℓ I ( h ′ ) -E ˆ S I ℓ I ( h ′ ) ∣ ∣ ∣ ≤ 1 n , and thus

Overall, if S is pseudorandom then with probability greater than 2 3 the algorithm A returns 1 , and if S is random then with probability greater than 2 3 the algorithm A returns 0 . Thus, the distinguishing advantage is greater than 1 3 . This concludes the proof of the theorem. It remains to prove the deffered lemma on the realizability of the examples returned by the examples oracle:

Lemma C.1. If S is pseudorandom then with probability at least 39 40 over ξ ∼ N ( 0 , τ 2 I p ) and ζ i ∼ N ( 0 , ω 2 I n 2 ) for i ∈ [ m ( n ) + n 3 ] , the examples (ˆ z 1 , ˆ y 1 ) , . . . , (ˆ z m ( n )+ n 3 , ˆ y m ( n )+ n 3 ) returned by the oracle are realized by ˆ N .

Proof. By our choice of τ and ω and the construction of N 1 , N 2 , with probability at least 1 -1 n over ξ ∼ N ( 0 , τ 2 I p ) , we have | ξ j | ≤ 1 10 for all j ∈ [ p ] , and for every z ∈ { 0 , 1 } n 2 the following holds: Let ζ ∼ N ( 0 , ω 2 I n 2 ) and let ˆ z = z + ζ . Then with probability at least 1 -exp( -n/ 2) over ζ the inputs to the neurons E 1 , E 2 in the computation ˆ N (ˆ z ) satisfy Properties (Q1) and (Q2). Hence, with probability at least 1 -1 n -( m ( n ) + n 3 ) exp( -n/ 2) ≥ 1 -2 n (for a sufficiently large n ), | ξ j | ≤ 1 10 for all j ∈ [ p ] , and Properties (Q1) and (Q2) hold for the computations ˆ N (ˆ z i ) for all i ∈ [ m ( n ) + n 3 ] . It remains to show that if | ξ j | ≤ 1 10 for all j ∈ [ p ] and Properties (Q1) and (Q2) hold, then the examples (ˆ z 1 , ˆ y 1 ) , . . . , (ˆ z m ( n )+ n 3 , ˆ y m ( n )+ n 3 ) are realized by ˆ N .

Let i ∈ [ m ( n ) + n 3 ] . We denote ˆ z i = z i + ζ i , namely, the i -th example returned by the oracle was obtained by adding noise ζ i to z i ∈ { 0 , 1 } n 2 . We also denote z ′ i = ( z i ) [ kn ] ∈ { 0 , 1 } kn . Since | ξ j | ≤ 1 10 for all j ∈ [ p ] , and all incoming weights to the output neuron in ˜ N are -1 , then in ˆ N all incoming weights to the output neuron are in [ -11 10 , -9 10 ] , and the bias term in the output neuron, denoted by ˆ b , is in [ 9 10 , 11 10 ] . Consider the following cases:

If z ′ i is not an encoding of a hyperedge then ˆ y i = 0 . Moreover, in the computation ˆ N (ˆ z i ) , there exists a neuron in E 2 with output at least 3 2 (by Property (Q2)) . Since all incoming weights to the output neuron in ˆ N are in [ -11 10 , -9 10 ] , and ˆ b ∈ [ 9 10 , 11 10 ] , then the input to the output neuron (including the bias term) is at most 11 10 -3 2 · 9 10 < 0 , and thus its output is 0 .

If z ′ is an encoding of a hyperedge S , then by the definition of the examples oracle we have S = S i . Hence:

-If y i = 0 then the oracle sets ˆ y i = ˆ b . Since S is pseudorandom, we have P x ( z S ) = P x ( z S i ) = y i = 0 . Hence, in the computation ˆ N (ˆ z i ) the inputs to all neurons in E 1 , E 2 are at most -1 2 (by Properties (Q1) and (Q2)), and thus their outputs are 0 . Therefore, ˆ N (ˆ z i ) = ˆ b .

-If y i = 1 then the oracle sets ˆ y i = 0 . Since S is pseudorandom, we have P x ( z S ) = P x ( z S i ) = y i = 1 . Hence, in the computation ˆ N (ˆ z i ) there exists a neuron in E 1 with output at least 3 2 (by Property (Q1)). Since all incoming weights to the output neuron in ˆ N are in [ -11 10 , -9 10 ] , and ˆ b ∈ [ 9 10 , 11 10 ] , then the input to output neuron (including the bias term) is at most 11 10 -3 2 · 9 10 < 0 , and thus its output is 0 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents results on the nonexistence of efficient learning algorithms for $3$-depth ReLU networks with smoothed parameters and Gaussian input. It also proves that in general, there is no efficient learning algorithm for $2$-depth ReLU networks with smoothed parameters and smoothed inputs (the smoothness is applied to a specially constructed input, which is Bernoulli). The neural networks they analyze, unlike previous works, have a ReLU unit at the single output.\n\nSTRENGTHS:\nThe paper is theoretical in nature and so it is good that it is rigorous in its analysis and seeks to provide enough detail to understand its proofs. The paper presents its results in a logical order.\n\nWEAKNESSES:\nI will first introduce my main concerns, and then other important observations to address. (I must say that I haven’t followed Section 4 with enough detail, though I do have some observations based on what I read.)\n\n1) I am trying to understand the relevance of the learning problem studied by the paper. When talking about learning, from what I can recall, I have basically seen two approaches in the literature: \n\nCase a) When training a neural network, e.g. using SGD, one of the things we are concerned about is the testing performance of the network after being trained for some number of iterations or number of epochs.\n\nCase b) I have seen other works that are concerned on generalization issues in terms of how close is the empirical loss (with respect to samples of the input data) from the population loss (defined by the expected value with respect to the input data distribution), which such closeness depending on different parameters such as the number of samples being used, the width of the network, the depth of the network, etc. \n\nBoth of these cases tackle the issue of how well a neural network learns and generalizes. However, the paper seems to tackle the problem of actually learning a given neural network (see Definitions 2.1 and 2.2), where the true parameter of the neural network is chosen adversarially (and sometimes smoothed afterwards). In the paper, essentially, the learning is done by some algorithm that outputs some hypothesis that seeks to approximate the adversarial neural network. However, how is this relevant in the context of training neural networks and their testing/generalization performance? The type of works in Case a) and  Case b), though theoretical, are trying to understand some practical issues in neural network learning that is relevant to the general ML community, but I don’t see such immediate connection in this paper. In which situations do we care about “learning a neural network” instead of “training a neural network so it learns”? This needs more motivation and clarification. \nAlso, the hypothesis $h$ that is supposed to learn the neural network in Definitions 2.1 and 2.2 seems to belong to any arbitrary class of functions. Is this right? If so, why isn’t this intractable to compute? In my mind, it would make sense for $h$ to belong to the same class of neural networks as defined by $N_\\theta$ --- maybe it is the case in Section 4, but I just don’t know; all I know is that in Section 2 and 3, this is not clear to me.\nCuriously, thinking about Case a) above, the paragraph that starts at line 102 mentions stochastic gradient descent (SGD), which makes me think that previous works have addressed the question of learning in a more practically motivated way, since SGD is used in practice. Can the authors also comment on this? How does it relate to your paper?\n\n2) The paper addresses computational complexity, and from reading the paper it seems that such computation is linked to the algorithm $\\mathcal{L}$. The computation by $\\mathcal{L}$ seems to be related to how many times the algorithm needs to access the oracle in order to compute the final hypothesis $h$. This number of times is, I believe, the sample complexity of the algorithm. So, are we talking about computational complexity or sample complexity in the paper? Is the paper concerned about sample complexity at all? This must be clear on the paper, probably since the beginning of it. As far as I can see, the concept of sample complexity was only referenced in line 251 of Section 4 without much more detail about its relevance. Please, address this, since sample complexity is very important in ML and learning in general.\n\n3) The first part of Section 4 explains how the proof used by the authors is related to [15] and what is the additional challenge that the authors present, i.e., the handling of smoothing. However, besides the first paragraph of Section 4, there is no more indication of whether some of the constructions being used throughout the proof correspond to ones in [15] or not. It would be nice to know which parts of the proof were taken from previous papers that also study learning of neural networks, and which parts weren’t.\n\n4) This comment applies to networks of $k$-depth where $k>2$. When considering the neural network in definition 2.1 and 2.2, as well as in the statement of Theorems 3.1, there is no mention on how the neurons are distributed in the feedforward network. Do all hidden layers have the same width? Are the number of neurons per width irrelevant in terms of learning? Since the paper is concerned with presenting negative results, I guess they consider specific constructions of the networks in order to prove their results. Is that correct? If so, could this be mentioned? In general, could there be some mention about the specific topology of the networks in terms of the widths per layer?\n\n5) Naturally, the paper recognizes that it might be possible to obtain results of efficient learning for different assumptions and topologies, such as not including an activation function at the output. I believe the authors should investigate this case for $3$-depth networks because this will strengthen their paper’s contribution. For example, if efficient learning is demonstrated for $3$-depth networks without activation function at the output, this will be very interesting because it elucidates more the role of the extra activation in efficient learning! I know that theoretical works are hard to do because proofs can be very non-trivial; however, since there is at least one previous work showing the efficiency of learning $2$-depth networks [6], how difficult would it be to adapt their proof to the case of $3$-depth networks?\n\nOther observations:\n\n1) The paper mentions that the neural network architecture they focus on have a ReLU at the single output layer. Previous works seem to focus on linear output, i.e. regression. Is having a ReLU at the output layer closer to real world applications? What is the motivation for it?\n\n2) Definition 2.1 and 2.2 says that $(x,y)$ is drawn iid; however, it seems that only $x$ must be drawn iid, since $y$ is a deterministic function of $x$. Please check.\n\n3) I think a proof outline is needed in Section 4, probably right before subsection 4.1 to better understand how the rest of the proof is built. Though the subsection titles in section 4 indicate what is being done, how they are pieced together in order to better understand the overall proof is not clear to me.\n\nMinor:\n\n1) In the abstract, line 9 mentions the word “hard”. It would be better to instead use expressions in terms of efficiency of the computation, etc.\n\n2) Line 35, it seems that it should say “bounded from below”.\n\n3) In line 86 it says that considering standard Gaussian distributions is “perhaps the most natural choice of an input distribution”, why is that? In real world applications, practitioners don’t care about this type of input. It may be good to insert a better motivation for it.\n\nQUESTIONS:\nPlease, see above, in the section \"Weaknesses\" of this review.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the computational complexity of learning 3-layer neural networks under the standard Gaussian distribution. Specifically, under a standard cryptographic assumption on the existence of local pseudorandom generators, the authors show that there is no poly-time algorithm that can learn 3-layer networks under the smoothed analysis framework. As a corollary, they show learning 3-layer networks is hard even with assuming a lower bound on the smallest singular values of the weight matrices.\n\nSTRENGTHS:\n- It is an interesting question to understand the worst-case hardness of learning neural networks, and under what assumptions learning is possible. Prior work has shown 2-layer networks can be learned under the smoothed analysis framework. The current paper, however, shows that learning 3-layer networks in the smoothed setting is hard, which is quite an interesting result.\n- At a high level the proof appears to be sound, however I admit that I do not work on complexity theory and am unable to verify the proofs fully.\n\nWEAKNESSES:\nThe novelty of the paper seems limited, particularly in light of the related work [11]. [11] shows that learning three-layer networks is hard, under the Learning With Rounding (LWR) assumption. [11] considers three-layer networks with no activation in the last layer, while the current paper considers networks with an activation in the last layer, but this last-layer activation seems like a very minor difference. Can the authors please comment further on the novelty of the current paper in comparison to this prior work?\n\n[11]  S. Chen, A. Gollakota, A. R. Klivans, and R. Meka. Hardness of noise-free learning for two-hidden-layer neural networks. arXiv preprint arXiv:2202.05258, 2022.\n\nQUESTIONS:\n- I find the main text of the paper, in particular section 4, to be quite difficult to understand. The proof of the main theorem is written in a lengthy paragraph format, and as such I think the paper could be improved by organizing the exposition in section 4. Some concrete suggestions are the following:\n    - The authors could add a high-level proof sketch at the beginning of section 4 before diving into the details of the argument. This could be particularly helpful to readers who are not familiar with the argument of [15], and complexity theory more generally (and may make the paper more suitable for a NeurIPS audience).\n    - Key lemmas (such as Lemmas A.4 - A.6) could be stated explicitly in the main text, to make more clear what the various steps in the proof are.\n    - Both the examples oracle and algorithm $\\mathcal{A}$ could be written in algorithm format, to make more clear what exactly they are computing.\n- I am a bit confused about what the point of the examples oracle is in Section 4.3, and how this is being used by the algorithm. Could you please clarify this further?\n- As mentioned in the weaknesses section, the paper would benefit from additional comparison to [11], in particular the novelty and a comparison between the LWR and PRG cryptographic assumptions.\n\n[15] A. Daniely and G. Vardi. From local pseudorandom generators to hardness of learning. In Conference on Learning Theory, pages 1358–1394. PMLR, 2021.\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission studies the classical problem of constructing ReLU-activated neural networks, specifically from the learning point of view. Previous work has established the existence of an efficient learning learning algorithm for learning depth-2 ReLU networks under the Gaussian distribution assuming a non-degenerate weight matrix. The authors provide a lower bound that rules out an extension of this result to depth-3 ReLU networks even under the \"smoothed analysis\" framework, where one assumes the presence of random noise to avoid the use of \"degenerate\" instances in the hardness reduction. As a second result, the authors provide a lower bound that rules out the extension of the aforementioned algorithm on the Gaussian distribution to general smoothed distributions.\n\nSTRENGTHS:\nThe construction of ReLU-activated neural networks is a central topic at NeurIPS, and other lower-bound results on the problem have also appeared at the proceedings. The article is well-written and the results are non-trivial.\n\nWEAKNESSES:\nMy main issue with the submission is that the newly obtained lower bounds seem to only shift the boundaries of intractability by a very small step compared to what was known previously. In particular, previous results of Daniely and Vardi [15] already ruled out (under the same complexity-theoretic assumptions) an efficient learning algorithm for the setting studied in this submission, with the distinction being that their lower bound does not operate in the \"smoothed analysis\" framework. The same work of Daniely and Vardi [15] also established a lower bound that matches the second result in this submission, but once again without the use of the \"smoothed analysis\" framework. In this sense, it seems to me that the main contribution of the submission is showing that previously established lower bounds do not require the use of purely degenerate cases. In combination with the fact that the lower-bound proof itself builds on the approach introduced by Daniely and Vardi [15], I cannot help but feel that the present submission is somewhat incremental in nature. \n\nAlso, a clearer and more to-the-point comparison of the submission's results to those obtained in the previous work(s) of Daniely and Vardi would have been appreciated... but that is only a minor (and perhaps subjective) point.\n\nQUESTIONS:\nN/A; however, the authors are of course welcome to respond to the individual points raised in the review.\n\nLIMITATIONS:\nThe limitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper addresses the complexity of learning neural networks, a
very fundamental problem in learning theory. Previously, some 
complexity and efficient solvability results were known for networks 
of depth 2.  The paper shows that when the depth is increased to
3, the learning problem becomes computationally intractable (under
a hypothesis on the existence of local pseudorandom generators) 
even when one uses the assumptions (e.g., Gaussian input distribution,
non-degeneracy of the weight matrix) that lead to efficient algorithms
for the depth-2 case.  The paper also shows that the learning problem
for depth-2 networks is hard under a smoothed analysis framework
(where both the input distribution and the network parameters are
perturbed).

STRENGTHS:
(a) Understanding the boundary between hard and easy cases of learning
    neural networks is an important problem in learning theory. This is a 
    very nice contribution to that area.
  
   (b) The paper provides a nice summary of previous work which makes
   it easier to understand the context for the contributions.

   (c) The technical results are presented very well.

WEAKNESSES:
This reviewer can't see any weaknesses.

QUESTIONS:
(1) You prove hardness results for depth-3 neural nets. Can the proofs
  be extended (in a simple way) to show that the hardness results also hold for
  depth-k neural nets for any k >= 3?

  (2) Assumption 2.1 (on the existence of local PRGs with certain properties)
  is used in your proofs. You cite references that provide evidence for the
  assumption. It will be useful to the readers if you can include a short discussion
  on the evidence in the paper itself. Are there any consequences if
  Assumption 2.1 does not hold?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-08