Title: Most Neural Networks Are Almost Learnable
Venue: NeurIPS 2023
Accepted: yes
Keywords: learning neural networks

computational complexity

random networks
Abstract: We present a PTAS for learning random constant-depth networks. We show that for any fixed $\epsilon>0$ and depth $i$, there is a poly-time algorithm that for any distribution on $\sqrt{d} \cdot \mathbb{S}^{d-1}$ learns random Xavier networks of depth $i$, up to an additive error of $\epsilon$. The algorithm runs in time and sample complexity of $(\bar{d})^{\mathrm{poly}(\epsilon^{-1})}$, where $\bar d$ is the size of the network. For some  cases of sigmoid and ReLU-like activations the bound can be improved to $(\bar{d})^{\mathrm{polylog}(\epsilon^{-1})}$, resulting in a quasi-poly-time algorithm for learning constant depth random networks.

Most Neural Networks Are Almost Learnable

Abstract

We present a PTAS for learning random constant-depth networks. We show that for any fixed ϵ > 0 and depth i , there is a poly-time algorithm that for any distribution on √ d · S d -1 learns random Xavier networks of depth i , up to an additive error of ϵ . The algorithm runs in time and sample complexity of ( ¯ d ) poly( ϵ -1 ) , where ¯ d is the size of the network. For some cases of sigmoid and ReLU-like activations the bound can be improved to ( ¯ d ) polylog( ϵ -1 ) , resulting in a quasi-poly-time algorithm for learning constant depth random networks.

1 Introduction

One of the greatest mysteries surrounding deep learning is the discrepancy between its phenomenal capabilities in practice and the fact that despite a great deal of research, polynomial-time algorithms for learning deep models are known only for very restrictive cases. Indeed, state of the art results are only capable of dealing with two-layer networks under assumptions on the input distribution and the network's weights. Furthermore, theoretical study shows that even with very naive architectures, learning neural networks is worst-case computationally intractable.

In this paper, we contrast the aforementioned theoretical state of affairs, and show that, perhaps surprisingly, even though constant-depth networks are completely out of reach from a worst-case perspective, most of them are not as hard as one would imagine. That is, they are distribution-free learnable in polynomial time up to any desired constant accuracy. This is the first polynomial-time approximation scheme (PTAS) for learning neural networks of depth greater than 2 (see the related work section for more details). Moreover, we show that the standard SGD algorithm on a ReLU network can be used as a PTAS for learning random networks. The question of whether learning random networks can be done efficiently was posed by Daniely et al. [15], and our work provides a positive result in that respect.

In a bit more detail, we consider constant-depth random networks obtained using the standard Xavier initialization scheme [22, 26], and any input distribution supported on the sphere √ d · S d -1 . For Lipschitz activation functions, our algorithm runs in time ( ¯ d ) poly( ϵ -1 ) , where ¯ d is the network's size including the d input components, and ϵ is the desired accuracy. While this complexity is polynomial for constant ϵ , we also consider the special cases of sigmoid and ReLU-like activations, where the bound can be improved to ( ¯ d ) polylog( ϵ -1 ) .

The main technical idea in our work is that constant-depth random neural networks with Lipschitz activations can be approximated sufficiently well by low-degree polynomials. This result follows by analyzing the network obtained by replacing each activation function with its polynomial approximation using Hermite polynomials. It implies that efficient algorithms for learning polynomials can be

used for learning random neural networks, and specifically that we can use the SGD algorithm on ReLU networks for this task.

1.1 Results

In this work, we show that random fully-connected feedforward neural networks can be wellapproximated by low-degree polynomials, which implies a PTAS for learning random networks. We start by defining the network architecture. We will denote by σ : R → R the activation function, and will assume that it is L -Lipschitz. To simplify the presentation, we will also assume that it is normalized in the sense that E X ∼N (0 , 1) σ 2 ( X ) = 1 . Define ϵ σ ( n ) = min deg( p )= n E X ∼N (0 , 1) ( σ ( X ) -p ( X )) 2 , namely, the error when approximating σ with a degreen polynomial, and note that lim n →∞ ϵ σ ( n ) = 0 . We will consider fully connected networks of depth i and will use d 0 = d to denote the input dimension and d 1 , . . . , d i to denote the number of neurons in each layer. Denote also ¯ d = ∑ i j =0 d j . Given weight matrices

and x ∈ R d 0 we define Ψ 0 ⃗ W ( x ) = x . Then for 1 ≤ j ≤ i we define recursively

We will consider random networks in which the weight matrices are random Xavier matrices [22, 26] . That is, each entry in W j is a centered Gaussian of variance 1 d j -1 . This choice is motivated by the fact that it is a standard practice to initialize the network's weights with Xavier matrices, and furthermore, it ensures that the scale across the network is the same. That is, for any example x and a neuron n , the second moment of the output of n (w.r.t. the choice of ⃗ W ) is 1 .

Our main result shows that Ψ i ⃗ W can be approximated, up to any constant accuracy ϵ , via constant degree polynomials (the constant will depend only on ϵ , the depth i , and the activation σ ). We will consider the input space ˜ S d -1 = { x ∈ R d : ∥ x ∥ = 1 } . Here, and throughout the paper, ∥ x ∥ stands for the normalized Euclidean norm ∥ x ∥ = √ 1 d ∑ d i =1 x 2 i .

Theorem 1.1. For every i and n such that ϵ σ ( n ) ≤ 1 2 there is a constant D = D ( n, i, σ ) such that if d 1 , . . . , d i -1 ≥ D the following holds. For any weights ⃗ W , there is a degree n i -1 polynomial p ⃗ W such that for any distribution D on ˜ S d -1

Furthermore, the coefficients of p ⃗ W are bounded by (2 ¯ d ) 4 n i -1 .

Since constant degree polynomials are learnable in polynomial time, Theorem 1.1 implies a PTAS for learning random networks of constant depth. In fact, as shown in [9], constant degree polynomials with polynomial coefficients are efficiently learnable via SGD on ReLU networks starting from standard Xavier initialization. Thus, this PTAS can be standard SGD on neural networks. To be more specific, for any constant ϵ > 0 there is an algorithm with ( ¯ d ) O ( ( 14( L +1) 3 ϵ ) ( i -1)2 i -1 ) time and sample complexity that is guaranteed to return a hypothesis whose loss is at most ϵ in expectation. For some specific activations, such as the sigmoid σ ( x ) = erf( x ) := 2 √ π ∫ x 0 e -t 2 2 dt , or the ReLUlike activation σ ( x ) = ∫ x 0 erf( t ) + 1 dt we have that ϵ σ ( n ) approaches to 0 exponentially fast (see Lemma A.4 in the appendix). In this case, we get get a quasi-polynomial time and sample complexity of ( ¯ d ) O ( ( log ( 14( L +1) 3 ϵ )) ( i -1) ) .

Corollary 1.2. For every constants ϵ, i and σ there is a constant D , a univariate-polynomial p and a polynomial-time algorithm A such that if d 1 , . . . , d i -1 ≥ D the following holds. For any distribution

D on ˜ S d -1 , if h is the output of A upon seeing p ( d 0 , . . . , d i ) examples from D , then 1

Furthermore, A can be taken to be SGD on a ReLU network starting from a Xavier initialization.

1.2 Related Work

Learning neural networks efficiently. Efficiently learning classes of neural networks has attracted much interest in recent years. Several works established polynomial-time algorithms for learning onehidden-layer neural networks with certain input distributions (such as the Gaussian distribution) under the assumption that the weight matrix of the hidden layer is non-degenerate [27, 34, 19, 20, 5, 32, 4]. For example, Awasthi et al. [4] showed such a result for non-degenerate one-hidden-layer ReLU networks with bias terms under Gaussian inputs, and also concluded that one-hidden-layer networks can be learned efficiently under the smoothed-analysis framework. Efficient algorithms for learning one-hidden-layer ReLU networks with Gaussian inputs were also shown in Diakonikolas et al. [18], Diakonikolas and Kane [17]. These results do not require non-degenerate weight matrices, but they require that the output layer weights are all positive, as well as a sub-linear upper bound on the number of hidden neurons. Chen et al. [8] recently showed an efficient algorithm for learning one-hidden-layer ReLU networks with Gaussian inputs, under the assumption that the number of hidden neurons is a constant. Note that all of the aforementioned works consider only one-hiddenlayer networks. Chen et al. [7] gave an algorithm for learning deeper ReLU networks, whose complexity is polynomial in the input dimension but exponential in the other parameters (such as the number of hidden units, depth, spectral norm of the weight matrices, and Lipschitz constant of the overall network). Finally, several works established algorithms for learning neural networks, whose complexity is exponential unless we impose strong assumptions on the norms of both the inputs and the weights [23, 30, 33, 24].

Hardness of learning neural networks. As we discussed in the previous paragraph, efficient algorithms for learning ReLU networks are known only for depth2 networks and under certain assumptions on both the network weights and the input distribution. The limited progress in learning ReLU networks can be partially understood by an abundance of hardness results.

Learning neural networks without any assumptions on the input distribution or the weights is known to be hard (under cryptographic and average-case hardness assumptions) already for depth2 ReLU networks [28, 3, 11]. For depth3 networks, hardness results were obtained already when the input distribution is Gaussian [13, 6]. All of the aforementioned hardness results are for improper learning, namely, they do not impose any restrictions on the learning algorithm or on the hypothesis that it returns. For statistical query (SQ) algorithms, unconditional superpolynomial lower bounds were obtained for learning depth3 networks with Gaussian inputs [6], and superpolynomial lower bounds for Correlational SQ (CSQ) algorithms were obtained already for learning depth2 networks with Gaussian inputs [25, 18].

The above negative results suggest that assumptions on the input distribution may not suffice for obtaining efficient learning algorithms. Since in one-hidden-layer networks efficient algorithms exist when imposing assumptions on both the input distribution and the weights, a natural question is whether this approach might also work for deeper networks. Recently, Daniely et al. [15] gave a hardness result for improperly learning depth3 ReLU networks under the Gaussian distribution even when the weight matrices are non-degenerate. This result suggests that learning networks of depth larger than 2 might require new approaches and new assumptions. Moreover, [15] showed hardness of learning depth3 networks under the Gaussian distribution even when a small random perturbation is added to the network's parameters, namely, they proved hardness in the smoothed-analysis framework. While adding a small random perturbation to the parameters does not seem to make the problem computationally easier, they posed the question of whether learning random networks, which roughly correspond to adding a large random perturbation, can be done efficiently. The current work gives a positive result in that respect.

Daniely and Vardi [12] studied whether there exist some 'natural' properties of the network's weights that may suffice to allow efficient distribution-free learning, where a 'natural' property is any property

that holds w.h.p. in random networks. More precisely, they considered a setting where the target network is random, an adversary chooses some input distribution (that may depend on the target network), and the learning algorithm needs to learn the random target network under this input distribution. They gave a hardness result for improper learning (within constant accuracy) in this setting. Thus, they showed that learning random networks is hard when the input distribution may depend on the random network. Note that in the current work, we give a positive result in a setting where we first fix an input distribution and then draw a random network. Finally, learning deep random networks was studied in Das et al. [16], Agarwal et al. [1], where the authors showed hardness of learning networks of depth ω (log( d )) in the SQ model.

2 Proof of Theorem 1.1

2.1 Notation

We recall that for vectors x ∈ R d we use the normalized Euclidean norm ∥ x ∥ = √ ∑ d i =1 x 2 i d and take the unit sphere ˜ S d -1 = { x ∈ R d : ∥ x ∥ = 1 } w.r.t. this norm as our instance space. Inner products will also be normalized: for x , y ∈ R d we denote ⟨ x , y ⟩ = ∑ d i =1 x i y i d . For x ∈ R d and a closed set A ⊂ R d we denote d ( x , A ) := min x ′ ∈ A ∥ x -x ′ ∥ . Unless otherwise specified, a random scalar is assumed to be a standard normal, a random vector in R d is assumed to be a centered Gaussian vector with covariance matrix 1 d I , and a random matrix is assumed to be a Xavier matrix. For f : R → R , we denote ∥ f ∥ 2 = E X f 2 ( X ) . We denote the Kronecker delta by δ ij , i.e. δ ij = 1 if i = j and 0 otherwise.

2.2 Some Preliminaries

We will use the Hermite Polynomials [29] which are defined via the following recursion formula.

The Hermite polynomials are the sequence of normalized orthogonal polynomials w.r.t. the standard Gaussian measure. That is, it holds that

More generally, if ( X,Y ) is a Gaussian vector with covariance matrix ( 1 ρ ρ 1 ) then

Let σ = ∑ ∞ i =0 a i h i be the representation of the activation function σ in the basis of the Hermite polynomials. We will also use the dual activation ˆ σ ( ρ ) = ∑ ∞ i =0 a 2 i ρ i as defined in [14]. We note that ˆ σ is defined in [ -1 , 1] and satisfies ˆ σ (1) = ∥ σ ∥ 2 = 1 .

2.3 Defining a Shadow Network

In order to approximate Ψ i ⃗ W via a polynomial, we will use a 'shadow network" that is obtained by replacing the activation σ with a polynomial approximation of it. We will show that for random networks we can approximate each activation sufficiently well with low-degree Hermite polynomials. Recall that σ = ∑ ∞ i =0 a i h i is the representation of σ in the basis of the Hermite polynomials. Define σ n = 1 √ ∑ n i =0 a 2 i ∑ n i =0 a i h i . We have ϵ σ ( n ) = ∑ ∞ i = n +1 a 2 i and hence σ n = 1 √ 1 -ϵ σ ( n ) ∑ n i =0 a i h i .

We will use the fact that and that for even n

We next define a shadow network. For x ∈ R d we let Ψ 0 ,n ⃗ W ( x ) = x . For 1 ≤ j ≤ i we define recursively

for 1 ≤ j ≤ i -1 and Ψ i,n ⃗ W ( x ) = W i Ψ i -1 ,n ⃗ W ( x ) . We will prove the following theorem, which implies Theorem 1.1.

Theorem 2.1. Fix i and let n be large enough so that ϵ σ ( n ) ≤ 1 2 . There is a constant D = D ( n, i, σ ) such that if d 1 , . . . , d i -2 ≥ D then for any x ∈ ˜ S d -1 ,

Since ϵ σ ( n ) is the error in the approximation of a single activation σ with a degreen polynomial, it is natural to expect that the above bound will depend on ϵ σ ( n ) . To see why Theorem 2.1 (together with Lemma A.3 which bounds ϵ σ ( n )) implies Theorem 1.1, note that Φ i,n ⃗ W ( x ) is a polynomial of degree n i -1 . This implies Theorem 1.1, except the requirement that the coefficients of the polynomial are polynomially bounded. To deal with this, define

As we show next lim min( d 1 ,...,d i -1 ) →∞ E ⃗ W ∥ ∥ ∥ Φ i,n ⃗ W ( x ) -˜ Φ i,n ⃗ W ( x ) ∥ ∥ ∥ = 0 . Hence, in the theorem we can replace Φ i,n by ˜ Φ i,n which has polynomially bounded coefficients. See Appendix A.3 and A.4 for the proofs.

Lemma 2.2. For every ϵ and n there is a constant D such that if d 1 , . . . , d i -1 ≥ D then for any x ∈ ˜ S d -1 , E ⃗ W ∥ ∥ Φ i,n ( x ) -˜ Φ i,n ( x ) ∥ ∥ < ϵ .

Lemma 2.3. ˜ Φ i,n ⃗ W computes a polynomial whose sum of coefficients is at most (2 ¯ d ) 4 n i -1 .

2.4 Proof of Theorem 2.1 for depth-two networks

We will first prove Theorem 2.1 for depth2 networks (i.e. for i = 2 ). We will prove Lemma 2.5 below which implies that for every ϵ there is n such that for any x ∈ ˜ S d -1 , E ⃗ W ∥ ∥ ∥ Ψ 1 ,n ⃗ W ( x ) -Ψ 1 ⃗ W ( x ) ∥ ∥ ∥ ≤ ϵ . We will then prove Lemma 2.6, that together with Lemma 2.5

will show that E ⃗ W ∥ ∥ ∥ Φ 2 ,n ⃗ W ( x ) -Φ 2 ⃗ W ( x ) ∥ ∥ ∥ ≤ ϵ , thus proving Theorem 2.1 for i = 2 . We will start however with the following lemma that will be useful throughout (see Appendix A.5 for the proof).

Lemma 2.4. Fix f, g : R → R , x , y ∈ R d 1 and a Xavier matrix W ∈ R d 2 × d 1 . Let ( X,Y ) be a centered Gaussian vector with covariance matrix ( ∥ x ∥ 2 ⟨ x , y ⟩ ⟨ x , y ⟩ ∥ y ∥ 2 ) . Then

Lemma 2.5. Fix x ∈ ˜ S d 1 -1 . Let W ∈ R d 2 × d 1 be a Xavier matrix. Then

Now, the above equals to

Lemma 2.5 implies that E ⃗ W ∥ ∥ ∥ Ψ 1 ,n ⃗ W ( x ) -Ψ 1 ⃗ W ( x ) ∥ ∥ ∥ ≤ √ 2 ϵ σ ( n ) . Thus, given ϵ > 0 , for sufficiently large n , E ⃗ W ∥ ∥ ∥ Ψ 1 ,n ⃗ W ( x ) -Ψ 1 ⃗ W ( x ) ∥ ∥ ∥ ≤ ϵ . The following lemma therefore implies that E ⃗ W ∥ ∥ ∥ Φ 2 ,n ⃗ W ( x ) -Φ 2 ⃗ W ( x ) ∥ ∥ ∥ ≤ √ 2 ϵ σ ( n ) and thus implies Theorem 2.1 for depth two networks.

Lemma 2.6. For any x ∈ ˜ S d -1

2.5 Proof of Theorem 2.1 for General Networks

For x ∈ ˜ R d i -1 we denote Ψ W i ( x ) = σ ( W i x ) and Ψ n W i ( x ) = σ n ( W i x ) . Lemma 2.5 can be roughly phrased as

In order to prove Theorem 2.1 for general networks we will extend it by replacing the strict equality conditions with softer ones. That is, we will show that

This will be enough to prove Theorem 2.1 for general networks. Indeed, the conditions ∥ x ∥ ≈ 1 and ∥ x ′ ∥ ≈ 1 are valid w.h.p. via a simple probabilistic argument. Thus, Eq. (5) implies that

Now, for x ∈ ˜ S d -1 Eq. (6) implies that Ψ W 1 ( x ) ≈ Ψ n W 1 ( x ′ ) . Using Eq. (6) again we get that Ψ W 2 ◦ Ψ W 1 ( x ) ≈ Ψ n W 2 ◦ Ψ n W 1 ( x ′ ) . Using it i -3 more times we we get that Ψ W i -1 ◦· · ·◦ Ψ W 1 ( x ) ≈ Ψ n W i -1 ◦ Ψ n W 1 ( x ′ ) , or in other words that Ψ i -1 ⃗ W ( x ) ≈ Ψ i -1 ,n ⃗ W ( x ) . As we will show ' ≈ ' stands for a sufficiently strong approximation, which guarantees that E ⃗ W ∥ Ψ i -1 ⃗ W ( x ) -Ψ i -1 ,n ⃗ W ( x ) ∥ ≤ ϵ , and hence Lemma 2.6 implies Theorem 2.1.

To prove Eq. (5) we first prove Lemma 2.7 which softens the requirement that x = x ′ . That is, it shows that

The second condition which requires that ∥ x ∥ = ∥ x ′ ∥ = 1 is softened via Lemmas 2.8 and 2.9. Lemma 2.10 then wraps the two softenings together, and shows that Eq. (5) is valid. Finally, in section 2.5.1 we use Lemma 2.10 to prove Theorem 2.1.

Lemma 2.7. Fix x , x + v ∈ ˜ S d 1 -1 with ∥ v ∥ ≤ ϵ . Let W ∈ R d 2 × d 1 be a Xavier matrix. Then

Proof. We have

By Lemma 2.5 we have E W ∥ σ ( W x ) -σ n ( W x ) ∥ ≤ √ 2 ϵ σ ( n ) . It remains to bound E W ∥ σ n ( W x ) -σ n ( W ( x + v )) ∥ . By Lemma 2.4 We have

where ( X,Y ) is a centered Gaussian vector with correlation matrix ( 1 ρ ρ 1 ) for ρ = ⟨ x , x + v ⟩ ≥ 1 -ϵ . Finally, we have

In Lemma A.1 we show that ˆ σ is L 2 -Lipschitz. Hence the above is at most 2 L 2 1 -ϵ σ ( n ) ϵ .

We next give a lemma that allows us to 'almost jointly project' a pair of points x 1 , x 2 ∈ R d on a closed set A ⊂ R d , without expanding the distance too much. See Appendix A.6 for the proof.

Lemma 2.8. Let A ⊂ R d a closed set and fix x 1 , x 2 ∈ R d . There are ˜ x 1 , ˜ x 2 ∈ A such that

Lemma 2.9. Let x , x + v ∈ R d 1 be vectors such that ∥ x ∥ = 1 and ∥ v ∥ ≤ ϵ ≤ 1 . Let W ∈ R d 2 × d 1 be a Xavier matrix. Then

and

Proof. Fix a centered Gaussain vector ( X,Y ) with covariance matrix ( 1 ⟨ x + v , x ⟩ ⟨ x + v , x ⟩ ∥ x + v ∥ 2 ) . Let Z = Y -X . Note that Var ( Z ) ≤ ϵ 2 . By Lemma 2.4 we have

We now prove the second part. In Lemma A.2, we show that | h i ( x ) -h i ( x + y ) | ≤ 2 i max( | x | , | x + y | , 1) i | y | . Therefore,

Hence,

Lemma 2.4 now implies that

Lemma 2.10. Let x , x + v ∈ R d 1 be vectors such that ∥ v ∥ ≤ ϵ , |∥ x ∥ -1 | ≤ δ ≤ 1 / 2 and |∥ x + v ∥ -1 | ≤ δ . Let W ∈ R d 2 × d 1 be a Xavier matrix. Then

Proof. By Lemma 2.8 there are vectors x ′ , v ′ such that ∥ x ′ ∥ = ∥ x ′ + v ′ ∥ = 1 and

Now, we have, by Lemmas 2.7 and 2.9,

2.5.1 Concluding the proof of Theorem 2.1

Define

and

We have

Theorem 2.1 now follows from Lemmas 2.11 and 2.12 below, together with Lemma 2.6.

√

Proof. We will prove the result by induction on i . The case i = 0 is clear as Ψ 0 ⃗ W ( x , δ ) = Ψ 0 ,n ⃗ W ( x , δ ) . Fix i > 0 . For every δ < 1 2 and n we have by Lemma 2.10

Lemma 2.12. Fix i, n, δ and ϵ > 0 . There is a constant D such that if d 1 , . . . , d i -1 ≥ D then

Proof sketch (see Appendix A.7 for the formal proof). Let B i,δ be the event that for some j < i , | 1 -∥ Ψ j ⃗ W ( x ) ∥| > δ or | 1 -∥ Ψ j,n ⃗ W ( x ) ∥| > δ . We have

Similarly,

Now, the lemma follows by proving that E ⃗ W ∥ ∥ ∥ Ψ i ⃗ W ( x ) ∥ ∥ ∥ 2 and E ⃗ W ∥ ∥ ∥ Ψ i,n ⃗ W ( x ) ∥ ∥ ∥ 2 are bounded by a constant (independent of d 0 , . . . , d i ), and that for every δ, ϵ ′ , i and n , there is a constant D such that if d 1 , . . . , d i -1 ≥ D then Pr( B i,δ ) < ϵ ′ .

3 Conclusion and Future work

One of the prominent approaches for explaining the success of neural networks is trying to show that they are capable of learning complex and 'deep' models. So far this approach has relatively limited success. Despite that significant progress has been made to show that neural networks can learn shallow models, so far, neural networks were shown to learn only 'toy' deep models (e.g. [21, 2, 10, 31]). Not only that, but there are almost no known rich families of deep models that are efficiently learnable by some algorithm (not necessarily gradient methods on neural networks). Our paper suggests that random neural networks might be candidate models. To take this approach further, a natural next step, and a central open question that arises from our work, is to show the existence of an algorithm that learns random networks in time that is polynomial both in 1 ϵ and the network size. This question is already open for depth-two ReLU networks with two hidden neurons. We note that as implied by [31], such a result, even for a single neuron, will have to go beyond polynomial approximation of the network, and even more generally, beyond kernel methods.

Our result requires a lower bound D for the network's width, where D is a constant. We conjecture that this requirement can be relaxed, and leave it to future work. Additional open directions are: (i) the analysis of random convolutional networks, (ii) achieving time and sample complexity of ( ¯ d ) O ( ϵ -2 ) for random networks of any constant depth (and not only for depth two), and (iii) finding a PTAS for random networks of depth ω (1) .

Acknowledgments and Disclosure of Funding

The research described in this paper was funded by the European Research Council (ERC) under the European Union's Horizon 2022 research and innovation program (grant agreement No. 101041711), and the Israel Science Foundation (grant number 2258/19). This research was done as part of the NSF-Simons Sponsored Collaboration on the Theoretical Foundations of Deep Learning.

References

N. Agarwal, P. Awasthi, and S. Kale. A deep conditioning treatment of neural networks. In Algorithmic Learning Theory , pages 249-305. PMLR, 2021.

Z. Allen-Zhu and Y. Li. What can resnet learn efficiently, going beyond kernels? arXiv preprint arXiv:1905.10337 , 2019.

B. Applebaum, B. Barak, and A. Wigderson. Public-key cryptography from different assumptions. In Proceedings of the forty-second ACM symposium on Theory of computing , pages 171-180, 2010.

P. Awasthi, A. Tang, and A. Vijayaraghavan. Efficient algorithms for learning depth-2 neural networks with general relu activations. Advances in Neural Information Processing Systems , 34:13485-13496, 2021.

A. Bakshi, R. Jayaram, and D. P. Woodruff. Learning two layer rectified neural networks in polynomial time. In Conference on Learning Theory , pages 195-268. PMLR, 2019.

S. Chen, A. Gollakota, A. R. Klivans, and R. Meka. Hardness of noise-free learning for two-hidden-layer neural networks. arXiv preprint arXiv:2202.05258 , 2022.

S. Chen, A. R. Klivans, and R. Meka. Learning deep relu networks is fixed-parameter tractable. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 696-707. IEEE, 2022.

S. Chen, Z. Dou, S. Goel, A. R. Klivans, and R. Meka. Learning narrow one-hidden-layer relu networks. arXiv preprint arXiv:2304.10524 , 2023.

A. Daniely. Sgd learns the conjugate kernel class of the network. In NIPS , 2017.

A. Daniely and E. Malach. Learning parities with neural networks. In NIPS , 2020.

A. Daniely and S. Shalev-Shwartz. Complexity theoretic limitations on learning dnf's. In Conference on Learning Theory , pages 815-830, 2016.

A. Daniely and G. Vardi. Hardness of learning neural networks with natural weightss. In NIPS , 2020.

A. Daniely and G. Vardi. From local pseudorandom generators to hardness of learning. In Conference on Learning Theory , pages 1358-1394. PMLR, 2021.

A. Daniely, R. Frostig, and Y. Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In NIPS , 2016.

A. Daniely, N. Srebro, and G. Vardi. Computational complexity of learning neural networks: Smoothness and degeneracy. arXiv preprint arXiv:2302.07426 , 2023.

A. Das, S. Gollapudi, R. Kumar, and R. Panigrahy. On the learnability of deep random networks. arXiv preprint arXiv:1904.03866 , 2019.

I. Diakonikolas and D. M. Kane. Small covers for near-zero sets of polynomials and learning latent variable models. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages 184-195. IEEE, 2020.

I. Diakonikolas, D. M. Kane, V. Kontonis, and N. Zarifis. Algorithms and sq lower bounds for pac learning one-hidden-layer relu networks. In Conference on Learning Theory , pages 1514-1539. PMLR, 2020.

R. Ge, J. D. Lee, and T. Ma. Learning one-hidden-layer neural networks with landscape design. arXiv preprint arXiv:1711.00501 , 2017.

R. Ge, R. Kuditipudi, Z. Li, and X. Wang. Learning two-layer neural networks with symmetric inputs. arXiv preprint arXiv:1810.06793 , 2018.

B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. Limitations of lazy training of two-layers neural network. In Advances in Neural Information Processing Systems , pages 9108-9118, 2019.

X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

S. Goel and A. R. Klivans. Learning neural networks with two nonlinear layers in polynomial time. In Conference on Learning Theory , pages 1470-1499. PMLR, 2019.

S. Goel, V. Kanade, A. Klivans, and J. Thaler. Reliably learning the relu in polynomial time. In Conference on Learning Theory , pages 1004-1042. PMLR, 2017.

S. Goel, A. Gollakota, Z. Jin, S. Karmalkar, and A. Klivans. Superpolynomial lower bounds for learning one-layer neural networks using gradient descent. arXiv preprint arXiv:2006.12011 , 2020.

K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

M. Janzamin, H. Sedghi, and A. Anandkumar. Beating the perils of non-convexity: Guaranteed training of neural networks using tensor methods. arXiv preprint arXiv:1506.08473 , 2015.

A. R. Klivans and A. A. Sherstov. Cryptographic hardness for learning intersections of halfspaces. In FOCS , 2006.

R. O'Donnell. Analysis of boolean functions . Cambridge University Press, 2014.

S. Vempala and J. Wilmes. Gradient descent for one-hidden-layer neural networks: Polynomial convergence and sq lower bounds. In Conference on Learning Theory , pages 3115-3117. PMLR, 2019.

G. Yehudai and O. Shamir. On the power and limitations of random features for understanding neural networks. arXiv preprint arXiv:1904.00687 , 2019.

X. Zhang, Y. Yu, L. Wang, and Q. Gu. Learning one-hidden-layer relu networks via gradient descent. In The 22nd international conference on artificial intelligence and statistics , pages 1524-1534. PMLR, 2019.

Y. Zhang, J. D. Lee, and M. I. Jordan. l1-regularized neural networks are improperly learnable in polynomial time. In International Conference on Machine Learning , pages 993-1001. PMLR, 2016.

K. Zhong, Z. Song, P. Jain, P. L. Bartlett, and I. S. Dhillon. Recovery guarantees for one-hiddenlayer neural networks. In International conference on machine learning , pages 4140-4149. PMLR, 2017.

A Missing proofs

A.1 Some Technical Lemmas

Lemma A.1. If σ is L -Lipschitz then ˆ σ is L 2 -Lipschitz in [ -1 , 1]

Proof. As shown in [14], (ˆ σ ) ′ = ̂ σ ′ . Hence, for ρ ∈ [ -1 , 1] ,

Lemma A.2. | h n ( x ) -h n ( x + y ) | ≤ 2 n max( | x | , | x + y | , 1) n | y |

Proof. It is not hard to verify by induction on Eq. (1) that

This implies that for

A.2 Bounds on ϵ σ ( n )

By Eq. (3) if σ is differentiable k times then we have σ ( k ) = ∑ ∞ i = k √ i ! ( i -k )! a i h i -k . Hence, for k ≤ n +1 ,

Lemma A.3. For any L -Lipschitz σ we have ϵ σ ( n ) ≤ L 2 n .

Proof. By Eq. (7) for k = 1 we get

Lemma A.4. For the sigmoid activation σ ( x ) = ∫ x 0 e -t 2 2 dt we have ϵ σ ( n ) ≤ 2 -n .

Proof. We have σ ( k ) ( x ) = ( -1) k -1 √ ( k -1)! h k -1 ( x ) e -x 2 2 . Indeed, it is not hard to verify it for k = 1 and k = 2 . For k > 2 we have via induction that

Hence, | σ ( k ) ( x ) | ≤ | √ ( k -1)! h k -1 ( x ) | , and now Eq. (7) implies that for any k ≤ n +1

ϵ

σ

(

n

)

n

+1

(

n

-

k

+1)!

(

k

-

1)! =

Taking k = ⌈ n +1 2 ⌉ we conclude that ϵ σ ( n ) ≤ 2 -n .

(

)!

(

n

+1

(

n

-

k

+1)!

)!

k

k

!

=

1

n

+1

k

k

(

)

A.3 Proof of Lemma 2.2

Let A be the event that there is an entry in ⃗ W that is greater than ∑ i j =0 d j . We have

Now, it is not hard to verify that E ∥ ∥ ∥ Φ i,n ⃗ W ( x ) ∥ ∥ ∥ 2 is polynomial in ∑ i j =0 d j while Pr( A ) converges to 0 exponentially fast in ∑ i j =0 d j . Thus, if min( d 1 , . . . , d i -1 ) is large enough then E ⃗ W ∥ ∥ ∥ Φ i,n ⃗ W ( x ) -˜ Φ i,n ⃗ W ( x ) ∥ ∥ ∥ < ϵ .

A.4 Proof of Lemma 2.3

We assume that ˜ Φ i,n ⃗ W = Φ i,n ⃗ W , as otherwise ˜ Φ i,n ⃗ W ≡ 0 , in which case the lemma is clear. Write σ n ( x ) = ∑ n k =0 b k x k and h j ( x ) = ∑ j k =0 c j,k x k . Via induction on Eq. (1), we have | c j,k | ≤ 2 j 2 . Hence,

Now, let M j be the maximal sum of coefficients of any polynomial computed by an output neuron of Ψ j,n ⃗ W . We next show by induction that M j ≤ (2 ¯ d ) 2 ∑ j k =1 n k . This will conclude the proof as it will imply that the sum of the coefficients of the polynomial computed by Φ i,n ⃗ W is at most (2 ¯ d ) 2 M i -1 ≤ (2 ¯ d ) 2 ∑ i -1 k =0 n k ≤ (2 ¯ d ) 4 n i -1 . For j = 0 we have M 0 = 1 . For j ≥ 1 we we have

By the induction hypothesis we have

A.5 Proof of Lemma 2.4

We have

Now, the lemma follows from the fact that { (( W x ) j , ( W y ) j ) } d 2 j =1 are independent centered Gaussian vectors with covariance matrix ( ∥ x ∥ 2 ⟨ x , y ⟩ ⟨ x , y ⟩ ∥ y ∥ 2 ) .

A.6 Proof of Lemma 2.8

Let P A : R d → A a function such that for any x ∈ R d , ∥ P A ( x ) -x ∥ = d ( x , A ) . Assume w.l.o.g. that ∥ x 1 -P A ( x 1 ) ∥ ≤ ∥ x 2 -P A ( x 2 ) ∥ .

Simply define ˜ x i = P A ( x i ) . We have

and

Define ˜ x 1 = ˜ x 2 = P A ( x 1 ) . We have

and

A.7 Proof of Lemma 2.12

Let B i,δ be the event that for some j < i , | 1 -∥ Ψ j ⃗ W ( x ) ∥| > δ or | 1 -∥ Ψ j,n ⃗ W ( x ) ∥| > δ . We have

Similarly,

the lemma now follows from the following two claims.

Claim 1. E ⃗ W ∥ ∥ ∥ Ψ i ⃗ W ( x ) ∥ ∥ ∥ 2 and E ⃗ W ∥ ∥ ∥ Ψ i,n ⃗ W ( x ) ∥ ∥ ∥ 2 are bounded by a constant (independent of d 0 , . . . , d i ).

Proof. We have

By induction on i , this implies that E ⃗ W ∥ ∥ ∥ Ψ i ⃗ W ( x ) ∥ ∥ ∥ 2 is bounded by a constant that depends only on i and L (but not on d 1 , . . . , d i ). For E ⃗ W ∥ ∥ ∥ Ψ i,n ⃗ W ( x ) ∥ ∥ ∥ 2 we have

Hence, E W i ∥ ∥ ∥ Ψ i,n ⃗ W ( x ) ∥ ∥ ∥ 2 is an even polynomial in ∥ ∥ ∥ Ψ i -1 ,n ⃗ W ( x ) ∥ ∥ ∥ of degree ≤ 2 n . The polynomial depends only on σ n . It therefore enough to show that for any i and k , E ⃗ W ∥ ∥ ∥ Ψ i,n ⃗ W ( x ) ∥ ∥ ∥ 2 k is bounded,

by a bound that is independent of d 0 , . . . , d i . We will show that via induction on i . For i = 0 this is trivial as ∥ ∥ ∥ Ψ 0 ,n ⃗ W ( x ) ∥ ∥ ∥ 2 k ≡ 1 . Fix i ≥ 1 . We have

The last expression is an even polynomial in ∥ Ψ i -1 ,n ⃗ W ( x ) ∥ . The polynomial depends only on 2 k and n . By the induction hypothesis we conclude that E ⃗ W ∥ ∥ ∥ Ψ i,n ⃗ W ( x ) ∥ ∥ ∥ 2 k is bounded by a bound that is independent from d 0 , . . . , d i .

Claim 2. For every δ, ϵ ′ , i and n , there is a constant D such that if d 1 , . . . , d i -1 ≥ D then Pr( B i,δ ) < ϵ ′ .

Proof. We will prove the lemma by induction on i . For i = 1 this is immediate as Pr( B i,δ ) = 0 . Fix i ≥ 2 . Let δ ′ be small enough so that if |∥ x ∥ -1 | ≤ δ ′ then

and we have

By Chebyshev inequality, Pr( B i,δ | B c i -1 ,δ ′ ) < ϵ ′ 2 for sufficiently large d i -1 . By the induction hypothesis, Pr( B i -1 ,δ ′ ) < ϵ ′ 2 for sufficiently large d 1 , . . . , d i -2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examines the extent to which neural networks learn deep models. Given the fact that there almost do not exist any 'rich' families of constant depth random neural networks that are efficiently learnable by some algorithm, the authors assert the existence of an algorithm that approximates these families of networks that are comprised by Lipschitz activation functions. To achieve this, they substitute each activation function with a low degree polynomial approximation (Hermite polynomial), and then show that the approximation is good enough, allowing for a standard SGD on the neural networks to be a PTAS.\n\nSTRENGTHS:\n-The paper examines the expressivity and learnabilty of random neural networks. This is a fundamental problem in the Theory of Deep Learning and the authors manage to address it by bounding the expected learning time, i.e. the # iterations for determining the weights of the networks approximately (if I undestand correctly Th.1.1.). This allows for a possible impact of the paper in this field. \n\n-From a first reading the results seem new.\n\nWEAKNESSES:\n-The presentation could be improved. The mathematical formulation could be more explained, e.g. with an example of a toy neural network, where the weights and random initialization can be displayed pictorially.\n\n-There could be a clear comparison to the previous approximation attempts (of the literature) to show the novelty in the current PTAS. E.g. in the 1.2. section there is a mention in Chen et al. [7] who provided an algorithm for learning deeper ReLU networks, with complexity polynomial in the input dimension but exponential in the other parameters. The current approach of this paper (from what I understand) considers fixed depth networks and asserts the existence of a polynomial which can be 'close' (approximately) to the learning outcome of the trained neural network: after i iterations, it can approximate it expectedly in 'well' (ideally constantly, as the iterations 'i' tend to infinity). This can be contrasted to previous approaches mentioned in 1.2 about learning depth-3 networks, hardness result of the setting where the target\nnetwork is random and an adversary chooses some input distribution which needs to be learned by the random target network etc.\n\nQUESTIONS:\nPlease refer to the limitations below.\n\nLIMITATIONS:\n-The paper does not attempt to make an experiment as a proof of concept. For example it could try several neural networks of constant depth and Lipschitz activations and vary the rest of the hyperparameters randomly. Then there could be some kind of benchmarking across several  neural networks that measures how good the approximation is, while the learning runs in polynomial time. The running time is also interesting in terms of such reporting. \nAlthough the paper is theoretical, it discusses a potential application of such neural networks (their polynomial approximation). This is the reason why it would be interesting to offer a numerical result of the efficiency, accuracy and complexity of learning a polynomial. \n\n-It seems to be inherently difficult to find the polynomial of Th. 1.1 in order to prove the above point. Is this within the scope of this paper, besides the existence result?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proves that a sufficiently wide neural network which is randomly sampled according to Xavier initialization can be learned in polynomial time (in the network size) up to an additive error $\\epsilon$ if the numbers of layers, the Lipschitz-constant of the activation function, and $\\epsilon$ are fixed constants.\n\nSTRENGTHS:\n- According to the authors, this is the first positive learnability result of this type for arbitrary deep neural networks. This seems to be a major milestone in learning theory for deep networks.\n- The obtained learning algorithm is actually SGD on ReLU networks, so exactly what people use in practice.\n\nWEAKNESSES:\n- The paper is crazily technical already in the main part, not even taking into account the appendix. Within the limited time available for a NeurIPS review, it is impossible to verify mathematical correctness. I tend to believe that this is inherent to the result and not the authors' fault, but I wonder if NeurIPS is actually the right venue to publish such a result. Maybe the authors should submit to a journal instead, where reviewers have enough time to verify mathematical correctness. If submitting to NeurIPS, I think the emphasis should be much more on providing intuitions of the proofs than on technical details.\n- The result is only valid if the network is sufficiently wide enough. Looking through the proof it seems that the required depth $D$ depends with a double-factorial on $n$, which in turn depends on the desired precision $\\epsilon$. This will be astronomically large for any reasonable value of $\\epsilon$. I do not think that any practical neural network can be so wide.\n- Obviously also all other constants seem to be huge / high exponential dependicies on the fixed quantities. But I think this is fine for a theoretical result, it does not concern me as much as the lower bound on the width mentioned above.\n\nMinor comments:\n\n- line 26: I think it would be good to define what you mean by \"ReLU-like\" already here.\n- line 39: you should say that p is a polynomial.\n- line 47: it is confusing to call the neuron n, because n denotes the degree of the polynomials to approximate sigma.\n- line 60: I find this sentence grammatically weird \"Thus, those PTAS can be standard SGD on neural networks\". What do you mean here?\n- line 63: erf not defined.\n- line 65: quasi-polynomial refers to the dependence on epsilon and \\bar{d}, still assuming constant i. Point this out.\n- line 144: what are the a_i in the definition of \\hat(\\sigma)? These are only defined a few lines later, please fix this.\n- line 196: soften -> softened\n\nQUESTIONS:\n- The result seems to assume realizability of the data, that is, it assumes that the ground truth is actually represented by a Xavier-initialized neural network. Did I understand this correctly and can you say anything about the non-realizable setting (where the data can be anything, but you compare yourself only against the error of a best-possible neural network)?\n- How much of a restriction is it to restrict the input data to the boundary of the sphere?\n- For sigmoid and \"ReLU-like\" activation functions, you prove a better dependence on $\\epsilon$. Can you obtain something similar for the actual ReLU activation function? If not, what are the obstacles?\n\nLIMITATIONS:\nGenerally the authors state all mathematical assumptions in a proper way. However, I think two limitations should be discussed earlier / in more detail:\n\n- The issue with the minimum width $D$ (see weaknesses) should be mentioned in the abstract already and pointed out during the introduction. It seems to be an actual important restriction of the validity of the result.\n\n- There is an issue with the sequence of drawing the random objects of the instance, as the authors point out themselves in lines 123 - 125. As previous work shows, the result can only hold with this particular sequence. While this is of course implicitly captured by the sequence of expectations in Thm. 1.1, I think the authors should point this out when they state Thm. 1.1 already, and not only hidden in the related work part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work shows that there is a PTAS for learning a random xavier feedforward network. They show rigorously that the algorithm runs in time and sample complexity $d^{t}$ where $t = poly(\\frac{1}{\\varepsilon})$. This is quite an impressive result since it is a distribution-free result in terms of the input $x$. Furthermore, a corollary of this result is that it shows that SGD can learn these networks.\n\nSTRENGTHS:\n1. This is a technically challenging and impressive result.\n2. Typically, learnability results for Neural Nets are restricted to 2-layer shallow networks. To extend the result to constant depth networks is quite impressive even if they are only random.\n\nWEAKNESSES:\n1. The paper can be better written. While the technical content is good, I feel there could be more intuition and better presentation. Proofs could be moved the appendix and detailed proof sketches could be included.\n2. Since the PTAS is for learning random networks, it's practical value is questionable.\n3. I found certain aspects of the theory hard to understand - I have included them in the review as questions.\n\nQUESTIONS:\n1. The abstract mentions Xavier networks - shouldn't this only be feedforward networks? Or is it easy to extend to other kinds?\n2. I understand that $\\epsilon_{\\sigma}(n)$ refers to the error when approximating the activation by a polynomial of degree $n$. I would like to see a few lines explaining the intuition behind this quantity and why it is important.\n3. The input space is restricted to be **on** the unit sphere. Can this be relaxed to handle $\\lVert x \\rVert \\leq 1$?\n4. Theorem 1.1 is stated as \"For any weights $W$\", but the bound takes an expectation over $W$. Is it on average or \"for any\"?\n5. The notation can be cleaned up. For example in Line 66, consider saying $d^t$ where $t = \\mathcal{O}(dots)$\n6. Why does Corollary 1.2 consider $\\epsilon, i$ as constants? Isn't it just for any $\\epsilon, i$? Is it possible to indicate the dependence on $i$ in the bound?\n7. Is the claim of Corollary 1.2 that SGD can be used to approximate a Xavier network starting from a Xavier network on average? I understand that this is non-trivial, but I am not able to get a good intuition as to why this is important.\n8. I feel line 113-116 provides excellent motivation for the work. I would recommend the authors to highlight it in the introduction.\n9. $\\delta_{ij}$ is the dirac-delta function?\n10. In Lemma 2.7, it says fix $W$ and then takes $\\mathbb{E}_W$. I don't understand this.\n11. Is the application of Jensen's Inequality in line 233 obvious? $\\psi$ isn't convex as a function of all the $W$'s right? Or is the expectation only with respect to the last layer's weights?\n12. I think the work is good, but presentation can be improved. I would recommend moving the proofs to the appendix and highlighting intuition and explaining the result regarding SGD in more detail.\n\nLIMITATIONS:\nI feel the authors should include a more detailed limitations section where they address some of the concerns I have listed above as questions (if relevant).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a polynomial-time approximation scheme (PTAS) for learning random Xavier networks of depth $i$ up to a fixed additive error of $\\epsilon$ with respect to any distribution on the hypersphere. For a fixed $\\epsilon$, the time and sample complexity is polynomial w.r.t. $\\bar{d}$, the total parameter size of the neural network. The work also provides improved bounds for $\\bar{d}^{polylog(\\epsilon^{-1})}$ for special activation functions such as sigmoid.\n\nPrior work has several limitations, mainly that most consider only one hidden-layer-layer networks. The most comparable work is Chen et al. [7] which provides an algorithm for learning deeper than one layer ReLU networks, yet while the complexity is polynomial in $\\bar{d}$, it is exponential in paramaters such as the number of hidden units, depth, spectral norm of the weight matrices, and Lipschitz constant. In fact, there are a plethora of negative results regarding hardness of learning. This work avoids these hardness results by fixing an input distribution and then drawing a random weights of the network.\n\nThe main idea of the paper is to approximate the activation function sufficiently close to the original by replacing it with a polynomial approximation using Hermite polynomials (Theorem 1.1).  The paper calls this a \"shadow network\" and requires that the shadow network's coefficients are polynomially bounded. Given the extensive work on learning polynomials efficiently, the authors can leverage these algorithms, such as SGD on learning polynomial approximations, to obtain a PTAS for learning random networks to a sufficiently small error.\n\nSTRENGTHS:\n- This work studies an important problem of learning deeper-than-two-layers neural networks efficiently. Given the high theoretical relevance of the problem regarding the effectiveness of deep learning, the significance of studying such problem is justified. \n- I find the presentation of the paper quite clear, though it may be helpful in describing prior work and results on learning polynomial approximations using Hermite polynomials. Otherwise, the related work and comparison is to prior work are clear.\n- The main result provides a PTAS that is distribution-free in its input distribution. Prior work did not get such flexible results and this work achieves this in part by incorporating a common initiailization method (random Xavier networks) and studying \"average-case\" results instead of worst-case instances.\n\nWEAKNESSES:\nI do not see any notable weaknesses. I have some questions, which may be seen as weaknesses, that I will list below.\n\nQUESTIONS:\n- Regarding constant $D(n,i,\\sigma)$: It is not entirely clear to me what $D$ explicitly is with respect to the parameters of $n,i,\\sigma$. Since $D$ is a lower bound to the number of neurons in each layer, this quantity seems highly relevant and would be helpful to make explicit in the main explicit theorem (like Theorem 2.1). My question is the following: how does the lower bound $D$ compare to prior work?  How would the results change if you do not have an explicit constant $D$ and assume $D := \\min(d_1, ..., d_{i-1})$? Would such guarantee exist given the current techniques and what would the guarantees be w.r.t. this new $D$?\n- L140: When defining Hermite polynomials, is $\\delta_{ij}$ defined? I think this is missing and should be added to Section 2.1.\n- L144: When defining dual activation, $a_i$ is undefined yet. Maybe reordering the definition so that dual activation comes after defining $\\sigma$ with Hermite polynomials would be clearer.\n\nLIMITATIONS:\nSee Questions. Other than that, the authors are clear with the limitations and comparison to prior work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}