SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Amit Daniely
Hebrew University and Google
amit.daniely@mail.huji.ac.il
Nathan Srebro
TTI-Chicago
nati@ttic.edu
Gal Vardi
TTI-Chicago and Hebrew University
galvardi@ttic.edu
Understanding when neural networks can be learned efficiently is a fundamental question in learning theory. Existing hardness results suggest that assumptions on both the input distribution and the network's weights are necessary for obtaining efficient algorithms. Moreover, it was previously shown that depth-2 networks can be efficiently learned under the assumptions that the input distribution is Gaussian, and the weight matrix is non-degenerate. In this work, we study whether such assumptions may suffice for learning deeper networks and prove negative results. We show that learning depth-3 ReLU networks under the Gaussian input distribution is hard even in the smoothed-analysis framework, where a random noise is added to the network's parameters. It implies that learning depth-3 ReLU networks under the Gaussian distribution is hard even if the weight matrices are non-degenerate. Moreover, we consider depth-2 networks, and show hardness of learning in the smoothed-analysis framework, where both the network parameters and the input distribution are smoothed. Our hardness results are under a well-studied assumption on the existence of local pseudorandom generators.
The computational complexity of learning neural networks has been extensively studied in recent years, and there has been much effort in obtaining both upper bounds and hardness results. Nevertheless, it is still unclear when neural networks can be learned in polynomial time, namely, under what assumptions provably efficient algorithms exist.
Existing results imply hardness already for learning depth-2 ReLU networks in the standard PAC learning framework (e.g., [31, 14]). Thus, without any assumptions on the input distribution or the network's weights, efficient learning algorithms might not be achievable. Even when assuming that the input distribution is Gaussian, strong hardness results were obtained for depth-3 ReLU networks [16, 11], suggesting that assumptions merely on the input distribution might not suffice. Also, a hardness result by Daniely and Vardi [15] shows that strong assumptions merely on the network's weights (without restricting the input distribution) might not suffice even for efficiently learning depth-2 networks. The aforementioned hardness results hold already for improper learning, namely, where the learning algorithm is allowed to return a hypothesis that does not belong to the considered hypothesis class. Thus, a combination of assumptions on the input distribution and the network's weights seems to be necessary for obtaining efficient algorithms.
Several polynomial-time algorithms for learning depth-2 neural networks have been obtained, under assumptions on the input distribution and the network's weights [6, 28, 43, 20, 7, 21]. In these works, it is assumed that the weight matrices are non-degenerate. That is, they either assume that the condition number of the weight matrix is bounded, or some similar non-degeneracy assumption. Specifically, Awasthi et al. [6] gave an efficient algorithm for learning depth-2 (one-hidden-layer) ReLU networks, that may include bias terms in the hidden neurons, under the assumption that the input distribution is Gaussian, and the weight matrix is non-degenerate. The non-degeneracy assumption holds w.h.p. if we add a small random noise to any weight matrix, and hence their result implies efficient learning of depth-2 ReLU networks under the Gaussian distribution in the smoothed-analysis framework.
The positive results on depth-2 networks suggest the following question:
Is there an efficient algorithm for learning ReLU networks of depth larger than 2 under the Gaussian distribution, where the weight matrices are non-degenerate, or in the smoothed-analysis framework where the network's parameters are smoothed?
In this work, we give a negative answer to this question, already for depth-3 networks<sup>1</sup>. We show that learning depth-3 ReLU networks under the Gaussian distribution is hard even in the smoothed-analysis framework, where a random noise is added to the network's parameters. As a corollary, we show that learning depth-3 ReLU networks under the Gaussian distribution is hard even if the weight matrices are non-degenerate. Our hardness results are under a well-studied cryptographic assumption on the existence of local pseudorandom generators (PRG) with polynomial stretch.
Motivated by the existing positive results on smoothed-analysis in depth-2 networks, we also study whether learning depth-2 networks with smoothed parameters can be done under weak assumptions on the input distribution. Specifically, we consider the following question:
Is there an efficient algorithm for learning depth-2 ReLU networks in the smoothed-analysis framework, where both the network's parameters and the input distribution are smoothed?
We give a negative answer to this question, by showing hardness of learning depth-2 ReLU networks where a random noise is added to the network's parameters, and the input distribution on  $\mathbb{R}^d$  is obtained by smoothing an i.i.d. Bernoulli distribution on  $\{0,1\}^d$ . This hardness result is also under the assumption on the existence of local PRGs.
Hardness of learning neural networks. Hardness of improperly learning depth-2 neural networks follows from hardness of improperly learning DNFs or intersections of halfspaces, since these classes can be expressed by depth-2 networks. Klivans and Sherstov [31] showed, assuming the hardness of the shortest vector problem, that learning intersections of halfspaces is hard. Hardness of learning DNF formulas is implied by Applebaum et al. [4] under a combination of two assumptions: the first is related to the planted dense subgraph problem in hypergraphs, and the second is related to local PRGs. Daniely and Shalev-Shwartz [14] showed hardness of learning DNFs under a common assumption, namely, that refuting a random  $K$ -SAT formula is hard. All of the above results are distribution-free, namely, they do not imply hardness of learning neural networks under some specific distribution.
Applebaum and Raykov [3] showed, under an assumption on a specific candidate for Goldreich's PRG (i.e., based on a predicate called XOR-MAJ), that learning depth-3 Boolean circuits under the uniform distribution on the hypercube is hard. Daniely and Vardi [16] proved distribution-specific hardness of learning Boolean circuits of depth-2 (namely, DNFs) and depth-3, under the assumption on the existence of local PRGs that we also use in this work. For DNF formulas, they showed hardness of learning under a distribution where each component is drawn i.i.d. from a Bernoulli distribution (which is non-uniform). For depth-3 Boolean circuits, they showed hardness of learning under the uniform distribution on the hypercube. Since the Boolean circuits can be expressed by ReLU networks of the same depth, these results readily translate to distribution-specific hardness of learning neural networks. Chen et al. [11] showed hardness of learning depth-2 neural networks under the uniform distribution on the hypercube, based on an assumption on the hardness of the
<sup>1</sup>We note that in our results the neural networks have the ReLU activation also in the output neuron.
Learning with Rounding (LWR) problem. Note that the input distributions in the above results are supported on the hypercube, and they do not immediately imply hardness of learning neural networks under continuous distributions.
When considering the computational complexity of learning neural networks, perhaps the most natural choice of an input distribution is the standard Gaussian distribution. Daniely and Vardi [16] established hardness of learning depth-3 networks under this distribution, based on the assumption on the existence of local PRGs. Chen et al. [11] also showed hardness of learning depth-3 networks under the Gaussian distribution, but their result holds already for networks that do not have an activation function in the output neuron, and it is based on the LWR assumption. They also showed hardness of learning constant depth ReLU networks from label queries (i.e., where the learner has the ability to query the value of the target network at any desired input) under the Gaussian distribution, based either on the decisional Diffie-Hellman or the Learning with Errors assumptions.
The above results suggest that assumptions on the input distribution might not suffice for achieving an efficient algorithm for learning depth-3 neural networks. A natural question is whether assumptions on the network weights may suffice. Daniely and Vardi [15] showed (under the assumption that refuting a random  $K$ -SAT formula is hard) that distribution-free learning of depth-2 neural networks is hard already if the weights are drawn from some "natural" distribution or satisfy some "natural" properties. Thus, if we do not impose any assumptions on the input distribution, then even very strong assumptions on the network's weights do not suffice for efficient learning.
Several works in recent years have shown hardness of distribution-specific learning shallow neural networks using gradient-methods or statistical query (SQ) algorithms [38, 39, 41, 24, 18, 11]. It is worth noting that while the SQ framework captures some variants of the gradient-descent algorithm, it does not capture, for example, stochastic gradient-descent (SGD), which examines training points individually (see a discussion in [24]).
We emphasize that none of the above distribution-specific hardness results for neural networks (either for improper learning or SQ learning) holds in the smoothed analysis framework or for non-degenerate weights.
Learning neural networks in polynomial time. Awasthi et al. [6] gave a polynomial-time algorithm for learning depth-2 (one-hidden-layer) ReLU networks, under the assumption that the input distribution is Gaussian, and the weight matrix of the target network is non-degenerate. Their algorithm is based on tensor decomposition, and it can handle bias terms in the hidden layer. Their result also implies that depth-2 ReLU networks with Gaussian inputs can be learned efficiently under the smoothed-analysis framework. Our work implies that such a result might not be possible in depth-3 networks (with activation in the output neuron). Prior to [6], several works gave polynomial time algorithms for learning depth-2 neural networks where the input distribution is Gaussian and the weight matrix is non-degenerate [28, 43, 20, 21, 7], but these works either do not handle the presence of bias terms or do not handle the ReLU activation. Some of the aforementioned works consider networks with multiple outputs, and allow certain non-Gaussian input distributions. Provable guarantees for learning neural networks in super-polynomial time were given in [18, 12, 17, 22, 41, 42, 23].
Notations. We use bold-face letters to denote vectors, e.g.,  $\mathbf{x} = (x_{1},\ldots ,x_{d})$ . For a vector  $\mathbf{x}$  and a sequence  $S = (i_1,\dots ,i_k)$  of  $k$  indices, we let  $\mathbf{x}_S = (x_{i_1},\dots,x_{i_k})$ , i.e., the restriction of  $\mathbf{x}$  to the indices  $S$ . We denote by  $\mathbb{1}[\cdot ]$  the indicator function, for example  $\mathbb{1}[t\geq 5]$  equals 1 if  $t\geq 5$  and 0 otherwise. For an integer  $d\geq 1$  we denote  $[d] = \{1,\dots ,d\}$ . We denote by  $\mathcal{N}(0,\sigma^2)$  the normal distribution with mean 0 and variance  $\sigma^2$ , and by  $\mathcal{N}(\mathbf{0},\Sigma)$  the multivariate normal distribution with mean  $\mathbf{0}$  and covariance matrix  $\Sigma$ . The identity matrix of size  $d$  is denoted by  $I_{d}$ . For  $\mathbf{x}\in \mathbb{R}^{d}$  we denote by  $\| \mathbf{x}\|$  the Euclidean norm. We use  $\mathrm{poly}(a_1,\dots ,a_n)$  to denote a polynomial in  $a_1,\ldots ,a_n$ .
Local pseudorandom generators. An  $(n,m,k)$ -hypergraph is a hypergraph over  $n$  vertices  $[n]$  with  $m$  hyperedges  $S_{1},\ldots ,S_{m}$ , each of cardinality  $k$ . Each hyperedge  $S = (i_1,\dots ,i_k)$  is ordered, and all the  $k$  members of a hyperedge are distinct. We let  $\mathcal{G}_{n,m,k}$  be the distribution over such hypergraphs in which a hypergraph is chosen by picking each hyperedge uniformly and independently at random among all the possible  $n\cdot (n - 1)\cdot \ldots \cdot (n - k + 1)$  ordered hyperedges. Let  $P:\{0,1\} ^k\to$
$\{0,1\}$  be a predicate, and let  $G$  be a  $(n,m,k)$ -hypergraph. We call Goldreich's pseudorandom generator (PRG) [25] the function  $f_{P,G}:\{0,1\}^n\to \{0,1\}^m$  such that for  $\mathbf{x}\in \{0,1\} ^n$ , we have  $f_{P,G}(\mathbf{x}) = (P(\mathbf{x}_{S_1}),\ldots ,P(\mathbf{x}_{S_m}))$ . The integer  $k$  is called the locality of the PRG. If  $k$  is a constant then the PRG and the predicate  $P$  are called local. We say that the PRG has polynomial stretch if  $m = n^s$  for some constant  $s > 1$ . We let  $\mathcal{F}_{P,n,m}$  denote the collection of functions  $f_{P,G}$  where  $G$  is an  $(n,m,k)$ -hypergraph. We sample a function from  $\mathcal{F}_{P,n,m}$  by choosing a random hypergraph  $G$  from  $\mathcal{G}_{n,m,k}$ .
We denote by  $G \stackrel{R}{\leftarrow} \mathcal{G}_{n,m,k}$  the operation of sampling a hypergraph  $G$  from  $\mathcal{G}_{n,m,k}$ , and by  $\mathbf{x} \stackrel{R}{\leftarrow} \{0,1\}^n$  the operation of sampling  $\mathbf{x}$  from the uniform distribution on  $\{0,1\}^n$ . We say that  $\mathcal{F}_{P,n,m}$  is  $\varepsilon$ -pseudorandom generator ( $\varepsilon$ -PRG) if for every polynomial-time probabilistic algorithm  $\mathcal{A}$  the distinguishing advantage
$$
\left| \begin{array}{c} \Pr_ {G \xleftarrow {R} \mathcal {G} _ {n, m, k}, \mathbf {x} \xleftarrow {R} \{0, 1 \} ^ {n}} [ \mathcal {A} (G, f _ {P, G} (\mathbf {x})) = 1 ] - \Pr_ {G \xleftarrow {R} \mathcal {G} _ {n, m, k}, \mathbf {y} \xleftarrow {R} \{0, 1 \} ^ {m}} [ \mathcal {A} (G, \mathbf {y}) = 1 ] \end{array} \right|
$$
is at most  $\varepsilon$ . Thus, the distinguisher  $\mathcal{A}$  is given a random hypergraph  $G$  and a string  $\mathbf{y} \in \{0,1\}^m$ , and its goal is to distinguish between the case where  $\mathbf{y}$  is chosen at random, and the case where  $\mathbf{y}$  is a random image of  $f_{P,G}$ .
Our assumption is that local PRGs with polynomial stretch and constant distinguishing advantage exist:
Assumption 2.1. For every constant  $s > 1$ , there exists a constant  $k$  and a predicate  $P: \{0,1\}^k \to \{0,1\}$ , such that  $\mathcal{F}_{P,n,n^s}$  is  $\frac{1}{3}$ -PRG.
We remark that the same assumption was used by Daniely and Vardi [16] to show hardness-of-learning results. Local PRGs have been extensively studied in the last two decades. In particular, local PRGs with polynomial stretch have shown to have remarkable applications, such as secure-computation with constant computational overhead [27, 5], and general-purpose obfuscation based on constant degree multilinear maps (cf. [33, 34]). Significant evidence for Assumption 2.1 was shown in Applebaum [1]. Moreover, a concrete candidate for a local PRG, based on the XOR-MAJ predicate, was shown to be secure against all known attacks [2, 13, 35, 3]. See [16] for further discussion on the assumption, and on prior work regarding the relation between Goldreich's PRG and hardness of learning.
Neural networks. We consider feedforward ReLU networks. Starting from an input  $\mathbf{x} \in \mathbb{R}^d$ , each layer in the network is of the form  $\mathbf{z} \mapsto \sigma(W_i \mathbf{z} + \mathbf{b}_i)$ , where  $\sigma(a) = [a]_+ = \max\{0, a\}$  is the ReLU activation which applies to vectors entry-wise,  $W_i$  is the weight matrix, and  $\mathbf{b}_i$  are the bias terms. The weights vector of the  $j$ -th neuron in the  $i$ -th layer is the  $j$ -th row of  $W_i$ , and its outgoing-weights vector is the  $j$ -th column of  $W_{i+1}$ . We define the depth of the network as the number of layers. Unless stated otherwise, the output neuron also has a ReLU activation function. Note that a depth- $k$  network with activation in the output neuron has  $k$  non-linear layers. A neuron which is not an input or output neuron is called a hidden neuron. We sometimes consider neural networks with multiple outputs. The parameters of the neural network is the set of its weight matrices and bias vectors. We often view the parameters as a vector  $\theta \in \mathbb{R}^p$  obtained by concatenating these matrices and vectors. For  $B \geq 0$ , we say that the parameters are  $B$ -bounded if the absolute values of all weights and biases are at most  $B$ .
Learning neural networks and the smoothed-analysis framework. We first define neural networks learning under the standard PAC framework:
Definition 2.1 (Distribution-specific PAC learning). Learning depth- $k$  neural networks under an input distribution  $\mathcal{D}$  on  $\mathbb{R}^d$  is defined by the following framework:
1. An adversary chooses a set of  $B$ -bounded parameters  $\theta \in \mathbb{R}^p$  for a depth- $k$  neural network  $N_{\theta}: \mathbb{R}^d \to \mathbb{R}$ , as well as some  $\epsilon > 0$ .
2. Consider an example oracle, such that each example  $(\mathbf{x},y)\in \mathbb{R}^d\times \mathbb{R}$  is drawn i.i.d. with  $\mathbf{x}\sim \mathcal{D}$  and  $y = N_{\theta}(\mathbf{x})$ . Then, given access to the examples oracle, the goal of the learning algorithm  $\mathcal{L}$  is to return with probability at least  $\frac{3}{4}$  a hypothesis  $h:\mathbb{R}^d\to \mathbb{R}$  such that  $\mathbb{E}_{\mathbf{x}\sim \mathcal{D}}\left[(h(\mathbf{x}) - N_{\theta}(\mathbf{x}))^{2}\right]\leq \epsilon$ . We say that  $\mathcal{L}$  is efficient if it runs in time  $\mathrm{poly}(d,p,B,1 / \epsilon)$ .
We consider learning in the smoothed-analysis framework [40], which is a popular paradigm for analyzing non-worst-case computational complexity [36]. The smoothed-analysis framework has been successfully applied to many learning problems (e.g., [6, 21, 30, 8, 9, 26, 19, 29, 10]). In the smoothed-analysis setting, the target network is not purely controlled by an adversary. Instead, the adversary can first generate an arbitrary network, and the parameters for this network (i.e., the weight matrices and bias terms) will be randomly perturbed to yield a perturbed network. The algorithm only needs to work with high probability on the perturbed network. This limits the power of the adversary and prevents it from creating highly degenerate cases. Formally, we consider the following framework (we note that a similar model was considered in [6, 21]):
Definition 2.2 (Learning with smoothed parameters). Learning depth- $k$  neural networks with smoothed parameters under an input distribution  $\mathcal{D}$  is defined as follows:
1. An adversary chooses a set of  $B$ -bounded parameters  $\theta \in \mathbb{R}^p$  for a depth- $k$  neural network  $N_{\theta}: \mathbb{R}^d \to \mathbb{R}$ , as well as some  $\tau, \epsilon > 0$ .
2. A perturbed set of parameters  $\hat{\pmb{\theta}}$  is obtained by a random perturbation to  $\pmb{\theta}$ , namely,  $\hat{\pmb{\theta}} = \pmb{\theta} + \pmb{\xi}$  for  $\pmb{\xi} \sim \mathcal{N}(\mathbf{0}, \tau^2 I_p)$ .
3. Consider an examples oracle, such that each example  $(\mathbf{x},y)\in \mathbb{R}^d\times \mathbb{R}$  is drawn i.i.d. with  $\mathbf{x}\sim \mathcal{D}$  and  $y = N_{\hat{\theta}}(\mathbf{x})$ . Then, given access to the examples oracle, the goal of the learning algorithm  $\mathcal{L}$  is to return with probability at least  $\frac{3}{4}$  (over the random perturbation  $\xi$  and the internal randomness of  $\mathcal{L}$ ) a hypothesis  $h:\mathbb{R}^d\to \mathbb{R}$  such that  $\mathbb{E}_{\mathbf{x}\sim \mathcal{D}}\left[\left(h(\mathbf{x}) - N_{\hat{\theta}}(\mathbf{x})\right)^2\right]\leq \epsilon$ . We say that  $\mathcal{L}$  is efficient if it runs in time  $\mathrm{poly}(d,p,B,1 / \epsilon ,1 / \tau)$ .
Finally, we also consider a setting where both the parameters and the input distribution are smoothed:
Definition 2.3 (Learning with smoothed parameters and inputs). Learning depth- $k$  neural networks with smoothed parameters and inputs under an input distribution  $\mathcal{D}$  is defined as follows:
1. An adversary chooses a set of  $B$ -bounded parameters  $\pmb{\theta} \in \mathbb{R}^p$  for a depth- $k$  neural network  $N_{\pmb{\theta}}: \mathbb{R}^d \to \mathbb{R}$ , as well as some  $\tau, \omega, \epsilon > 0$ .
2. A perturbed set of parameters  $\hat{\pmb{\theta}}$  is obtained by a random perturbation to  $\pmb{\theta}$ , namely,  $\hat{\pmb{\theta}} = \pmb{\theta} + \pmb{\xi}$  for  $\pmb{\xi} \sim \mathcal{N}(\mathbf{0}, \tau^2 I_p)$ . Moreover, a smoothed input distribution  $\hat{\mathcal{D}}$  is obtained from  $\mathcal{D}$ , such that  $\hat{\mathbf{x}} \sim \hat{\mathcal{D}}$  is chosen by drawing  $\mathbf{x} \sim \mathcal{D}$  and adding a random perturbation from  $\mathcal{N}(\mathbf{0}, \omega^2 I_d)$ .
3. Consider an examples oracle, such that each example  $(\mathbf{x},y)\in \mathbb{R}^d\times \mathbb{R}$  is drawn i.i.d. with  $\mathbf{x}\sim \hat{\mathcal{D}}$  and  $y = N_{\hat{\pmb{\theta}}}(\mathbf{x})$ . Then, given access to the examples oracle, the goal of the learning algorithm  $\mathcal{L}$  is to return with probability at least  $\frac{3}{4}$  (over the random perturbation  $\xi$  and the internal randomness of  $\mathcal{L}$ ) a hypothesis  $h:\mathbb{R}^d\to \mathbb{R}$  such that  $\mathbb{E}_{\mathbf{x}\sim \hat{\mathcal{D}}}\left[\left(h(\mathbf{x}) - N_{\hat{\pmb{\theta}}}(\mathbf{x})\right)^2\right]\leq \epsilon$ . We say that  $\mathcal{L}$  is efficient if it runs in time  $\mathrm{poly}(d,p,B,1 / \epsilon ,1 / \tau ,1 / \omega)$ .
We emphasize that all of the above definitions consider learning in the distribution-specific setting. Thus, the learning algorithm may depend on the specific input distribution  $\mathcal{D}$ .
As we discussed in the introduction, there exist efficient algorithms for learning depth-2 (one-hidden-layer) ReLU networks with smoothed parameters under the Gaussian distribution. We now show that such a result may not be achieved for depth-3 networks:
Theorem 3.1. Under Assumption 2.1, there is no efficient algorithm that learns depth-3 networks with smoothed parameters (in the sense of Definition 2.2) under the standard Gaussian distribution.
We prove the theorem in Section 4. Next, we conclude that learning depth-3 neural networks under the Gaussian distribution on  $\mathbb{R}^d$  is hard in the standard PAC framework even if all weight matrices are non-degenerate, namely, when the minimal singular values of the weight matrices are lower bounded by  $1 / \mathrm{poly}(d)$ . As we discussed in the introduction, in one-hidden-layer networks with similar assumptions there exist efficient learning algorithms.
Corollary 3.1. Under Assumption 2.1, there is no efficient algorithm that learns depth-3 networks (in the sense of Definition 2.1) under the standard Gaussian distribution on  $\mathbb{R}^d$ , even if the smallest singular value of each weight matrix is at least  $1/\mathrm{poly}(d)$ .
The proof of the above corollary follows from Theorem 3.1, using the fact that by adding a small random perturbation to the weight matrices, we get w.h.p. non-degenerate matrices. Hence, an efficient algorithm that learns non-degenerate networks suffices for obtaining an efficient algorithm that learns under the smoothed analysis framework. See Appendix B for the formal proof.
The above hardness results consider depth-3 networks that include activation in the output neuron. Thus, the networks have three non-linear layers. We remark that these results readily imply hardness also for depth-4 networks without activation in the output, and hardness for depth- $k$  networks for any  $k > 3$ . We also note that the hardness results hold already for networks where all hidden layers are of the same width, e.g., where all layers are of width  $d$ .