SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We emphasize that our hardness results are for neural networks that include the ReLU activation also in the output neuron. In contrast, the positive results on learning depth-2 networks that we discussed in the introduction do not include activation in the output neuron. Therefore, as far as we are aware, there is still a small gap between the upper bounds and our hardness results: (1) Under the assumption that the input is Gaussian and the weights are non-degenerate, the cases of depth-2 networks with activation in the output neuron and of depth-3 networks without activation in the output are not settled; (2) In the setting where both the parameters and the input distribution are smoothed, the case of depth-2 networks without activation in the output is not settled. These gaps are an intriguing subject for further research.
This work was done as part of the NSF-Simons Sponsored Collaboration on the Theoretical Foundations of Deep Learning.
[1] B. Applebaum. Pseudorandom generators with long stretch and low locality from random local one-way functions. SIAM Journal on Computing, 42(5):2008-2037, 2013.
[2] B. Applebaum and S. Lovett. Algebraic attacks against random local functions and their countermeasures. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 1087-1100, 2016.
[3] B. Applebaum and P. Raykov. Fast pseudorandom functions based on expander graphs. In Theory of Cryptography Conference, pages 27-56. Springer, 2016.
[4] B. Applebaum, B. Barak, and A. Wigderson. Public-key cryptography from different assumptions. In Proceedings of the forty-second ACM symposium on Theory of computing, pages 171-180, 2010.
[5] B. Applebaum, I. Damgård, Y. Ishai, M. Nielsen, and L. Zichron. Secure arithmetic computation with constant computational overhead. In Annual International Cryptology Conference, pages 223–254. Springer, 2017.
[6] P. Awasthi, A. Tang, and A. Vijayaraghavan. Efficient algorithms for learning depth-2 neural networks with general relu activations. Advances in Neural Information Processing Systems, 34:13485–13496, 2021.
[7] A. Bakshi, R. Jayaram, and D. P. Woodruff. Learning two layer rectified neural networks in polynomial time. In Conference on Learning Theory, pages 195-268. PMLR, 2019.
[8] A. Bhaskara, M. Charikar, A. Moitra, and A. Vijayaraghavan. Smoothed analysis of tensor decompositions. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing, pages 594-603, 2014.
[9] A. Bhaskara, A. Chen, A. Perreault, and A. Vijayaraghavan. Smoothed analysis in unsupervised learning via decoupling. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS), pages 582-610. IEEE, 2019.
[10] A. Brutzkus, A. Daniely, and E. Malach. Id3 learns juntas for smoothed product distributions. In Conference on Learning Theory, pages 902-915. PMLR, 2020.
[31] A. R. Klivans and A. A. Sherstov. Cryptographic hardness for learning intersections of half-spaces. In FOCS, 2006.
[32] B. Laurent and P. Massart. Adaptive estimation of a quadratic functional by model selection. Annals of Statistics, pages 1302-1338, 2000.
[33] H. Lin. Indistinguishability obfuscation from constant-degree graded encoding schemes. In Annual International Conference on the Theory and Applications of Cryptographic Techniques, pages 28-57. Springer, 2016.
[34] H. Lin and V. Vaikuntanathan. Indistinguishability obfuscation from ddh-like assumptions on constant-degree graded encodings. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pages 11-20. IEEE, 2016.
[35] P. Méaux, C. Carlet, A. Journault, and F.-X. Standaert. Improved filter permutators: Combining symmetric encryption design, boolean functions, low complexity cryptography, and homomorphic encryption, for private delegation of computations. IACR Cryptol. ePrint Arch., 2019:483, 2019.
[36] T. Roughgarden. Beyond the worst-case analysis of algorithms. Cambridge University Press, 2021.
[37] A. Sankar, D. A. Spielman, and S.-H. Teng. Smoothed analysis of the condition numbers and growth factors of matrices. SIAM Journal on Matrix Analysis and Applications, 28(2):446-476, 2006.
[38] O. Shamir. Distribution-specific hardness of learning neural networks. The Journal of Machine Learning Research, 19(1):1135-1163, 2018.
[39] L. Song, S. Vempala, J. Wilmes, and B. Xie. On the complexity of learning neural networks. In Advances in neural information processing systems, pages 5514-5522, 2017.
[40] D. A. Spielman and S.-H. Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. Journal of the ACM (JACM), 51(3):385-463, 2004.
[41] S. Vempala and J. Wilmes. Gradient descent for one-hidden-layer neural networks: Polynomial convergence and sq lower bounds. In Conference on Learning Theory, pages 3115-3117. PMLR, 2019.
[42] Y. Zhang, J. D. Lee, and M. I. Jordan. 11-regularized neural networks are improperly learnable in polynomial time. In International Conference on Machine Learning, pages 993–1001. PMLR, 2016.
[43] K. Zhong, Z. Song, P. Jain, P. L. Bartlett, and I. S. Dhillon. Recovery guarantees for one-hidden-layer neural networks. In International conference on machine learning, pages 4140-4149. PMLR, 2017.
A Missing lemmas for the proof of Theorem 3.1
Lemma A.1 (Daniely and Vardi [16]). For every predicate  $P: \{0,1\}^k \to \{0,1\}$  and  $\mathbf{x} \in \{0,1\}^n$ , there is a DNF formula  $\psi$  over  $\{0,1\}^{kn}$  with at most  $2^k$  terms, such that for every hyperedge  $S$  we have  $P_{\mathbf{x}}(\mathbf{z}^S) = \psi(\mathbf{z}^S)$ . Moreover, each term in  $\psi$  is a conjunction of positive literals.
Proof. The following proof is from Daniely and Vardi [16], and we give it here for completeness.
We denote by  $\mathcal{B} \subseteq \{0,1\}^k$  the set of satisfying assignments of  $P$ . Note that the size of  $\mathcal{B}$  is at most  $2^k$ . Consider the following DNF formula over  $\{0,1\}^{kn}$ :
$$
\psi (\mathbf{z}) = \bigvee_{\mathbf{b}\in \mathcal{B}}\bigwedge_{j\in [k]}\bigwedge_{\{l:x_{l}\neq b_{j}\}}z_{j,l}  .
$$
For a hyperedge  $S = (i_1, \ldots, i_k)$ , we have
$$
\begin{array}{l} \psi (\mathbf {z} ^ {S}) = 1 \iff \exists \mathbf {b} \in \mathcal {B} \forall j \in [ k ] \forall x _ {l} \neq b _ {j}, z _ {j, l} ^ {S} = 1 \\ \Longleftrightarrow \exists \mathbf {b} \in \mathcal {B} \forall j \in [ k ] \forall x _ {l} \neq b _ {j}, i _ {j} \neq l \\ \Longleftrightarrow \exists \mathbf {b} \in \mathcal {B} \forall j \in [ k ], x _ {i j} = b _ {j} \\ \Longleftrightarrow \exists \mathbf {b} \in \mathcal {B}, \mathbf {x} _ {S} = \mathbf {b} \\ \iff P (\mathbf {x} _ {S}) = 1 \\ \Longleftrightarrow P _ {\mathbf {x}} \left(\mathbf {z} ^ {S}\right) = 1. \\ \end{array}
$$
Lemma A.2. Let  $\mathbf{x} \in \{0,1\}^n$ . There exists an affine layer with at most  $2^k$  outputs, weights bounded by a constant and bias terms bounded by  $n \log(n)$  (for a sufficiently large  $n$ ), such that given an input  $\mathbf{z}^S \in \{0,1\}^{kn}$  for some hyperedge  $S$ , it satisfies the following: For  $S$  with  $P_{\mathbf{x}}(\mathbf{z}^S) = 0$  all outputs are at most  $-1$ , and for  $S$  with  $P_{\mathbf{x}}(\mathbf{z}^S) = 1$  there exists an output greater or equal to 2.
Proof. By Lemma A.1, there exists a DNF formula  $\varphi_{\mathbf{x}}$  over  $\{0,1\}^{kn}$  with at most  $2^k$  terms, such that  $\varphi_{\mathbf{x}}(\mathbf{z}^S) = P_{\mathbf{x}}(\mathbf{z}^S)$ . Thus, if  $P_{\mathbf{x}}(\mathbf{z}^S) = 0$  then all terms in  $\varphi_{\mathbf{x}}$  are not satisfied for the input  $\mathbf{z}^S$ , and if  $P_{\mathbf{x}}(\mathbf{z}^S) = 1$  then there is at least one term in  $\varphi_{\mathbf{x}}$  which is satisfied for the input  $\mathbf{z}^S$ . Therefore, it suffices to construct an affine layer such that for an input  $\mathbf{z}^S$ , the  $j$ -th output will be at most  $-1$  if the  $j$ -th term of  $\varphi_{\mathbf{x}}$  is not satisfied, and at least 2 otherwise. Each term  $C_j$  in  $\varphi_{\mathbf{x}}$  is a conjunction of positive literals. Let  $I_j \subseteq [kn]$  be the indices of these literals. The  $j$ -th output of the affine layer will be
$$
\left(\sum_ {l \in I _ {j}} 3 z _ {l} ^ {S}\right) - 3 | I _ {j} | + 2.
$$
Note that if the conjunction  $C_j$  holds, then this expression is exactly  $3|I_j| - 3|I_j| + 2 = 2$ , and otherwise it is at most  $3(|I_j| - 1) - 3|I_j| + 2 = -1$ . Finally, note that all weights are bounded by 3 and all bias terms are bounded by  $n\log (n)$  (for large enough  $n$ ).
Lemma A.3. Let  $\mathbf{x} \in \{0,1\}^n$ . There exists a depth-2 neural network  $N_{1}$  with input dimension  $kn$ ,  $2kn$  hidden neurons, at most  $2^{k}$  output neurons, and parameter magnitudes bounded by  $n^3$  (for a sufficiently large  $n$ ), which satisfies the following. We denote the set of output neurons of  $N_{1}$  by  $\mathcal{E}_1$ . Let  $\mathbf{z}' \in \mathbb{R}^{kn}$  be such that  $\Psi(\mathbf{z}') = \mathbf{z}^S$  for some hyperedge  $S$ , and assume that for every  $i \in [kn]$  we have  $z_i' \notin (c, c + \frac{1}{n^2})$ . Then, for  $S$  with  $P_{\mathbf{x}}(\mathbf{z}^S) = 0$  the inputs to all neurons  $\mathcal{E}_1$  are at most  $-1$ , and for  $S$  with  $P_{\mathbf{x}}(\mathbf{z}^S) = 1$  there exists a neuron in  $\mathcal{E}_1$  with input at least 2. Moreover, only the second layer of  $N_{1}$  depends on  $\mathbf{x}$ .
Proof. First, we construct a depth-2 neural network  $N_{\Psi}:\mathbb{R}^{kn}\to [0,1]^{kn}$  with a single layer of nonlinearity, such that for every  $\mathbf{z}'\in \mathbb{R}^{kn}$  with  $z_i'\notin (c,c + \frac{1}{n^2})$  for every  $i\in [kn]$ , we have  $N_{\Psi}(\mathbf{z}') = \Psi (\mathbf{z}')$ . The network  $N_{\Psi}$  has  $2kn$  hidden neurons, and computes  $N_{\Psi}(\mathbf{z}') = (f(z_1'),\ldots ,f(z_{kn}')$ , where  $f:\mathbb{R}\rightarrow [0,1]$  is such that
$$
f (t) = n ^ {2} \cdot \left(\left[ t - c \right] _ {+} - \left[ t - \left(c + \frac {1}{n ^ {2}}\right) \right] _ {+}\right).
$$
Note that if  $t \leq c$  then  $f(t) = 0$ , if  $t \geq c + \frac{1}{n^2}$  then  $f(t) = 1$ , and if  $c < t < c + \frac{1}{n^2}$  then  $f(t) \in (0,1)$ . Also, note that all weights and bias terms can be bounded by  $n^2$  (for large enough  $n$ ). Moreover, the network  $N_{\Psi}$  does not depend on  $\mathbf{x}$ .
Let  $\mathbf{z}' \in \mathbb{R}^{kn}$  such that  $\Psi(\mathbf{z}') = \mathbf{z}^S$  for some hyperedge  $S$ , and assume that for every  $i \in [kn]$  we have  $z_i' \notin (c, c + \frac{1}{n^2})$ . For such  $\mathbf{z}'$ , we have  $N_{\Psi}(\mathbf{z}') = \Psi(\mathbf{z}') = \mathbf{z}^S$ . Hence, it suffices to show that we can construct an affine layer with at most  $2^k$  outputs, weights bounded by a constant and bias terms bounded by  $n^3$ , such that given an input  $\mathbf{z}^S$  it satisfies the following: For  $S$  with  $P_{\mathbf{x}}(\mathbf{z}^S) = 0$  all outputs are at most  $-1$ , and for  $S$  with  $P_{\mathbf{x}}(\mathbf{z}^S) = 1$  there exists an output greater or equal to 2. We construct such an affine layer in Lemma A.2.
Lemma A.4. There exists an affine layer with  $2k + n$  outputs, weights bounded by a constant and bias terms bounded by  $n\log (n)$  (for a sufficiently large  $n$ ), such that given an input  $\mathbf{z}\in \{0,1\}^{kn}$ , if it is an encoding of a hyperedge then all outputs are at most  $-1$ , and otherwise there exists an output greater or equal to 2.
Proof. Note that  $\mathbf{z} \in \{0,1\}^{kn}$  is not an encoding of a hyperedge iff at least one of the following holds:
1. At least one of the  $k$  size-  $n$  slices in  $\mathbf{z}$  contains 0 more than once.
2. At least one of the  $k$  size-  $n$  slices in  $\mathbf{z}$  does not contain 0.
3. There are two size-  $n$  slices in  $\mathbf{z}$  that encode the same index.
We define the outputs of our affine layer as follows. First, we have  $k$  outputs that correspond to (1). In order to check whether slice  $i \in [k]$  contains 0 more than once, the output will be  $3n - 4 - (\sum_{j \in [n]} 3z_{i,j})$ . Second, we have  $k$  outputs that correspond to (2): in order to check whether slice  $i \in [k]$  does not contain 0, the output will be  $(\sum_{j \in [n]} 3z_{i,j}) - 3n + 2$ . Finally, we have  $n$  outputs that correspond to (3): in order to check whether there are two slices that encode the same index  $j \in [n]$ , the output will be  $3k - 4 - (\sum_{i \in [k]} 3z_{i,j})$ . Note that all weights are bounded by 3 and all bias terms are bounded by  $n \log(n)$  for large enough  $n$ .
Lemma A.5. There exists a depth-2 neural network  $N_{2}$  with input dimension  $kn$ , at most  $2kn$  hidden neurons,  $2k + n$  output neurons, and parameter magnitudes bounded by  $n^3$  (for a sufficiently large  $n$ ), which satisfies the following. We denote the set of output neurons of  $N_{2}$  by  $\mathcal{E}_2$ . Let  $\mathbf{z}' \in \mathbb{R}^{kn}$  be such that for every  $i \in [kn]$  we have  $z_i' \notin (c, c + \frac{1}{n^2})$ . If  $\Psi(\mathbf{z}')$  is an encoding of a hyperedge then the inputs to all neurons  $\mathcal{E}_2$  are at most  $-1$ , and otherwise there exists a neuron in  $\mathcal{E}_2$  with input at least 2.
Proof. Let  $N_{\Psi} : \mathbb{R}^{kn} \to [0,1]^{kn}$  be the depth-2 neural network from the proof of Lemma A.3, with a single layer of non-linearity with  $2kn$  hidden neurons, and parameter magnitudes bounded by  $n^2$ , such that for every  $\mathbf{z}' \in \mathbb{R}^{kn}$  with  $z_i' \notin (c, c + \frac{1}{n^2})$  for every  $i \in [kn]$ , we have  $N_{\Psi}(\mathbf{z}') = \Psi(\mathbf{z}')$ .
Let  $\mathbf{z}' \in \mathbb{R}^{kn}$  be such that for every  $i \in [kn]$  we have  $z_i' \notin \left(c, c + \frac{1}{n^2}\right)$ . For such  $\mathbf{z}'$  we have  $N_{\Psi}(\mathbf{z}') = \Psi(\mathbf{z}')$ . Hence, it suffices to show that we can construct an affine layer with  $2k + n$  outputs, weights bounded by a constant and bias terms bounded by  $n^3$ , such that given an input  $\mathbf{z} \in \{0, 1\}^{kn}$ , if it is an encoding of a hyperedge then all outputs are at most  $-1$ , and otherwise there exists an output greater or equal to 2. We construct such an affine layer in Lemma A.4.
Lemma A.6. There exists a depth-2 neural network  $N_{3}$  with input dimension  $kn$ , at most  $n\log (n)$  hidden neurons,  $kn \leq n\log (n)$  output neurons, and parameter magnitudes bounded by  $n^3$  (for a sufficiently large  $n$ ), which satisfies the following. We denote the set of output neurons of  $N_{3}$  by  $\mathcal{E}_3$ . Let  $\mathbf{z}' \in \mathbb{R}^{kn}$ . If there exists  $i \in [kn]$  such that  $z_i' \in \left(c, c + \frac{1}{n^2}\right)$  then there exists a neuron in  $\mathcal{E}_3$  with input at least 2. If for all  $i \in [kn]$  we have  $z_i' \notin \left(c - \frac{1}{n^2}, c + \frac{2}{n^2}\right)$  then the inputs to all neurons in  $\mathcal{E}_3$  are at most  $-1$ .
Proof. It suffices to construct a univariate depth-2 network  $f: \mathbb{R} \to \mathbb{R}$  with one non-linear layer and a constant number of hidden neurons, such that for every input  $z_i' \in (c, c + \frac{1}{n^2})$  we have  $f(z_i') = 2$ , and for every  $z_i' \notin (c - \frac{1}{n^2}, c + \frac{2}{n^2})$  we have  $f(z_i') = -1$ .
We construct  $f$  as follows:
$$
\begin{array}{l} f (z _ {i} ^ {\prime}) = (3 n ^ {2}) \left(\left[ z _ {i} ^ {\prime} - \left(c - \frac {1}{n ^ {2}}\right) \right] _ {+} - [ z _ {i} ^ {\prime} - c ] _ {+}\right) - \\ (3 n ^ {2}) \left(\left[ z _ {i} ^ {\prime} - \left(c + \frac {1}{n ^ {2}}\right) \right] _ {+} - \left[ z _ {i} ^ {\prime} - \left(c + \frac {2}{n ^ {2}}\right) \right] _ {+}\right) - 1. \\ \end{array}
$$
Note that all weights and bias terms are bounded by  $n^3$  (for large enough  $n$ ).
Lemma A.7. Let  $q = \mathrm{poly}(n)$  and  $r = \mathrm{poly}(n)$ . Then, there exists  $\tau = \frac{1}{\mathrm{poly}(n)}$  such that for a sufficiently large  $n$ , with probability at least  $1 - \exp(-n/2)$  a vector  $\xi \sim \mathcal{N}(\mathbf{0}, \tau^2 I_r)$  satisfies  $\| \xi \| \leq \frac{1}{q}$ .
Proof. Let  $\tau = \frac{1}{q\sqrt{2r n}}$ . Every component  $\xi_{i}$  in  $\pmb{\xi}$  has the distribution  $\mathcal{N}(0,\tau^2)$ . By a standard tail bound of the Gaussian distribution, we have for every  $i\in [r]$  and  $t\geq 0$  that  $\operatorname*{Pr}[\xi_i\geq t]\leq 2\exp \left(-\frac{t^2}{2\tau^2}\right)$ . Hence, for  $t = \frac{1}{q\sqrt{r}}$ , we get
$$
\operatorname * {P r} \left[ \xi_ {i} \geq \frac {1}{q \sqrt {r}} \right] \leq 2 \exp \left(- \frac {1}{2 \tau^ {2} q ^ {2} r}\right) = 2 \exp \left(- \frac {2 r n q ^ {2}}{2 q ^ {2} r}\right) = 2 \exp (- n).
$$
By the union bound, with probability at least  $1 - r \cdot 2e^{-n}$ , we have
$$
\| \boldsymbol {\xi} \| ^ {2} \leq r \cdot \frac {1}{r q ^ {2}} = \frac {1}{q ^ {2}}.
$$
Thus, for a sufficiently large  $n$ , with probability at least  $1 - \exp(-n/2)$  we have  $\|\pmb{\xi}\| \leq \frac{1}{q}$ .
Lemma A.8. If  $S$  is pseudorandom then with probability at least  $\frac{39}{40}$  (over  $\xi \sim \mathcal{N}(0, \tau^2 I_p)$  and the i.i.d. inputs  $\tilde{\mathbf{z}}_i \sim \mathcal{D}$ ) the examples  $(\tilde{\mathbf{z}}_1, \tilde{y}_1), \ldots, (\tilde{\mathbf{z}}_{m(n) + n^3}, \tilde{y}_{m(n) + n^3})$  returned by the oracle are realized by  $\hat{N}$ .
Proof. By our choice of  $\tau$ , with probability at least  $1 - \frac{1}{n}$  over  $\xi \sim \mathcal{N}(\mathbf{0}, \tau^2 I_p)$ , we have  $|\xi_j| \leq \frac{1}{10}$  for all  $j \in [p]$ , and for every  $\tilde{\mathbf{z}}$  with  $\|\tilde{\mathbf{z}}\| \leq 2n$  the inputs to the neurons  $\mathcal{E}_1, \mathcal{E}_2, \mathcal{E}_3$  in the computation  $\hat{N}(\tilde{\mathbf{z}})$  satisfy Properties (P1) through (P3). We first show that with probability at least  $1 - \frac{1}{n}$  all examples  $\tilde{\mathbf{z}}_1, \ldots, \tilde{\mathbf{z}}_{m(n) + n^3}$  satisfy  $\|\tilde{\mathbf{z}}_i\| \leq 2n$ . Hence, with probability at least  $1 - \frac{2}{n}$ , Properties (P1) through (P3) hold for the computations  $\hat{N}(\tilde{\mathbf{z}}_i)$  for all  $i \in [m(n) + n^3]$ .
Note that  $\| \tilde{\mathbf{z}}_i\|^2$  has the Chi-squared distribution. Since  $\tilde{\mathbf{z}}_i$  is of dimension  $n^2$ , a concentration bound by Laurent and Massart [32, Lemma 1] implies that for all  $t > 0$  we have
$$
\operatorname * {P r} \left[ \| \tilde {\mathbf {z}} _ {i} \| ^ {2} - n ^ {2} \geq 2 n \sqrt {t} + 2 t \right] \leq e ^ {- t}.
$$
Plugging-in  $t = \frac{n^2}{4}$ , we get
$$
\begin{array}{l} \Pr \left[ \| \tilde {\mathbf {z}} _ {i} \| ^ {2} \geq 4 n ^ {2} \right] = \Pr \left[ \| \tilde {\mathbf {z}} _ {i} \| ^ {2} - n ^ {2} \geq 3 n ^ {2} \right] \\ \leq \Pr \left[ \| \tilde {\mathbf {z}} _ {i} \| ^ {2} - n ^ {2} \geq \frac {3 n ^ {2}}{2} \right] \\ = \Pr \left[ \| \tilde {\mathbf {z}} _ {i} \| ^ {2} - n ^ {2} \geq 2 n \sqrt {\frac {n ^ {2}}{4}} + 2 \cdot \frac {n ^ {2}}{4} \right] \\ \leq \exp \left(- \frac {n ^ {2}}{4}\right) . \\ \end{array}
$$
Thus, we have  $\operatorname*{Pr}\left[\| \tilde{\mathbf{z}}_i\| \geq 2n\right] \leq \exp \left(-\frac{n^2}{4}\right)$ . By the union bound, with probability at least
$$
1 - \left(m (n) + n ^ {3}\right) \exp \left(- \frac {n ^ {2}}{4}\right) \geq 1 - \frac {1}{n}
$$
(for a sufficiently large  $n$ ), all examples  $(\tilde{\mathbf{z}}_i, \tilde{y}_i)$  satisfy  $\| \tilde{\mathbf{z}}_i \| \leq 2n$ .
Thus, we showed that with probability at least  $1 - \frac{2}{n} \geq \frac{39}{40}$  (for a sufficiently large  $n$ ), we have  $|\xi_j| \leq \frac{1}{10}$  for all  $j \in [p]$ , and Properties (P1) through (P3) hold for the computations  $\hat{N}(\tilde{\mathbf{z}}_i)$  for all  $i \in [m(n) + n^3]$ . It remains to show that if these properties hold, then the examples  $(\tilde{\mathbf{z}}_1, \tilde{y}_1), \ldots, (\tilde{\mathbf{z}}_{m(n) + n^3}, \tilde{y}_{m(n) + n^3})$  are realized by  $\hat{N}$ .
Let  $i \in [m(n) + n^3]$ . For brevity, we denote  $\tilde{\mathbf{z}} = \tilde{\mathbf{z}}_i$ ,  $\tilde{y} = \tilde{y}_i$ , and  $\mathbf{z}' = \tilde{\mathbf{z}}_{[kn]}$ . Since  $|\xi_j| \leq \frac{1}{10}$  for all  $j \in [p]$ , and all incoming weights to the output neuron in  $\tilde{N}$  are  $-1$ , then in  $\hat{N}$  all incoming weights to the output neuron are in  $\left[-\frac{11}{10}, -\frac{9}{10}\right]$ , and the bias term in the output neuron, denoted by  $\hat{b}$ , is in  $\left[\frac{9}{10}, \frac{11}{10}\right]$ . Consider the following cases:
- If  $\Psi(\mathbf{z}^{\prime})$  is not an encoding of a hyperedge then  $\tilde{y} = 0$ , and  $\hat{N}(\tilde{\mathbf{z}})$  satisfies:
1. If  $\mathbf{z}'$  does not have components in  $\left(c, c + \frac{1}{n}\right)$ , then there exists a neuron in  $\mathcal{E}_2$  with output at least  $\frac{3}{2}$  (by Property (P2)).
2. If  $\mathbf{z}'$  has a component in  $\left(c, c + \frac{1}{n}\right)$ , then there exists a neuron in  $\mathcal{E}_3$  with output at least  $\frac{3}{2}$  (by Property (P3)).
In both cases, since all incoming weights to the output neuron in  $\hat{N}$  are in  $\left[-\frac{11}{10}, -\frac{9}{10}\right]$ , and  $\hat{b} \in \left[\frac{9}{10}, \frac{11}{10}\right]$ , then the input to the output neuron (including the bias term) is at most  $\frac{11}{10} - \frac{3}{2} \cdot \frac{9}{10} < 0$ , and thus its output is 0.
- If  $\Psi(\mathbf{z}^{\prime})$  is an encoding of a hyperedge  $S$ , then by the definition of the examples oracle we have  $S = S_{i}$ . Hence:
- If  $\mathbf{z}'$  does not have components in  $\left(c - \frac{1}{n^2}, c + \frac{2}{n^2}\right)$ , then:
* If  $y_{i} = 0$  then the oracle sets  $\tilde{y} = \hat{b}$ . Since  $S$  is pseudorandom, we have  $P_{\mathbf{x}}(\mathbf{z}^{S}) = P_{\mathbf{x}}(\mathbf{z}^{S_{i}}) = y_{i} = 0$ . Hence, in the computation  $\hat{N}(\tilde{\mathbf{z}})$  the inputs to all neurons in  $\mathcal{E}_1, \mathcal{E}_2, \mathcal{E}_3$  are at most  $-\frac{1}{2}$  (by Properties (P1), (P2) and (P3)), and thus their outputs are 0. Therefore,  $\hat{N}(\tilde{\mathbf{z}}) = \hat{b}$ .
* If  $y_{i} = 1$  then the oracle sets  $\tilde{y} = 0$ . Since  $S$  is pseudorandom, we have  $P_{\mathbf{x}}(\mathbf{z}^{S}) = P_{\mathbf{x}}(\mathbf{z}^{S_{i}}) = y_{i} = 1$ . Hence, in the computation  $\hat{N}(\tilde{\mathbf{z}})$  there exists a neuron in  $\mathcal{E}_{1}$  with output at least  $\frac{3}{2}$  (by Property (P1)). Since all incoming weights to the output neuron in  $\hat{N}$  are in  $[-\frac{11}{10}, -\frac{9}{10}]$ , and  $\hat{b} \in [\frac{9}{10}, \frac{11}{10}]$ , then the input to output neuron (including the bias term) is at most  $\frac{11}{10} - \frac{3}{2} \cdot \frac{9}{10} < 0$ , and thus its output is 0.
- If  $\mathbf{z}'$  has a component in  $\left(c, c + \frac{1}{n^2}\right)$ , then  $\tilde{y} = 0$ . Also, in the computation  $\hat{N}(\tilde{\mathbf{z}})$  there exists a neuron in  $\mathcal{E}_3$  with output at least  $\frac{3}{2}$  (by Property (P3)). Since all incoming weights to the output neuron in  $\hat{N}$  are in  $[- \frac{11}{10}, - \frac{9}{10}]$ , and  $\hat{b} \in \left[\frac{9}{10}, \frac{11}{10}\right]$ , then the input to output neuron (including the bias term) is at most  $\frac{11}{10} - \frac{3}{2} \cdot \frac{9}{10} < 0$ , and thus its output is 0.
- If  $\mathbf{z}'$  does not have components in the interval  $(c, c + \frac{1}{n^2})$ , but has a component in the interval  $(c - \frac{1}{n^2}, c + \frac{2}{n^2})$ , then: