SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $N_{2}$  be a depth-1 neural network (i.e., one layer, with activations in the output neurons) with input dimension  $kn$ , at most  $2n$  output neurons, and parameter magnitudes bounded by  $n^3$  (for a sufficiently large  $n$ ), which satisfies the following. We denote the set of output neurons of  $N_{2}$  by  $\mathcal{E}_2$ . Let  $\mathbf{z}' \in \{0,1\}^{kn}$  be an input to  $N_{2}$  (note that it has components only in  $\{0,1\}$ ). If  $\mathbf{z}'$  is an encoding of a hyperedge then the inputs to all output neurons  $\mathcal{E}_2$  are at most  $-1$ , and otherwise there exists a neuron in  $\mathcal{E}_2$  with input at least 2. The construction of the network  $N_{2}$  is given in Lemma A.4. Let  $N_2': \mathbb{R}^{kn} \to \mathbb{R}$  be a depth-2 neural network with no activation function in the output neuron, obtained from  $N_{2}$  by summing the outputs from all neurons  $\mathcal{E}_2$ .
Let  $N^{\prime}:\mathbb{R}^{kn}\to \mathbb{R}$  be a depth-2 network obtained from  $N_1^\prime ,N_2^\prime$  as follows. For  $\mathbf{z}'\in \mathbb{R}^{kn}$  we have  $N^{\prime}(\mathbf{z}^{\prime}) = [1 - N_{1}^{\prime}(\mathbf{z}^{\prime}) - N_{2}^{\prime}(\mathbf{z}^{\prime})]_{+}$ . The network  $N^{\prime}$  has at most  $n^2$  neurons, and parameter magnitudes bounded by  $n^3$  (all bounds are for a sufficiently large  $n$ ). Finally, let  $\tilde{N}:\mathbb{R}^{n^2}\rightarrow \mathbb{R}$  be a depth-2 neural network such that  $\tilde{N} (\tilde{\mathbf{z}}) = N^{\prime}\left(\tilde{\mathbf{z}}_{[kn]}\right)$ .
C.2 Defining the noise magnitudes  $\tau, \omega$  and analyzing the perturbed network under perturbed inputs
In order to use the algorithm  $\mathcal{L}$  w.r.t. some neural network with parameters  $\theta$  and a certain input distribution, we need to implement an examples oracle, such that the examples are drawn from a smoothed input distribution, and labeled according to a neural network with parameters  $\theta +\xi$ , where  $\pmb{\xi}$  is a random perturbation. Specifically, we use  $\mathcal{L}$  with an examples oracle where the input distribution  $\hat{D}$  is obtained from  $\mathcal{D}$  by smoothing, and the labels correspond to a network  $\tilde{N}:\mathbb{R}^{n^2}\to \mathbb{R}$  obtained from  $\tilde{N}$  (w.r.t. an appropriate  $\mathbf{x}\in \{0,1\} ^n$  in the construction of  $N_{1}$ ) by adding a small perturbation to the parameters. The smoothing magnitudes  $\omega ,\tau$  of the inputs and the network's parameters (respectively) are such that the following hold.
We first choose the parameter  $\tau = 1 / \mathrm{poly}(n)$  as follows. Let  $f_{\theta}:\mathbb{R}^{n^2}\to \mathbb{R}$  be any depth-2 neural network parameterized by  $\pmb {\theta}\in \mathbb{R}^r$  for some  $r > 0$  with at most  $n^2$  neurons, and parameter magnitudes bounded by  $n^3$  (note that  $r$  is polynomial in  $n$ ). Then,  $\tau$  is such that with probability at least  $1 - \frac{1}{n}$  over  $\xi \sim \mathcal{N}(\mathbf{0},\tau^2 I_r)$ , we have  $|\xi_i|\leq \frac{1}{10}$  for all  $i\in [r]$ , and the network  $f_{\theta +\xi}$  is such that
for every input  $\tilde{\mathbf{z}}\in \mathbb{R}^{n^2}$  with  $\| \tilde{\mathbf{z}}\| \leq n$  and every neuron we have: Let  $a,b$  be the inputs to the neuron in the computations  $f_{\theta}(\tilde{\mathbf{z}})$  and  $f_{\theta +\xi}(\tilde{\mathbf{z}})$  (respectively), then  $|a - b|\leq \frac{1}{4}$ . Thus,  $\tau$  is sufficiently small, such that w.h.p. adding i.i.d. noise  $\mathcal{N}(0,\tau^2)$  to each parameter does not change the inputs to the neurons by more than  $\frac{1}{4}$ . Note that such an inverse-polynomial  $\tau$  exists, since when the network size, parameter magnitudes, and input size are bounded by some poly(n), then the input to each neuron in  $f_{\theta}(\tilde{\mathbf{z}})$  is poly(n)-Lipschitz as a function of  $\theta$ , and thus it suffices to choose  $\tau$  that implies with probability at least  $1 - \frac{1}{n}$  that  $\| \pmb {\xi}\| \leq \frac{1}{q(n)}$  for a sufficiently large polynomial  $q(n)$  (see Lemma A.7 for details).
Next, we choose the parameter  $\omega = 1 / \mathrm{poly}(n)$  as follows. Let  $f_{\theta}:\mathbb{R}^{n^2}\to \mathbb{R}$  be any depth-2 neural network parameterized by  $\pmb{\theta}$  with at most  $n^2$  neurons, and parameter magnitudes bounded by  $n^3 +\frac{1}{10}$ . Then,  $\omega$  is such that for every  $\mathbf{z}\in \{0,1\}^{n^2}$ , with probability at least  $1 - \exp (-n / 2)$  over  $\zeta \sim \mathcal{N}(\mathbf{0},\omega^2 I_{n^2})$  the following holds for every neuron in the  $f_{\theta}$ : Let  $a,b$  be the inputs to the neuron in the computations  $f_{\theta}(\mathbf{z})$  and  $f_{\theta}(\mathbf{z} + \boldsymbol {\zeta})$  (respectively), then  $|a - b|\leq \frac{1}{4}$ . Thus,  $\omega$  is sufficiently small, such that w.h.p. adding noise  $\mathcal{N}(\mathbf{0},\omega^2 I_{n^2})$  to the input  $\mathbf{z}$  does not change the inputs to the neurons by more than  $\frac{1}{4}$ . Note that such an inverse-polynomial  $\omega$  exists, since when the network size and parameter magnitudes are bounded by some poly(n), then the input to each neuron in  $f_{\theta}(\mathbf{z})$  is poly(n)-Lipschitz as a function of  $\mathbf{z}$ , and thus it suffices to choose  $\omega$  that implies with probability at least  $1 - \exp (-n / 2)$  that  $\| \zeta \| \leq \frac{1}{q(n)}$  for a sufficiently large polynomial  $q(n)$  (see Lemma A.7 for details).
Let  $\tilde{\theta} \in \mathbb{R}^p$  be the parameters of the network  $\tilde{N}$ . Recall that the parameters vector  $\tilde{\theta}$  is the concatenation of all weight matrices and bias terms. Let  $\hat{\theta} \in \mathbb{R}^p$  be the parameters of  $\hat{N}$ , namely,  $\hat{\theta} = \tilde{\theta} + \xi$  where  $\xi \sim \mathcal{N}(\mathbf{0}, \tau^2 I_p)$ . By our choice of  $\tau$  and the construction of the networks  $N_1, N_2$ , with probability at least  $1 - \frac{1}{n}$  over  $\xi$ , for every  $\mathbf{z} \in \{0, 1\}^{n^2}$  the following holds: Let  $\zeta \sim \mathcal{N}(\mathbf{0}, \omega^2 I_{n^2})$  and let  $\hat{\mathbf{z}} = \mathbf{z} + \boldsymbol{\zeta}$ . Then with probability at least  $1 - \exp(-n/2)$  over  $\zeta$  the differences between inputs to all neurons in the computations  $\hat{N}(\hat{\mathbf{z}})$  and  $\tilde{N}(\mathbf{z})$  are at most  $\frac{1}{2}$ . Indeed, w.h.p. for all  $\mathbf{z} \in \{0, 1\}^{n^2}$  the computations  $\tilde{N}(\mathbf{z})$  and  $\hat{N}(\mathbf{z})$  are roughly similar (up to change of  $1/4$  in the input to each neuron), and w.h.p. the computations  $\hat{N}(\mathbf{z})$  and  $\hat{N}(\hat{\mathbf{z}})$  are roughly similar (up to change of  $1/4$  in the input to each neuron). Thus, with probability at least  $1 - \frac{1}{n}$  over  $\boldsymbol{\xi}$ , the network  $\hat{N}$  is such that for every  $\mathbf{z} \in \{0, 1\}^{n^2}$ , we have with probability at least  $1 - \exp(-n/2)$  over  $\boldsymbol{\zeta}$  that the computation  $\hat{N}(\hat{\mathbf{z}})$  satisfies the following properties, where  $\mathbf{z}' := \mathbf{z}_{[kn]}$ :
(Q1) If  $\mathbf{z}' = \mathbf{z}^S$  for some hyperedge  $S$ , then the inputs to  $\mathcal{E}_1$  satisfy:
- If  $P_{\mathbf{x}}(\mathbf{z}^S) = 0$  the inputs to all neurons in  $\mathcal{E}_1$  are at most  $-\frac{1}{2}$ .
- If  $P_{\mathbf{x}}(\mathbf{z}^S) = 1$  there exists a neuron in  $\mathcal{E}_1$  with input at least  $\frac{3}{2}$ .
(Q2) The inputs to  $\mathcal{E}_2$  satisfy:
- If  $\mathbf{z}'$  is an encoding of a hyperedge then the inputs to all neurons  $\mathcal{E}_2$  are at most  $-\frac{1}{2}$ .
- Otherwise, there exists a neuron in  $\mathcal{E}_2$  with input at least  $\frac{3}{2}$ .
C.3 Stating the algorithm  $\mathcal{A}$
Given a sequence  $(S_1, y_1), \ldots, (S_{n^s}, y_{n^s})$ , where  $S_1, \ldots, S_{n^s}$  are i.i.d. random hyperedges, the algorithm  $\mathcal{A}$  needs to distinguish whether  $\mathbf{y} = (y_1, \ldots, y_{n^s})$  is random or that  $\mathbf{y} = (P(\mathbf{x}_{S_1}), \ldots, P(\mathbf{x}_{S_{n^s}})) = (P_{\mathbf{x}}(\mathbf{z}^{S_1}), \ldots, P_{\mathbf{x}}(\mathbf{z}^{S_{n^s}}))$  for a random  $\mathbf{x} \in \{0, 1\}^n$ . Let  $\mathcal{S} = ((\mathbf{z}^{S_1}, y_1), \ldots, (\mathbf{z}^{S_{n^s}}, y_{n^s}))$ .
We use the efficient algorithm  $\mathcal{L}$  in order to obtain distinguishing advantage greater than  $\frac{1}{3}$  as follows. Let  $\xi$  be a random perturbation, and let  $\hat{N}$  be the perturbed network as defined above, w.r.t. the unknown  $\mathbf{x} \in \{0,1\}^n$ . Note that given a perturbation  $\xi$ , only the weights in the second layer of the subnetwork  $N_{1}$  in  $\hat{N}$  are unknown, since all other parameters do not depend on  $\mathbf{x}$ . The algorithm  $\mathcal{A}$  runs  $\mathcal{L}$  with the following examples oracle. In the  $i$ -th call, the oracle first draws  $\mathbf{z}' \in \{0,1\}^{kn}$  such that each component is drawn i.i.d. from a Bernoulli distribution which takes the value 0 with probability  $\frac{1}{n}$ . If  $\mathbf{z}'$  is an encoding of a hyperedge then the oracle replaces  $\mathbf{z}'$  with  $\mathbf{z}^{S_i}$ . Let  $\mathbf{z} \in \{0,1\}^{n^2}$  be such that  $\mathbf{z}_{[kn]} = \mathbf{z}'$ , and the other  $n^2 - kn$  components of  $\mathbf{z}$  are drawn i.i.d. from
a Bernoulli distribution which takes the value 0 with probability  $\frac{1}{n}$ . Note that the vector  $\mathbf{z}$  has the distribution  $\mathcal{D}$ , since replacing an encoding of a random hyperedge by an encoding of another random hyperedge does not change the distribution of  $\mathbf{z}'$ . Let  $\hat{\mathbf{z}} = \mathbf{z} + \boldsymbol{\zeta}$ , where  $\boldsymbol{\zeta} \sim \mathcal{N}(\mathbf{0}, \omega^2 I_{n^2})$ . Note that  $\hat{\mathbf{z}}$  has the distribution  $\hat{\mathcal{D}}$ . Let  $\hat{b} \in \mathbb{R}$  be the bias term of the output neuron of  $\hat{N}$ . The oracle returns  $(\hat{\mathbf{z}}, \hat{y})$ , where the labels  $\hat{y}$  are chosen as follows:
- If  $\mathbf{z}'$  is not an encoding of a hyperedge, then  $\hat{y} = 0$ .
- If  $\mathbf{z}'$  is an encoding of a hyperedge:
- If  $y_{i} = 0$  we set  $\hat{y} = \hat{b}$ .
- If  $y_{i} = 1$  we set  $\hat{y} = 0$ .
Let  $h$  be the hypothesis returned by  $\mathcal{L}$ . Recall that  $\mathcal{L}$  uses at most  $m(n)$  examples, and hence  $S$  contains at least  $n^3$  examples that  $\mathcal{L}$  cannot view. We denote the indices of these examples by  $I = \{m(n) + 1, \dots, m(n) + n^3\}$ , and the examples by  $S_I = \{(\mathbf{z}^{S_i}, y_i)\}_{i \in I}$ . By  $n^3$  additional calls to the oracle, the algorithm  $\mathcal{A}$  obtains the examples  $\hat{S}_I = \{(\hat{\mathbf{z}}_i, \hat{y}_i)\}_{i \in I}$  that correspond to  $S_I$ . Let  $h'$  be a hypothesis such that for all  $\tilde{\mathbf{z}} \in \mathbb{R}^{n^2}$  we have  $h'(\tilde{\mathbf{z}}) = \max \{0, \min \{\hat{b}, h(\tilde{\mathbf{z}})\}\}$ , thus, for  $\hat{b} \geq 0$  the hypothesis  $h'$  is obtained from  $h$  by clipping the output to the interval  $[0, \hat{b}]$ . Let  $\ell_I(h') = \frac{1}{|I|} \sum_{i \in I} (h'(\hat{\mathbf{z}}_i) - \hat{y}_i)^2$ . Now, if  $\ell_I(h') \leq \frac{2}{n}$ , then  $\mathcal{A}$  returns 1, and otherwise it returns 0. We remark that the decision of our algorithm is based on  $h'$  (rather than  $h$ ) since we need the outputs to be bounded, in order to allow using Hoeffding's inequality in our analysis, which we discuss in the next subsection.
C.4 Analyzing the algorithm  $\mathcal{A}$
Note that the algorithm  $\mathcal{A}$  runs in  $\mathrm{poly}(n)$  time. We now show that if  $S$  is pseudorandom then  $\mathcal{A}$  returns 1 with probability greater than  $\frac{2}{3}$ , and if  $S$  is random then  $\mathcal{A}$  returns 1 with probability less than  $\frac{1}{3}$ . To that end, we use similar arguments to the proof of Theorem 3.1.
In Lemma C.1, we show that if  $\mathcal{S}$  is pseudorandom then with probability at least  $\frac{39}{40}$  (over  $\xi \sim \mathcal{N}(\mathbf{0},\tau^2 I_p)$  and  $\zeta_i \sim \mathcal{N}(\mathbf{0},\omega^2 I_{n^2})$  for all  $i \in [m(n)]$ ) the examples  $(\hat{\mathbf{z}}_1,\hat{y}_1),\ldots,(\hat{\mathbf{z}}_{m(n)},\hat{y}_{m(n)})$  returned by the oracle are realized by  $\hat{N}$ . Recall that the algorithm  $\mathcal{L}$  is such that with probability at least  $\frac{3}{4}$  (over  $\xi \sim \mathcal{N}(\mathbf{0},\tau^2 I_p)$ , the i.i.d. inputs  $\hat{\mathbf{z}}_i \sim \hat{\mathcal{D}}$ , and possibly its internal randomness), given a size- $m(n)$  dataset labeled by  $\hat{N}$ , it returns a hypothesis  $h$  such that  $\mathbb{E}_{\hat{\mathbf{z}} \sim \hat{\mathcal{D}}}[(h(\hat{\mathbf{z}}) - \hat{N}(\hat{\mathbf{z}}))^2] \leq \frac{1}{n}$ . Hence, with probability at least  $\frac{3}{4} - \frac{1}{40}$  the algorithm  $\mathcal{L}$  returns such a good hypothesis  $h$ , given  $m(n)$  examples labeled by our examples oracle. Indeed, note that  $\mathcal{L}$  can return a bad hypothesis only if the random choices are either bad for  $\mathcal{L}$  (when used with realizable examples) or bad for the realizability of the examples returned by our oracle. By the definition of  $h'$  and the construction of  $\hat{N}$ , if  $h$  has small error then  $h'$  also has small error, namely,
$$
\underset {\hat {\mathbf {z}} \sim \hat {\mathcal {D}}} {\mathbb {E}} \left[ (h ^ {\prime} (\hat {\mathbf {z}}) - \hat {N} (\hat {\mathbf {z}})) ^ {2} \right] \leq \underset {\tilde {\mathbf {z}} \sim \hat {\mathcal {D}}} {\mathbb {E}} \left[ (h (\hat {\mathbf {z}}) - \hat {N} (\hat {\mathbf {z}})) ^ {2} \right] \leq \frac {1}{n}.
$$
Let  $\hat{\ell}_I(h') = \frac{1}{|I|}\sum_{i\in I}(h'(\hat{\mathbf{z}}_i) - \hat{N} (\hat{\mathbf{z}}_i))^2$ . Recall that by our choice of  $\tau$  we have  $\operatorname*{Pr}[\hat{b} >\frac{11}{10} ]\leq \frac{1}{n}$ . Since,  $(h^{\prime}(\hat{\mathbf{z}}) - \hat{N} (\hat{\mathbf{z}}))^{2}\in [0,\hat{b}^{2}]$  for all  $\hat{\mathbf{z}}\in \mathbb{R}^{n^2}$ , by Hoeffding's inequality, we have for a sufficiently large  $n$  that
$$
\begin{array}{l} \Pr \left[\left| \hat {\ell} _ {I} \left(h ^ {\prime}\right) - \underset {\hat {S} _ {I}} {\mathbb {E}} \hat {\ell} _ {I} \left(h ^ {\prime}\right)\right| \geq \frac {1}{n} \right] = \Pr \left[ \right.\left| \hat {\ell} _ {I} \left(h ^ {\prime}\right) - \underset {\hat {S} _ {I}} {\mathbb {E}} \hat {\ell} _ {I} \left(h ^ {\prime}\right)\right| \geq \frac {1}{n} \left. \right| \hat {b} \leq \frac {1 1}{1 0} \left. \right] \cdot \Pr \left[ \hat {b} \leq \frac {1 1}{1 0} \right] \\ + \Pr \left[ \right.\left| \hat {\ell} _ {I} \left(h ^ {\prime}\right) - \underset {\hat {\mathcal {S}} _ {I}} {\mathbb {E}} \hat {\ell} _ {I} \left(h ^ {\prime}\right)\right| \geq \frac {1}{n} \left. \right\rvert   \hat {b} > \frac {1 1}{1 0} \left. \right] \cdot \Pr \left[ \hat {b} > \frac {1 1}{1 0} \right] \\ \leq 2 \exp \left(- \frac {2 n ^ {3}}{n ^ {2} (1 1 / 1 0) ^ {4}}\right) \cdot 1 + 1 \cdot \frac {1}{n} \\ \leq \frac {1}{4 0}. \\ \end{array}
$$
Moreover, by Lemma C.1,
$$
\Pr \left[ \ell_ {I} \left(h ^ {\prime}\right) \neq \hat {\ell} _ {I} \left(h ^ {\prime}\right) \right] \leq \Pr \left[ \exists i \in I \text {s . t .} \hat {y} _ {i} \neq \hat {N} (\hat {\mathbf {z}} _ {i}) \right] \leq \frac {1}{4 0}.
$$
$\mathbb{E}_{\hat{\mathcal{S}}_I}\hat{\ell}_I(h') = \mathbb{E}_{\hat{\mathbf{z}}\sim \hat{\mathcal{D}}}\left[(h'(\hat{\mathbf{z}}) - \hat{N} (\hat{\mathbf{z}}))^2\right]\leq \frac{1}{n}.$
-  $\left|\hat{\ell}_I(h') - \mathbb{E}_{\hat{\mathcal{S}}_I}\hat{\ell}_I(h')\right| \leq \frac{1}{n}$ .
-  $\ell_I(h') - \hat{\ell}_I(h') = 0$ .
$$
\ell_ {I} (h ^ {\prime}) = \left(\ell_ {I} (h ^ {\prime}) - \hat {\ell} _ {I} (h ^ {\prime})\right) + \left(\hat {\ell} _ {I} (h ^ {\prime}) - \underset {\hat {\mathcal {S}} _ {I}} {\mathbb {E}} \hat {\ell} _ {I} (h ^ {\prime})\right) + \underset {\hat {\mathcal {S}} _ {I}} {\mathbb {E}} \hat {\ell} _ {I} (h ^ {\prime}) \leq 0 + \frac {1}{n} + \frac {1}{n} = \frac {2}{n}.
$$
We now consider the case where  $S$  is random. For an example  $\hat{\mathbf{z}}_i = \mathbf{z}_i + \boldsymbol{\zeta}_i$  returned by the oracle, we denote  $\mathbf{z}_i' = (\mathbf{z}_i)_{[kn]} \in \{0,1\}^{kn}$ . Thus,  $\mathbf{z}_i'$  is the input that the oracle used before adding the  $n^2 - kn$  additional components and adding noise  $\boldsymbol{\zeta}_i$ . Let  $\mathcal{Z}' \subseteq \{0,1\}^{kn}$  be such that  $\mathbf{z}' \in \mathcal{Z}'$  iff  $\mathbf{z}' = \mathbf{z}^S$  for some hyperedge  $S$ . If  $S$  is random, then by the definition of our examples oracle, for every  $i \in [m(n) + n^3]$  such that  $\mathbf{z}_i' \in \mathcal{Z}'$ , we have  $\hat{y}_i = \hat{b}$  with probability  $\frac{1}{2}$  and  $\hat{y}_i = 0$  otherwise. Also, by the definition of the oracle,  $\hat{y}_i$  is independent of  $S_i$ , independent of the  $n^2 - kn$  additional components that were added, and independent of the noise  $\boldsymbol{\zeta}_i \sim \mathcal{N}(\mathbf{0}, \omega^2 I_{n^2})$  that corresponds to  $\hat{\mathbf{z}}_i$ .
If  $\hat{b} \geq \frac{9}{10}$  then for a sufficiently large  $n$  the hypothesis  $h'$  satisfies for each random example  $(\hat{\mathbf{z}}_i, \hat{y}_i) \in \hat{S}_I$  the following:
$$
\begin{array}{l} \operatorname * {P r} _ {(\hat {\mathbf {z}} _ {i}, \hat {y} _ {i})} \left[ (h ^ {\prime} (\hat {\mathbf {z}} _ {i}) - \hat {y} _ {i}) ^ {2} \geq \frac {1}{5} \right] \\ \geq \operatorname * {P r} _ {(\hat {\mathbf {z}} _ {i}, \hat {y} _ {i})} \left[ (h ^ {\prime} (\hat {\mathbf {z}} _ {i}) - \hat {y} _ {i}) ^ {2} \geq \frac {1}{5} \left| \mathbf {z} _ {i} ^ {\prime} \in \mathcal {Z} ^ {\prime} \right] \cdot \operatorname * {P r} [ \mathbf {z} _ {i} ^ {\prime} \in \mathcal {Z} ^ {\prime} ] \right. \\ \geq \operatorname * {P r} _ {(\hat {\mathbf {z}} _ {i}, \hat {y} _ {i})} \left[ (h ^ {\prime} (\hat {\mathbf {z}} _ {i}) - \hat {y} _ {i}) ^ {2} \geq \left(\frac {\hat {b}}{2}\right) ^ {2} \Bigg | \mathbf {z} _ {i} ^ {\prime} \in \mathcal {Z} ^ {\prime} \right] \cdot \operatorname * {P r} [ \mathbf {z} _ {i} ^ {\prime} \in \mathcal {Z} ^ {\prime} ] \\ \geq \frac {1}{2} \cdot \Pr \left[ \mathbf {z} _ {i} ^ {\prime} \in \mathcal {Z} ^ {\prime} \right]. \\ \end{array}
$$
In Lemma A.10, we show that for a sufficiently large  $n$  we have  $\operatorname*{Pr}\left[\mathbf{z}_i' \in \mathcal{Z}'\right] \geq \frac{1}{\log(n)}$ . Hence,
$$
\operatorname * {P r} _ {(\hat {\mathbf {z}} _ {i}, \hat {y} _ {i})} \left[ (h ^ {\prime} (\hat {\mathbf {z}} _ {i}) - \hat {y} _ {i}) ^ {2} \geq \frac {1}{5} \right] \geq \frac {1}{2} \cdot \frac {1}{\log (n)} \geq \frac {1}{2 \log (n)}.
$$
$$
\underset {\dot {S} _ {I}} {\mathbb {E}} \left[ \ell_ {I} (h ^ {\prime}) \right] \geq \frac {1}{5} \cdot \frac {1}{2 \log (n)} = \frac {1}{1 0 \log (n)}.
$$
$$
\operatorname * {P r} \left[ \underset {\hat {\mathcal {S}} _ {I}} {\mathbb {E}} \left[ \ell_ {I} (h ^ {\prime}) \right] \geq \frac {1}{1 0 \log (n)} \right] \geq 1 - \frac {1}{n} \geq \frac {7}{8}.
$$
Since,  $(h'(\hat{\mathbf{z}}) - \hat{y})^2 \in [0, \hat{b}^2]$  for all  $\hat{\mathbf{z}}, \hat{y}$  returned by the examples oracle, and the examples  $\hat{\mathbf{z}}_i$  for  $i \in I$  are i.i.d., then by Hoeffding's inequality, we have for a sufficiently large  $n$  that
$$
\begin{array}{l} \Pr \left[\left| \ell_ {I} \left(h ^ {\prime}\right) - \underset {\hat {S} _ {I}} {\mathbb {E}} \ell_ {I} \left(h ^ {\prime}\right)\right| \geq \frac {1}{n} \right] = \Pr \left[ \right.\left| \ell_ {I} \left(h ^ {\prime}\right) - \underset {\hat {S} _ {I}} {\mathbb {E}} \ell_ {I} \left(h ^ {\prime}\right)\right| \geq \frac {1}{n} \left. \right| \hat {b} \leq \frac {1 1}{1 0} \left. \right] \cdot \Pr \left[ \hat {b} \leq \frac {1 1}{1 0} \right] \\ + \Pr \left[ \right.\left| \ell_ {I} \left(h ^ {\prime}\right) - \underset {\hat {\mathcal {S}} _ {I}} {\mathbb {E}} \ell_ {I} \left(h ^ {\prime}\right)\right| \geq \frac {1}{n} \left. \right\rvert   \hat {b} > \frac {1 1}{1 0} \left. \right] \cdot \Pr \left[ \hat {b} > \frac {1 1}{1 0} \right] \\ \leq 2 \exp \left(- \frac {2 n ^ {3}}{n ^ {2} (1 1 / 1 0) ^ {4}}\right) \cdot 1 + 1 \cdot \frac {1}{n} \\ \leq \frac {1}{8}. \\ \end{array}
$$
Hence, for large enough  $n$ , with probability at least  $1 - \frac{1}{8} - \frac{1}{8} = \frac{3}{4} > \frac{2}{3}$  we have both  $\mathbb{E}_{\hat{\mathcal{S}}_I}[\ell_I(h')] \geq \frac{1}{10\log(n)}$  and  $\left|\ell_I(h') - \mathbb{E}_{\hat{\mathcal{S}}_I}\ell_I(h')\right| \leq \frac{1}{n}$ , and thus
$$
\ell_ {I} \left(h ^ {\prime}\right) \geq \frac {1}{1 0 \log (n)} - \frac {1}{n} > \frac {2}{n}.
$$
Overall, if  $S$  is pseudorandom then with probability greater than  $\frac{2}{3}$  the algorithm  $\mathcal{A}$  returns 1, and if  $S$  is random then with probability greater than  $\frac{2}{3}$  the algorithm  $\mathcal{A}$  returns 0. Thus, the distinguishing advantage is greater than  $\frac{1}{3}$ . This concludes the proof of the theorem. It remains to prove the deferred lemma on the realizability of the examples returned by the examples oracle:
Lemma C.1. If  $S$  is pseudorandom then with probability at least  $\frac{39}{40}$  over  $\xi \sim \mathcal{N}(\mathbf{0},\tau^2 I_p)$  and  $\zeta_i \sim \mathcal{N}(\mathbf{0},\omega^2 I_{n^2})$  for  $i \in [m(n) + n^3]$ , the examples  $(\hat{\mathbf{z}}_1,\hat{y}_1),\dots,(\hat{\mathbf{z}}_{m(n) + n^3},\hat{y}_{m(n) + n^3})$  returned by the oracle are realized by  $\hat{N}$ .
Proof. By our choice of  $\tau$  and  $\omega$  and the construction of  $N_{1}, N_{2}$ , with probability at least  $1 - \frac{1}{n}$  over  $\xi \sim \mathcal{N}(\mathbf{0}, \tau^{2}I_{p})$ , we have  $|\xi_{j}| \leq \frac{1}{10}$  for all  $j \in [p]$ , and for every  $\mathbf{z} \in \{0, 1\}^{n^{2}}$  the following holds: Let  $\zeta \sim \mathcal{N}(\mathbf{0}, \omega^{2}I_{n^{2}})$  and let  $\hat{\mathbf{z}} = \mathbf{z} + \boldsymbol{\zeta}$ . Then with probability at least  $1 - \exp(-n/2)$  over  $\zeta$  the inputs to the neurons  $\mathcal{E}_{1}, \mathcal{E}_{2}$  in the computation  $\hat{N}(\hat{\mathbf{z}})$  satisfy Properties (Q1) and (Q2). Hence, with probability at least  $1 - \frac{1}{n} - (m(n) + n^{3})\exp(-n/2) \geq 1 - \frac{2}{n}$  (for a sufficiently large  $n$ ),  $|\xi_{j}| \leq \frac{1}{10}$  for all  $j \in [p]$ , and Properties (Q1) and (Q2) hold for the computations  $\hat{N}(\hat{\mathbf{z}}_{i})$  for all  $i \in [m(n) + n^{3}]$ . It remains to show that if  $|\xi_{j}| \leq \frac{1}{10}$  for all  $j \in [p]$  and Properties (Q1) and (Q2) hold, then the examples  $(\hat{\mathbf{z}}_{1}, \hat{y}_{1}), \ldots, (\hat{\mathbf{z}}_{m(n) + n^{3}}, \hat{y}_{m(n) + n^{3}})$  are realized by  $\hat{N}$ .
Let  $i \in [m(n) + n^3]$ . We denote  $\hat{\mathbf{z}}_i = \mathbf{z}_i + \boldsymbol{\zeta}_i$ , namely, the  $i$ -th example returned by the oracle was obtained by adding noise  $\boldsymbol{\zeta}_i$  to  $\mathbf{z}_i \in \{0,1\}^{n^2}$ . We also denote  $\mathbf{z}_i' = (\mathbf{z}_i)_{[kn]} \in \{0,1\}^{kn}$ . Since  $|\xi_j| \leq \frac{1}{10}$  for all  $j \in [p]$ , and all incoming weights to the output neuron in  $\tilde{N}$  are  $-1$ , then in  $\hat{N}$  all incoming weights to the output neuron are in  $\left[-\frac{11}{10}, -\frac{9}{10}\right]$ , and the bias term in the output neuron, denoted by  $\hat{b}$ , is in  $\left[\frac{9}{10}, \frac{11}{10}\right]$ . Consider the following cases:
- If  $\mathbf{z}_i'$  is not an encoding of a hyperedge then  $\hat{y}_i = 0$ . Moreover, in the computation  $\hat{N}(\hat{\mathbf{z}}_i)$ , there exists a neuron in  $\mathcal{E}_2$  with output at least  $\frac{3}{2}$  (by Property (Q2)). Since all incoming weights to the output neuron in  $\hat{N}$  are in  $[- \frac{11}{10}, - \frac{9}{10}]$ , and  $\hat{b} \in [\frac{9}{10}, \frac{11}{10}]$ , then the input to the output neuron (including the bias term) is at most  $\frac{11}{10} - \frac{3}{2} \cdot \frac{9}{10} < 0$ , and thus its output is 0.
- If  $\mathbf{z}'$  is an encoding of a hyperedge  $S$ , then by the definition of the examples oracle we have  $S = S_{i}$ . Hence: