SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {g} := g (\bullet / N) = \sum_ {j = 1} ^ {m} \sigma_ {j} \cdot \phi \left(\frac {\rho_ {j} ^ {T}}{N} \cdot \bullet + \eta_ {j}\right), \tag {F.7}
$$
we see for every  $\alpha \in \{-N,\dots,N\} ^n$  that
$$
\tilde {g} (\alpha) = g \left(\frac {\alpha}{N}\right) \left\{ \begin{array}{l l} \geq y _ {\alpha} - \varepsilon = c _ {1} N ^ {- k} - \varepsilon^ {\text {(F . 6)}} > (c _ {1} / 2)   N ^ {- k}, & \text {i f} z _ {\alpha} = 1, \\ \leq y _ {\alpha} + \varepsilon^ {\text {(F . 6)}} <   (c _ {1} / 2)   N ^ {- k}, & \text {i f} z _ {\alpha} = 0. \end{array} \right.
$$
Therefore, we get  $\mathbb{1}\left(\tilde{g} > (c_1 / 2)N^{-k}\right)(\alpha) = z_{\alpha}$  for any  $\alpha \in \{-N, \dots, N\}^n$ . Since the choice of  $z_{\alpha}$  has been arbitrary, it follows that the set
$$
H := \left\{\left. \mathbb {1} \left(\tilde {g} > (c _ {1} / 2) N ^ {- k}\right) \right| _ {\{- N, \dots , N \} ^ {n}}: \tilde {g} \text {o f f o r m (F . 7)} \right\}
$$
shatters the whole set  $\{-N,\dots ,N\} ^n$  . Therefore, we conclude that
$$
\operatorname {V C} (H) \geq (2 N + 1) ^ {n} \geq N ^ {n}.
$$
On the other hand, [4, Theorem 8.11] shows that
$$
\operatorname {V C} (H) \leq 2 m (n + 2) \log_ {2} (6 0 n \cdot N) \leq c _ {3} \cdot m \cdot \ln (N)
$$
with a suitably chosen constant  $c_{3} = c_{3}(n)$ . Here we used that  $N \geq 3$  so that  $\ln (N) \geq \ln (3) > 0$ . Combining those two inequalities yields
$$
m \geq \frac {N ^ {n}}{c _ {3} \cdot \ln (N)}.
$$
Using that  $N \geq c_4 \cdot \varepsilon^{-1/k}$  with  $c_4 \coloneqq c_4(n, k) = \left(\frac{c_1}{2^{k+1}}\right)^{1/k}$  and  $N \leq c_5 \cdot \varepsilon^{-1/k}$  with the definition  $c_5 \coloneqq c_5(n, k) = \left(\frac{c_1}{2}\right)^{1/k}$ , we see that
$$
m \geq \frac {c _ {4} ^ {n} \cdot \varepsilon^ {- n / k}}{c _ {3} \cdot \ln (c _ {5} \cdot \varepsilon^ {- 1 / k})} \geq c _ {6} \cdot \frac {\varepsilon^ {- n / k}}{\ln (1 / \varepsilon)}
$$
with  $c_{6} = c_{6}(n,k) > 0$  chosen appropriately.
Figure 5: Illustration of the function  $f$  considered in the proof of Theorem F.5.
As a corollary, we get a similar result for complex-valued neural networks.
Corollary F.6. Let  $n, k \in \mathbb{N}$  and
$$
\phi : \mathbb {C} \to \mathbb {C}, \quad \phi (z) := \frac {1}{1 + e ^ {- \operatorname {R e} (z)}}.
$$
Then there exists a constant  $c = c(n,k) > 0$  with the following property: If  $\varepsilon \in (0,\frac{1}{2})$  and  $m\in \mathbb{N}$  are such that for every function  $f\in C^k (\Omega_n;\mathbb{C})$  with  $\| f\|_{C^k (\Omega_n;\mathbb{C})}\leq 1$  there exist coefficients  $\rho_1,\ldots ,\rho_m\in \mathbb{C}^n$ $\eta_1,\dots ,\eta_m\in \mathbb{C}$  and  $\sigma_1,\ldots ,\sigma_m\in \mathbb{C}$  satisfying
$$
\left\| f (z) - \sum_ {j = 1} ^ {m} \sigma_ {j} \cdot \phi \left(\rho_ {j} ^ {T} z + \eta_ {j}\right) \right\| _ {L ^ {\infty} \left(\Omega_ {n}; \mathbb {C}\right)} \leq \varepsilon ,
$$
$$
m \geq c \cdot \frac {\varepsilon^ {- 2 n / k}}{\ln \left(1 / \varepsilon\right)}.
$$
Proof. We choose the constant  $c = c(2n,k)$  according to the previous Theorem F.5 and let  $\varphi_{n}$  as in (A.1). Then, let  $\varepsilon \in (0,\frac{1}{2})$  and  $m\in \mathbb{N}$  with the properties assumed in the statement of the corollary. If we then take an arbitrary function  $f\in C^k ([-1,1]^{2n};\mathbb{R})$  with  $\| f\|_{C^k ([-1,1]^{2n};\mathbb{R})}\leq 1$ , we deduce the existence of  $\rho_1,\ldots ,\rho_m\in \mathbb{C}^n$ $\eta_1,\dots ,\eta_m\in \mathbb{C}$  and  $\sigma_1,\ldots ,\sigma_m\in \mathbb{C}$ , such that
$$
\begin{array}{l} \left\| f (x) - \operatorname {R e} \left(\sum_ {j = 1} ^ {m} \sigma_ {j} \cdot \phi \left(\rho_ {j} ^ {T} \cdot \varphi_ {n} (x) + \eta_ {j}\right)\right) \right\| _ {L ^ {\infty} ([ - 1, 1 ] ^ {2 n}; \mathbb {R})} \\ \leq \left\| (f \circ \varphi_ {n} ^ {- 1}) (z) - \sum_ {j = 1} ^ {m} \sigma_ {j} \cdot \phi (\rho_ {j} ^ {T} z + \eta_ {j}) \right\| _ {L ^ {\infty} (\Omega_ {n}; \mathbb {C})} \leq \varepsilon . \\ \end{array}
$$
In the next step, we show that
$$
\mathbb {R} ^ {2 n} \ni x \mapsto \operatorname {R e} \left(\sum_ {j = 1} ^ {m} \sigma_ {j} \cdot \phi (\rho_ {j} ^ {T} \cdot \varphi_ {n} (x) + \eta_ {j})\right)
$$
is a real-valued shallow neural network with  $m$  neurons in the hidden layer and the real sigmoid function as activation function. Then the claim follows using Theorem F.5.
For every  $j \in \{1, \dots, m\}$  we pick a matrix  $\tilde{\rho}_j \in \mathbb{R}^{2n \times 2}$  with the property that one has
$$
\tilde {\rho} _ {j} ^ {T} \cdot \varphi_ {n} ^ {- 1} (z) = \varphi_ {1} ^ {- 1} \left(\rho_ {j} ^ {T} \cdot z\right)
$$
for every  $z\in \mathbb{C}^n$  . This is possible, since this is equivalent to
$$
\tilde {\rho_ {j}} ^ {T} v = \varphi_ {1} ^ {- 1} (\rho_ {j} ^ {T} \varphi_ {n} (v))
$$
for all  $v \in \mathbb{R}^{2n}$ , where the right-hand side is an  $\mathbb{R}$ -linear map  $\mathbb{R}^{2n} \to \mathbb{R}^2$ . Denoting the first column of  $\tilde{\rho}_j$  by  $\hat{\rho}_j$ , we get
$$
\widehat {\rho} _ {j} ^ {T} \cdot \varphi_ {n} ^ {- 1} (z) = \operatorname {R e} \left(\rho_ {j} ^ {T} \cdot z\right) \quad \text {f o r a l l} z \in \mathbb {C} ^ {n}.
$$
Writing  $\gamma$  for the classical real-valued sigmoid function (i.e.  $\gamma(x) = \frac{1}{1 + e^{-x}}$ ), we deduce for arbitrary  $x \in \mathbb{R}^{2n}$  that
$$
\begin{array}{l} \operatorname {R e} \left(\sum_ {j = 1} ^ {m} \sigma_ {j} \cdot \phi (\rho_ {j} ^ {T} \cdot \varphi_ {n} (x) + \eta_ {j})\right) = \operatorname {R e} \left(\sum_ {j = 1} ^ {m} \sigma_ {j} \cdot \gamma (\operatorname {R e} (\rho_ {j} ^ {T} \cdot \varphi_ {n} (x) + \eta_ {j}))\right) \\ = \operatorname {R e} \left(\sum_ {j = 1} ^ {m} \sigma_ {j} \cdot \gamma \left(\widehat {\rho_ {j}} ^ {T} x + \operatorname {R e} \left(\eta_ {j}\right)\right)\right) \\ = \sum_ {j = 1} ^ {m} \operatorname {R e} \left(\sigma_ {j}\right) \cdot \gamma \left(\widehat {\rho_ {j}} ^ {T} x + \operatorname {R e} \left(\eta_ {j}\right)\right), \\ \end{array}
$$
where in the last step we used that  $\gamma$  is real-valued. As noted above, this completes the proof.
Now, we can finally prove Theorem 4.3.
Proof of Theorem 4.3. Let  $\alpha = \frac{2n}{k}$  and choose  $c_{2} = c_{2}(\alpha) = c_{2}(n,k) > 0$  such that the inequality  $\ln (x)\leq c_2\cdot x^{\alpha /2}$  holds for all  $x\geq 1$ . Furthermore, let  $c_{1} = c_{1}(n,k) > 0$  as in Corollary F.6. By choosing  $c = c(n,k) > 0$  sufficiently small, we can ensure that  $\varepsilon_{m}\coloneqq c\cdot (m\cdot \ln (m))^{-k / (2n)}$  satisfies
$$
\ln \left(\frac {c _ {1}}{c _ {2}}\right) + \frac {\alpha}{2} \ln (1 / \varepsilon_ {m}) \geq \frac {\alpha}{4} \cdot \ln (1 / \varepsilon_ {m}) \quad \text {f o r a l l} m \in \mathbb {N} _ {\geq 2}.
$$
By further shrinking  $c = c(n,k) > 0$  if necessary, we may assume
$$
c \cdot (2 \cdot \ln (2)) ^ {- k / (2 n)} <   \frac {1}{2}
$$
and hence  $c \cdot (m \cdot \ln(m))^{-k/(2n)} < \frac{1}{2}$  for all  $m \in \mathbb{N}_{\geq 2}$ . Finally, setting  $c_3 \coloneqq \frac{\alpha}{4}$  and shrinking  $c$  even further, we can arrange that  $c^\alpha < c_1 \cdot c_3$ . Now, assume towards a contradiction that for every  $f \in C^k(\Omega_n; \mathbb{C})$  with  $\|f\|_{C^k(\Omega_n; \mathbb{C})} \leq 1$  there are coefficients  $\rho_1, \ldots, \rho_m \in \mathbb{C}^n, \sigma_1, \ldots, \sigma_m \in \mathbb{C}$  and  $\eta_1, \ldots, \eta_m \in \mathbb{C}$  such that
$$
\left\| f (z) - \sum_ {j = 1} ^ {m} \sigma_ {j} \cdot \phi \left(\rho_ {j} ^ {T} z + \eta_ {j}\right) \right\| _ {L ^ {\infty} \left(\Omega_ {n}; \mathbb {C}\right)} <   c \cdot \left(m \cdot \ln (m)\right) ^ {- k / (2 n)}.
$$
Applying Corollary F.6, we then get
$$
m \geq c _ {1} \cdot \frac {\varepsilon^ {- 2 n / k}}{\ln (1 / \varepsilon)}
$$
with  $\varepsilon = c\cdot (m\cdot \ln m)^{-k / (2n)}\in (0,\frac{1}{2})$  and  $c_{1} = c_{1}(n,k) > 0$ . Recall from the beginning of the proof that  $\alpha \coloneqq 2n / k$  and that  $\ln (x)\leq c_2x^{\alpha /2}$  for every  $x\geq 1$ . We observe
$$
m \geq c _ {1} \cdot \frac {\varepsilon^ {- \alpha}}{\ln (1 / \varepsilon)} \geq \frac {c _ {1}}{c _ {2}} \varepsilon^ {- \alpha / 2},
$$
$$
\ln (m) \geq \ln \left(\frac {c _ {1}}{c _ {2}}\right) + \frac {\alpha}{2} \cdot \ln (1 / \varepsilon) \geq \frac {\alpha}{4} \cdot \ln (1 / \varepsilon) = c _ {3} \cdot \ln (1 / \varepsilon).
$$
Overall we then get
$$
m \geq c _ {1} \cdot \frac {\varepsilon^ {- \alpha}}{\ln (1 / \varepsilon)} = c _ {1} \cdot c ^ {- \alpha} \cdot \frac {m \cdot \ln (m)}{\ln (1 / \varepsilon)} \geq c _ {1} \cdot c _ {3} \cdot c ^ {- \alpha} \cdot \frac {m \cdot \ln (m)}{\ln (m)} = c _ {1} \cdot c _ {3} \cdot c ^ {- \alpha} \cdot m.
$$
Since we chose  $c$  such that  $c^{\alpha} < c_{1}\cdot c_{3}$  we get the desired contradiction.