SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. The transfer to the complex-valued setting works in the same manner as the proof of Theorem 4.1 (see Appendix E.1). We write  $m \coloneqq 2^{2n - 1} - 1$  and note  $2m = 2^{2n} - 2 \leq 2^{2n} - 1$ . We define  $\tilde{a}: C^{\infty, *, 2n} \to \mathbb{R}^{2m}$  and  $\widetilde{M}: \mathbb{R}^{2m} \to C([-1,1]^{2n}; \mathbb{R})$  in the same way as in the proof of Theorem 4.1. Using again the same technique as in the proof of Theorem 4.1, we get
$$
\sup_{\substack{f\in C^{\infty ,*,n}_{\mathbb{C}}\\ \| f\|_{C^{\infty}(\Omega_{n};\mathbb{C})}\leq 1}}\| f - M(\overline{a} (f))\|_{L^{\infty}(\Omega_{n};\mathbb{C})}\geq \sup_{\substack{\tilde{f}\in C^{\infty ,*,2n}\\ \| \tilde{f}\|_{C^{\infty}([-1,1]^{2n};\mathbb{R})}\leq 1}}\left\| \widetilde{f} - \widetilde{M} (\tilde{a} (\tilde{f}))\right\|_{L^{\infty}([-1,1]^{2n};\mathbb{R})}\geq 1,
$$
applying Theorem 5.1 in the last inequality, using  $2m \leq 2^{2n} - 1$ .
We conclude this appendix by adding a note on the constant appearing in our main approximation bound.
Corollary E.7. Let  $n \in \mathbb{N}$  with  $n \geq 2$  and  $\alpha > 0$  and let  $\phi \in C(\mathbb{C};\mathbb{C})$ . Let  $\tilde{c} = \tilde{c}(n,\alpha) > 0$  be such that for every  $m \in \mathbb{N}$  there exists a mapping
$$
\eta : \quad C _ {\mathbb {C}} ^ {\infty , *, n} \to (\mathbb {C} ^ {n}) ^ {m} \times \mathbb {C} ^ {m} \times \mathbb {C} ^ {m}, \quad g \mapsto (\eta_ {1} (g), \eta_ {2} (g), \eta_ {3} (g))
$$
that is continuous with respect to any norm on  $C_{\mathbb{C}}^{\infty, *, n}$  and such that
$$
\| f - \Psi (f) \| _ {L ^ {\infty} (\Omega_ {n}; \mathbb {C})} \leq (\tilde {c} \cdot m) ^ {- \alpha} \cdot \| f \| _ {C ^ {\infty} (\Omega_ {n}; \mathbb {C})},
$$
for every  $f\in C_{\mathbb{C}}^{\infty ,*,n}$  . Here,  $\Psi (f)\in C(\Omega_n;\mathbb{C})$  is given by
Then it necessarily holds  $\tilde{c} \leq 16 \cdot 2^{-n}$ .
Proof. We first assume  $n \geq 4$ . We take  $m = \left\lfloor \frac{2^{2n - 1} - 1}{n + 2} \right\rfloor$  and note that then  $m(n + 2) \leq 2^{2n - 1} - 1$ . Therefore, Corollary E.6 applies and we infer that for each  $\varepsilon \in (0,1)$ , there exists  $f = f_{\varepsilon} \in C_{\mathbb{C}}^{\infty,*n}$  with  $\| f \|_{C^{\infty}(\Omega_n; \mathbb{C})} \leq 1$  and such that
$$
1 - \varepsilon \leq \| f - \Psi (f) \| _ {L ^ {\infty} (\Omega_ {n}; \mathbb {C})} \leq (\tilde {c} \cdot m) ^ {- \alpha} \cdot \| f \| _ {C ^ {\infty} (\Omega_ {n}; \mathbb {C})} \leq (\tilde {c} \cdot m) ^ {- \alpha}.
$$
This then necessarily implies  $\tilde{c} \cdot m \leq 1$  or equivalently  $\tilde{c} \leq 1 / m$ . It therefore suffices to derive a lower bound for  $m$ . Firstly, we note
$$
2 ^ {2 n - 1} = 2 ^ {n - 3} \cdot 2 ^ {n + 2} = 2 ^ {n - 3} \cdot (1 + 1) ^ {n + 2} \geq 2 ^ {n - 3} (n + 3),
$$
where we applied Bernoulli's inequality. Because of  $n \geq 4 \geq 3$ , this yields
$$
2 ^ {2 n - 1} - 1 \geq 2 ^ {n - 3} (n + 3) - 2 ^ {n - 3} = 2 ^ {n - 3} (n + 2).
$$
$$
m \geq \frac {2 ^ {2 n - 1} - 1}{n + 2} - 1 \geq 2 ^ {n - 3} - 1 = 2 ^ {n - 3} (1 - 2 ^ {3 - n}) \geq 2 ^ {n - 4} = \frac {2 ^ {n}}{1 6}.
$$
Here, we used  $n \geq 4$  in the last inequality. An explicit computation shows that the same bounds also hold in the cases  $n = 2$  and  $n = 3$ . This proves the claim.
F Postponed proofs for the optimality results in the case of unrestricted weight selection
F.1 Approximation using Ridge Functions
In this section we prove for  $s \in \mathbb{N}_{\geq 2}$  that every function in  $C^k ([-1,1]^s;\mathbb{R})$  can be uniformly approximated with an error of the order  $m^{-k / (s - 1)}$  using a linear combination of  $m$  so-called ridge functions. In fact, we only consider ridge polynomials, meaning functions of the form
$$
\mathbb {R} ^ {s} \to \mathbb {R}, \quad x \mapsto p (a ^ {T} x)
$$
for a fixed vector  $a \in \mathbb{R}^s$  and a polynomial  $p: \mathbb{R} \to \mathbb{R}$ . Note that this result has already been obtained in a slightly different form in [30, Theorem 1]; namely, it is shown there that the rate of approximation  $m^{-k/(s-1)}$  can be achieved by functions of the form  $\sum_{j=1}^{m} f_j(a_j^T x)$  with  $a_j \in \mathbb{R}^s$  and  $f_j \in L_{\mathrm{loc}}^1(\mathbb{R}^s)$ . We will need the fact that the  $f_j$  can actually be chosen as polynomials and that the vectors  $a_1, \ldots, a_m$  can be chosen independently from the particular function  $f$ . This is shown in the proof of [30], but not stated explicitly. For this reason, and in order to clarify the proof itself and to make the paper more self-contained, we decided to present the proof in this appendix.
Lemma F.1. Let  $m, s \in \mathbb{N}$ . Then we denote by
$$
P_{m}^{s}:= \left\{\mathbb{R}^{s}\to \mathbb{R},\quad x\mapsto \sum_{\substack{\boldsymbol {k}\in \mathbb{N}_{0}^{s}\\ |\boldsymbol {k}|\leq m}}a_{\boldsymbol{k}}x^{\boldsymbol{k}}:  a_{\boldsymbol{k}}\in \mathbb{R}\right\}
$$
the set of real polynomials of degree at most  $m$ . The subset of homogeneous polynomials of degree  $m$  is defined as
$$
H_{m}^{s}:= \left\{\mathbb{R}^{s}\to \mathbb{R},\quad x\mapsto \sum_{\substack{\boldsymbol {k}\in \mathbb{N}_{0}^{s}\\ |\boldsymbol {k}| = m}}a_{\boldsymbol{k}}x^{\boldsymbol{k}}: a_{\boldsymbol{k}}\in \mathbb{R}\right\} .
$$
Then there exists a constant  $c = c(s) > 0$  satisfying
$$
\dim (H _ {m} ^ {s}) \leq c \cdot m ^ {s - 1} \quad \forall m \in \mathbb {N}.
$$
Proof. It is immediate that the set
$$
\left\{\mathbb {R} ^ {s} \rightarrow \mathbb {R}, x \mapsto x ^ {\mathbf {k}}: \mathbf {k} \in \mathbb {N} _ {0} ^ {s}, | \mathbf {k} | = m \right\}
$$
forms a basis of  $H_{m}^{s}$ , hence
$$
\dim (H _ {m} ^ {s}) = \# \left\{\mathbf {k} \in \mathbb {N} _ {0} ^ {s}: | \mathbf {k} | = m \right\}.
$$
This quantity clearly equals the number of possibilities for drawing  $m$  times from a set with  $s$  elements with replacement. Hence, we see
$$
\dim (H _ {m} ^ {s}) = \left( \begin{array}{c} s + m - 1 \\ m \end{array} \right),
$$
see for instance [12, Identity 143]. A further estimation shows
$$
\binom {s + m - 1} {m} = \prod_ {j = 1} ^ {s - 1} \frac {m + j}{j} = \prod_ {j = 1} ^ {s - 1} \left(1 + \frac {m}{j}\right) \leq (1 + m) ^ {s - 1} \leq 2 ^ {s - 1} \cdot m ^ {s - 1}.
$$
Hence, the claim follows with  $c(s) = 2^{s - 1}$ .
A combination of results from [36] together with the fact that it is possible to approximate  $C^k$ -functions using polynomials of degree at most  $m$  with an error of the order  $m^{-k}$ , as shown in Theorem C.15, yields the desired result.
Theorem F.2. Let  $s, k \in \mathbb{N}$  with  $s \geq 2$  and  $r > 0$ . Then there exists a constant  $c = c(s, k) > 0$  with the following property: For every  $m \in \mathbb{N}$  there exist  $a_1, \ldots, a_m \in \mathbb{R}^s \setminus \{0\}$  with  $\| a_j \|_2 = r$ , such that for every function  $f \in C^k([-1,1]^s; \mathbb{R})$  there exist polynomials  $p_1, \ldots, p_m \in P_m^1$  satisfying
$$
\left\| f (x) - \sum_ {j = 1} ^ {m} p _ {j} \left(a _ {j} ^ {T} x\right) \right\| _ {L ^ {\infty} ([ - 1, 1 ] ^ {s}; \mathbb {R})} \leq c \cdot m ^ {- k / (s - 1)} \cdot \| f \| _ {C ^ {k} ([ - 1, 1 ] ^ {s}; \mathbb {R})}.
$$
Proof. We first pick the constant  $c_{1} = c_{1}(s)$  according to Lemma F.1. Then we define the constant  $c_{2} = c_{2}(s) \coloneqq (2s)^{s - 1} \cdot c_{1}(s)$  and let  $M \in \mathbb{N}$  be the largest integer satisfying
$$
c _ {2} \cdot M ^ {s - 1} \leq m.
$$
Here, we assume without loss of generality that  $m \geq c_2$ , which can be justified by choosing  $p_j = 0$  for every  $j \in \{1, \dots, m\}$  if  $m < c_2$ , at the cost of possibly enlarging  $c$ . Note that the choice of  $M$  implies  $c_2 \cdot (2M)^{s-1} \geq c_2 \cdot (M+1)^{s-1} > m$ , and thus
$$
M \geq \frac {1}{2} \cdot c _ {2} ^ {- 1 / (s - 1)} \cdot m ^ {1 / (s - 1)} = c _ {3} \cdot m ^ {1 / (s - 1)} \tag {F.1}
$$
with  $c_{3} = c_{3}(s)\coloneqq 1 / 2\cdot c_{2}^{-1 / (s - 1)}$
Using [36, Proposition 5.9] and Lemma F.1 we can pick  $a_1,\ldots ,a_m\in \mathbb{R}^s\setminus \{0\}$  satisfying
$$
H _ {s (2 M - 1)} ^ {s} = \operatorname {s p a n} \left\{x \mapsto \left(a _ {j} ^ {T} x\right) ^ {s (2 M - 1)}: j \in \{1, \dots , m \} \right\}, \tag {F.2}
$$
where we used that
$$
c _ {1} \cdot (s (2 M - 1)) ^ {s - 1} \leq c _ {1} \cdot (2 s) ^ {s - 1} \cdot M ^ {s - 1} = c _ {2} \cdot M ^ {s - 1} \leq m.
$$
Here we can assume  $\| a_j\| _2 = r$  for every  $j\in \{1,\dots,m\}$  since multiplying each  $a_{j}$  with a positive constant does not change the span in (F.2). From [36, Corollary 5.12] we infer that
$$
P _ {s (2 M - 1)} ^ {s} = \operatorname {s p a n} \left\{x \mapsto \left(a _ {j} ^ {T} x\right) ^ {r}: j \in \{1, \dots , m \}, 0 \leq r \leq s (2 M - 1) \right\}. \tag {F.3}
$$
Let  $f \in C^{k}([-1,1]^{s};\mathbb{R})$ . Then, according to Theorem C.15, there exists a polynomial  $P: \mathbb{R}^s \to \mathbb{R}$  of coordinatewise degree at most  $2M - 1$  satisfying
$$
\left\| f - P \right\| _ {L ^ {\infty} ([ - 1, 1 ] ^ {s}; \mathbb {R})} \leq c _ {4} \cdot M ^ {- k} \cdot \left\| f \right\| _ {C ^ {k} ([ - 1, 1 ] ^ {s}; \mathbb {R})},
$$
where  $c_{4} = c_{4}(s,k) > 0$ . Note that by construction it holds  $P \in P_{s(2M - 1)}^{s}$ . Using (F.3) we deduce the existence of polynomials  $p_1, \ldots, p_m : \mathbb{R} \to \mathbb{R}$  such that
$$
P (x) = \sum_ {j = 1} ^ {m} p _ {j} \left(a _ {j} ^ {T} x\right) \quad \text {f o r a l l} x \in \mathbb {R} ^ {s}.
$$
Combining the previously shown bounds, we get
$$
\begin{array}{l} \left\| f (x) - \sum_ {j = 1} ^ {m} p _ {j} \left(a _ {j} ^ {T} x\right) \right\| _ {L ^ {\infty} ([ - 1, 1 ] ^ {s}; \mathbb {R})} = \| f (x) - P (x) \| _ {L ^ {\infty} ([ - 1, 1 ] ^ {s}; \mathbb {R})} \leq c _ {4} \cdot M ^ {- k} \cdot \| f \| _ {C ^ {k} ([ - 1, 1 ] ^ {s}; \mathbb {R})} \\ \stackrel {\text {(F . 1)}} {\leq} c \cdot m ^ {- k / (s - 1)} \cdot \| f \| _ {C ^ {k} ([ - 1, 1 ] ^ {s}; \mathbb {R})}, \\ \end{array}
$$
as desired. Here, we defined  $c = c(s,k) \coloneqq c_4 \cdot c_3^{-k}$ .
F.2 Proof of Theorem 4.2
Using Theorem F.2, we can prove the following statement for complex-valued  $C^k$ -functions, which will play an important role in the proof of Theorem 4.2.
Proposition F.3. Let  $n, k \in \mathbb{N}$ . Then there exists a constant  $c = c(n, k) > 0$  with the following property: For any  $m \in \mathbb{N}$  there exist complex vectors  $b_1, \ldots, b_m \in \mathbb{C}^n$  with  $\| b_j \|_2 = 1 / \sqrt{2n}$  for  $j = 1, \ldots, m$  and with the property that for any function  $f \in C^k(\Omega_n; \mathbb{C})$  there exist functions  $g_1, \ldots, g_m \in C(\Omega_1; \mathbb{C})$  such that
$$
\left\| f (z) - \sum_ {j = 1} ^ {m} g _ {j} \left(b _ {j} ^ {T} \cdot z\right) \right\| _ {L ^ {\infty} \left(\Omega_ {n}; \mathbb {C}\right)} \leq c \cdot m ^ {- k / (2 n - 1)} \cdot \| f \| _ {C ^ {k} \left(\Omega_ {n}; \mathbb {C}\right)}.
$$
Note that the vectors  $b_{1}, \ldots, b_{m}$  can be chosen independently from the considered function  $f$ , whereas  $g_{1}, \ldots, g_{m}$  do depend on  $f$ .
Proof. Theorem F.2 yields the existence of a constant  $c_{1} = c_{1}(n,k) > 0$  with the property that for any  $m\in \mathbb{N}$  there exist real vectors  $a_1,\ldots ,a_m\in \mathbb{R}^{2n}$  with  $\| a_j\| _2 = 1 / \sqrt{2n}$  such that for any function  $\tilde{f}\in C^k ([-1,1]^{2n};\mathbb{R})$  there exist functions  $\tilde{g}_1,\dots,\tilde{g}_m\in C([-1,1];\mathbb{R})$  satisfying
$$
\left\| \tilde {f} (x) - \sum_ {j = 1} ^ {m} \tilde {g} _ {j} \left(a _ {j} ^ {T} x\right) \right\| _ {L ^ {\infty} ([ - 1, 1 ] ^ {2 n}; \mathbb {R})} \leq c _ {1} \cdot m ^ {- k / (2 n - 1)} \cdot \| \tilde {f} \| _ {C ^ {k} ([ - 1, 1 ] ^ {2 n}; \mathbb {R})}.
$$
We then define the vectors  $b_{1}, \ldots, b_{m} \in \mathbb{C}^{n}$  componentwise via
$$
\left(b _ {j}\right) _ {\ell} := \left(a _ {j}\right) _ {\ell} - i \cdot \left(a _ {j}\right) _ {n + \ell}, \quad \ell \in \{1, \dots , n \}, j \in \{1, \dots , m \}.
$$
First we see  $\| b_{j}\|_{2} = \| a_{j}\|_{2} = 1 / \sqrt{2n}$ . We first consider real-valued functions, i.e.,  $f\in C^{k}(\Omega_{n};\mathbb{R})$ . Let  $\varphi_{n}$  be defined as in (A.1). By the choice of the constant  $c_{1}$  we can find continuous functions  $\tilde{g}_1,\ldots ,\tilde{g}_m\in C$ $[-1,1];\mathbb{R})$  such that
$$
\left\| (f \circ \varphi_ {n}) (x) - \sum_ {j = 1} ^ {m} \tilde {g} _ {j} \left(a _ {j} ^ {T} x\right) \right\| _ {L ^ {\infty} ([ - 1, 1 ] ^ {2 n})} \leq c _ {1} \cdot m ^ {- k / (2 n - 1)} \cdot \| f \circ \varphi_ {n} \| _ {C ^ {k} ([ - 1, 1 ] ^ {2 n}; \mathbb {R})}.
$$
We then define  $g_{j} \in C(\Omega_{1};\mathbb{R})$  by  $g_{j}(z) \coloneqq \tilde{g}_{j}\left(\operatorname {Re}(z)\right)$  for any  $j \in \{1,\dots,m\}$ . For  $z \in \Omega_{n}$  we then have
$$
\begin{array}{l} g _ {j} \left(\left(b _ {j}\right) ^ {T} z\right) = \tilde {g} _ {j} \left(\operatorname {R e} \left(\sum_ {\ell = 1} ^ {n} \left(b _ {j}\right) _ {\ell} \cdot z _ {\ell}\right)\right) \\ = \tilde {g} _ {j} \left(\operatorname {R e} \left(\sum_ {\ell = 1} ^ {n} \left((a _ {j}) _ {\ell} - i \cdot (a _ {j}) _ {n + \ell}\right) \left(\varphi_ {n} ^ {- 1} (z) _ {\ell} + i \cdot \varphi_ {n} ^ {- 1} (z) _ {n + \ell}\right)\right)\right) \\ = \tilde {g} _ {j} \left(\sum_ {\ell = 1} ^ {n} \left[ (a _ {j}) _ {\ell} \varphi_ {n} ^ {- 1} (z) _ {\ell} + (a _ {j}) _ {n + \ell} \varphi_ {n} ^ {- 1} (z) _ {n + \ell} \right]\right) \\ = \tilde {g} _ {j} \left(\left(a _ {j}\right) ^ {T} \cdot \varphi_ {n} ^ {- 1} (z)\right). \tag {F.4} \\ \end{array}
$$
$$
\begin{array}{l} \left\| f (z) - \sum_ {j = 1} ^ {m} g _ {j} \left(b _ {j} ^ {T} z\right) \right\| _ {L ^ {\infty} \left(\Omega_ {n}; \mathbb {R}\right)} = \left\| \left(f \circ \varphi_ {n}\right) (x) - \sum_ {j = 1} ^ {m} g _ {j} \left(b _ {j} ^ {T} \cdot \varphi_ {n} (x)\right) \right\| _ {L ^ {\infty} \left([ - 1, 1 ] ^ {2 n}; \mathbb {R}\right)} \\ \stackrel {\mathrm {(F 4)}} {=} \left\| (f \circ \varphi_ {n}) (x) - \sum_ {j = 1} ^ {m} \tilde {g} _ {j} \left(a _ {j} ^ {T} \cdot x\right) \right\| _ {L ^ {\infty} ([ - 1, 1 ] ^ {2 n}; \mathbb {R})} \\ \leq c _ {1} \cdot m ^ {- k / (2 n - 1)} \cdot \| f \circ \varphi_ {n} \| _ {C ^ {k} ([ - 1, 1 ] ^ {2 n}; \mathbb {R})}. \\ \end{array}
$$
By the above, for  $f \in C^k(\Omega_n; \mathbb{C})$  we can pick functions  $g_1^{\mathrm{Re}}, \dots, g_m^{\mathrm{Re}}, g_1^{\mathrm{Im}}, \dots, g_m^{\mathrm{Im}} \in C(\Omega_1; \mathbb{R})$  satisfying
$$
\left\| \operatorname {R e} (f (z)) - \sum_ {j = 1} ^ {m} g _ {j} ^ {\operatorname {R e}} \left(b _ {j} ^ {T} z\right) \right\| _ {L ^ {\infty} \left(\Omega_ {n}; \mathbb {R}\right)} \leq c _ {1} \cdot m ^ {- k / (2 n - 1)} \cdot \| \operatorname {R e} \left(f \circ \varphi_ {n}\right) \| _ {C ^ {k} \left([- 1, 1 ] ^ {2 n}; \mathbb {R}\right)},
$$
$$
\left\| \operatorname {I m} (f (z)) - \sum_ {j = 1} ^ {m} g _ {j} ^ {\operatorname {I m}} \left(b _ {j} ^ {T} z\right) \right\| _ {L ^ {\infty} (\Omega_ {n}; \mathbb {R})} \leq c _ {1} \cdot m ^ {- k / (2 n - 1)} \cdot \| \operatorname {I m} \left(f \circ \varphi_ {n}\right) \| _ {C ^ {k} ([ - 1, 1 ] ^ {2 n}; \mathbb {R})}.
$$
Defining  $g_{j} \coloneqq g_{j}^{\mathrm{Re}} + i \cdot g_{j}^{\mathrm{Im}}$  yields
$$
\left\| f (z) - \sum_ {j = 1} ^ {m} g _ {j} \left(b _ {j} ^ {T} z\right) \right\| _ {L ^ {\infty} \left(\Omega_ {n}; \mathbb {C}\right)} \leq c _ {1} \cdot \sqrt {2} \cdot m ^ {- k / (2 n - 1)} \cdot \| f \| _ {C ^ {k} \left(\Omega_ {n}; \mathbb {C}\right)},
$$
The special activation function that yields the improved approximation rate of  $m^{-k / (2n - 1)}$  (see Theorem 4.2) is constructed in the following lemma.
Lemma F.4. Let  $\{u_{\ell}\}_{\ell = 1}^{\infty}$  be an enumeration of the set of complex polynomials in  $z$  and  $\overline{z}$  with coefficients in  $\mathbb{Q} + i\mathbb{Q}$ . Then there exists a function  $\phi \in C^{\infty}(\mathbb{C};\mathbb{C})$  with the following properties:
1. For every  $\ell \in \mathbb{N}$  and  $z\in \Omega_1$  one has
$$
\phi (z + 3 \ell) = u _ {\ell} (z).
$$
2.  $\phi$  is non-polyharmonic.
Proof. Let  $\psi \in C^{\infty}(\mathbb{C};\mathbb{R})$  with  $0\leq \psi \leq 1$  and
$$
\left. \psi \right| _ {\Omega_ {1}} \equiv 1, \quad \operatorname {s u p p} (\psi) \subseteq \widetilde {\Omega},
$$
where  $\widetilde{\Omega} := \{z \in \mathbb{C} : |\operatorname{Re}(z)|, |\operatorname{Im}(z)| < \frac{3}{2}\}$ . We then define
$$
\phi := f \cdot \psi + \sum_ {\ell = 1} ^ {\infty} u _ {\ell} (\bullet - 3 \ell) \cdot \psi (\bullet - 3 \ell),
$$
where  $f(z) = e^{\mathrm{Re}(z)}$ . Note that  $\phi$  is smooth since it is a locally finite sum of smooth functions. Furthermore,  $\phi$  is non-polyharmonic on the interior of  $\Omega_1$ , since the calculation in the proof of Proposition A.2 shows for  $z$  in the interior of  $\Omega_1$  and  $\rho : \mathbb{R} \to \mathbb{R}$ ,  $t \mapsto e^t$  that
$$
\left| \partial_ {\mathrm {w i r t}} ^ {m} \overline {{\partial}} _ {\mathrm {w i r t}} ^ {\ell} \phi (z) \right| = \left| \partial_ {\mathrm {w i r t}} ^ {m} \overline {{\partial}} _ {\mathrm {w i r t}} ^ {\ell} f (z) \right| = \frac {1}{2 ^ {m + \ell}} \left| \rho^ {(m + \ell)} (\mathrm {R e} (z)) \right| > 0
$$
for arbitrary  $m, \ell \in \mathbb{N}_0$ . Finally, property (1) follows directly by construction of  $\phi$  because
$$
\left(\widetilde {\Omega} + 3 \ell\right) \cap \left(\widetilde {\Omega} + 3 \ell^ {\prime}\right) = \varnothing
$$
for  $\ell \neq \ell^{\prime}$
Using the properties of the special activation function constructed in Lemma F.4 and applying the approximation result from Proposition F.3 we can now prove Theorem 4.2.
Proof of Theorem 4.2. Let  $\phi$  be the activation function constructed in Lemma F.4. We choose the constant  $c$  according to Proposition F.3. Let  $m\in \mathbb{N}$  and  $f\in C^k (\Omega_n;\mathbb{C})$ . We can without loss of generality assume that  $f\neq 0$ . Again, according to Proposition F.3, we can choose  $\rho_{1},\ldots ,\rho_{m}\in \mathbb{C}^{n}$  with  $\|_2\rho_j\| = 1 / \sqrt{2n}$  and  $g_{1},\dots,g_{m}\in C(\Omega ;\mathbb{C})$  with the property
$$
\left\| f (z) - \sum_ {j = 1} ^ {m} g _ {j} \left(\rho_ {j} ^ {T} z\right) \right\| _ {L ^ {\infty} \left(\Omega_ {n}\right)} \leq c \cdot m ^ {- k / (2 n - 1)} \cdot \| f \| _ {C ^ {k} \left(\Omega_ {n}; \mathbb {C}\right)}.
$$
Recall from Lemma F.4 that  $\{u_{\ell}\}_{\ell = 1}^{\infty}$  is an enumeration of the set of complex polynomials in  $z$  and  $\overline{z}$ . Hence, using the complex version of the Stone-Weierstraβ-Theorem (see for instance [21, Theorem 4.51]), we can pick  $\ell_1,\ldots ,\ell_m\in \mathbb{N}$  such that
$$
\left\| g _ {j} - u _ {\ell_ {j}} \right\| _ {L ^ {\infty} \left(\Omega_ {1}; \mathbb {C}\right)} \leq m ^ {- 1 - k / (2 n - 1)} \cdot \| f \| _ {C ^ {k} \left(\Omega_ {n}; \mathbb {C}\right)} \tag {F.5}
$$
for every  $j \in \{1, \dots, m\}$ . Since  $\phi(\bullet + 3\ell) = u_{\ell}$  on  $\Omega_1$  for each  $\ell \in \mathbb{N}$ , and since  $\rho_j^T z \in \Omega_1$  for  $j \in \{1, \dots, m\}$  and  $z \in \Omega_n$ , we estimate
$$
\begin{array}{l} \left\| f (z) - \sum_ {j = 1} ^ {m} \phi \left(\rho_ {j} ^ {T} z + 3 \ell_ {j}\right) \right\| _ {L ^ {\infty} \left(\Omega_ {n}; \mathbb {C}\right)} \\ \leq \left\| f (z) - \sum_ {j = 1} ^ {m} g _ {j} \left(\rho_ {j} ^ {T} z\right) \right\| _ {L ^ {\infty} \left(\Omega_ {n}; \mathbb {C}\right)} + \sum_ {j = 1} ^ {m} \left\| g _ {j} \left(\rho_ {j} ^ {T} z\right) - \phi \left(\rho_ {j} ^ {T} \cdot z + 3 \ell_ {j}\right) \right\| _ {L ^ {\infty} \left(\Omega_ {n}; \mathbb {C}\right)} \\ \leq c \cdot m ^ {- k / (2 n - 1)} \cdot \| f \| _ {C ^ {k} \left(\Omega_ {n}; \mathbb {C}\right)} + \sum_ {j = 1} ^ {m} \left\| g _ {j} (z) - u _ {\ell_ {j}} (z) \right\| _ {L ^ {\infty} \left(\Omega_ {1}; \mathbb {C}\right)} \\ \stackrel {\text {(F . 5)}} {\leq} c \cdot m ^ {- k / (2 n - 1)} \cdot \| f \| _ {C ^ {k} \left(\Omega_ {n}; \mathbb {C}\right)} + m ^ {- k / (2 n - 1)} \cdot \| f \| _ {C ^ {k} \left(\Omega_ {n}; \mathbb {C}\right)} \\ = (c + 1) \cdot m ^ {- k / (2 n - 1)} \cdot \| f \| _ {C ^ {k} (\Omega_ {n}; \mathbb {C})}. \\ \end{array}
$$
F.3 Proof of Theorem 4.3
As a preparation for the proof of Theorem 4.3, we first prove a similar result in the real-valued setting. We remark that the proof idea is inspired by the proof of [46, Theorem 4].
Theorem F.5. Let  $n, k \in \mathbb{N}$  and
$$
\phi : \quad \mathbb {R} \to \mathbb {R}, \quad \phi (x) := \frac {1}{1 + e ^ {- x}}
$$
be the sigmoid function. Then there exists a constant  $c = c(n,k) > 0$  with the following property: If the numbers  $\varepsilon \in (0,\frac{1}{2})$  and  $m\in \mathbb{N}$  are such that for every function  $f\in C^{k}([-1,1]^{n};\mathbb{R})$  with  $\| f\|_{C^k ([-1,1]^n;\mathbb{R})}\leq 1$  there exist coefficients  $\rho_1,\ldots ,\rho_m\in \mathbb{R}^n$ $\eta_1,\dots,\eta_m\in \mathbb{R}$  and  $\sigma_1,\ldots ,\sigma_m\in \mathbb{R}$  satisfying
$$
\left\| f (x) - \sum_ {j = 1} ^ {m} \sigma_ {j} \cdot \phi \left(\rho_ {j} ^ {T} x + \eta_ {j}\right) \right\| _ {L ^ {\infty} ([ - 1, 1 ] ^ {n}; \mathbb {R})} \leq \varepsilon ,
$$
then necessarily
$$
m \geq c \cdot \frac {\varepsilon^ {- n / k}}{\ln (1 / \varepsilon)}.
$$
Proof. We first pick a function  $\psi \in C^{\infty}(\mathbb{R}^{n};\mathbb{R})$  with the property that  $\psi (0) = 1$  and  $\psi (x) = 0$  for every  $x\in \mathbb{R}^n$  with  $\| x\| _2 > \frac{1}{4}$ . We then choose
$$
c _ {1} = c _ {1} (n, k) := \left(\| \psi \| _ {C ^ {k} ([ - 1, 1 ] ^ {n}; \mathbb {R})}\right) ^ {- 1}.
$$
Now, let  $\varepsilon \in (0,\frac{1}{2})$  and  $m\in \mathbb{N}$  be arbitrary with the property stated in the formulation of the theorem. If  $\varepsilon >\frac{c_1}{2}\cdot \frac{1}{6^k}$ , then  $m\geq c\cdot \frac{\varepsilon^{-n / k}}{\ln(1 / \varepsilon)}$  trivially holds (as long as  $c = c(n,k) > 0$  is sufficiently small). Hence, we can assume that  $\varepsilon \leq \frac{c_1}{2}\cdot \frac{1}{6^k}$ . Now, let  $N$  be the smallest integer with  $N\geq 2$ , for which
$$
\frac {c _ {1}}{2 ^ {k + 1}} \cdot N ^ {- k} \leq \varepsilon .
$$
Note that this implies
$$
N ^ {k} \geq \frac {c _ {1}}{\varepsilon} \cdot \frac {1}{2 ^ {k + 1}} \geq \frac {c _ {1}}{2 ^ {k + 1}} \cdot \frac {2}{c _ {1}} \cdot 6 ^ {k} = 3 ^ {k}
$$
and hence  $N \geq 3$ , whence  $N - 1 \geq 2$ . Therefore, by minimality of  $N$ , and since  $\frac{N}{2} \leq N - 1$  because of  $N \geq 2$ , it follows that
$$
\varepsilon <   \frac {c _ {1}}{2 ^ {k + 1}} \cdot (N - 1) ^ {- k} \leq \frac {c _ {1}}{2 ^ {k + 1}} 2 ^ {k} \cdot N ^ {- k} = \frac {c _ {1}}{2} \cdot N ^ {- k}. \tag {F.6}
$$
Now, for every  $\alpha \in \{-N,\dots,N\} ^n$  pick  $z_{\alpha}\in \{0,1\}$  arbitrary and let  $y_{\alpha}\coloneqq z_{\alpha}c_{1}N^{-k}$ . Define the function
$$
f (x) := \sum_ {\alpha \in \{- N, \dots , N \} ^ {n}} y _ {\alpha} \cdot \psi \left(N x - \alpha\right), \quad x \in \mathbb {R} ^ {n}.
$$
Clearly,  $f \in C^{\infty}(\mathbb{R}^{n};\mathbb{R})$ . Furthermore, since the supports of the functions  $\psi (\bullet -\alpha)$ ,  $\alpha \in \mathbb{Z}^n$  are pairwise disjoint, we see for any multi-index  $\mathbf{k} \in \mathbb{N}_0^n$  with  $|\mathbf{k}| \leq k$  that
$$
\begin{array}{l} \left\| \partial^ {\mathbf {k}} f \right\| _ {L ^ {\infty} ([ - 1, 1 ] ^ {n}; \mathbb {R})} \leq N ^ {| \mathbf {k} |} \cdot \max  _ {\alpha} | y _ {\alpha} | \cdot \left\| \partial^ {\mathbf {k}} \psi \right\| _ {L ^ {\infty} ([ - 1, 1 ] ^ {n}; \mathbb {R})} \\ \leq N ^ {k} \cdot \max  _ {\alpha} | y _ {\alpha} | \cdot \| \psi \| _ {C ^ {k} ([ - 1, 1 ] ^ {n}; \mathbb {R})} \leq 1, \\ \end{array}
$$
so we conclude that  $\| f\|_{C^k ([-1,1]^n;\mathbb{R})}\leq 1$ . Additionally, for any fixed  $\beta \in \{-N,\dots,N\} ^n$  we see
$$
f \left(\frac {\beta}{N}\right) = y _ {\beta},
$$
by choice of  $\psi$ . See also Figure 5 for an illustration of the function  $f$ .
By assumption, we can choose suitable coefficients  $\rho_1,\ldots ,\rho_m\in \mathbb{R}^n$ $\eta_{1},\dots,\eta_{m}\in \mathbb{R}$  and furthermore  $\sigma_1,\ldots ,\sigma_m\in \mathbb{R}$  such that
$$
\left\| f - g \right\| _ {L ^ {\infty} ([ - 1, 1 ] ^ {n}; \mathbb {R})} \leq \varepsilon
$$
$$
g := \sum_ {j = 1} ^ {m} \sigma_ {j} \cdot \phi (\rho_ {j} ^ {T} \cdot \bullet + \eta_ {j}).
$$