Title: Optimal approximation using complex-valued neural networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: complex-valued neural networks

approximation rates
Abstract: Complex-valued neural networks (CVNNs) have recently shown promising empirical success, for instance for increasing the stability of recurrent neural networks and for improving the performance in tasks with complex-valued inputs, such as MRI fingerprinting. While the overwhelming success of Deep Learning in the real-valued case is supported by a growing mathematical foundation, such a foundation is still largely lacking in the complex-valued case. We thus analyze the expressivity of CVNNs by studying their approximation properties. Our results yield the first quantitative approximation bounds for CVNNs that apply to a wide class of activation functions including the popular modReLU and complex cardioid activation functions. Precisely, our results apply to any activation function that is smooth but not polyharmonic on some non-empty open set; this is the natural generalization of the class of smooth and non-polynomial activation functions to the complex setting. Our main result shows that the approximation error scales as $m^{-k/(2n)}$ for $m \to \infty$ where $m$ is the number of neurons, $k$ the smoothness of the target function and $n$ is the (complex) input dimension. Under a natural continuity assumption, we show that this rate is optimal; we further discuss the optimality when dropping this assumption. Moreover, we prove that the problem of approximating $C^k$-functions using continuous approximation methods unavoidably suffers from the curse of dimensionality.

Optimal approximation using complex-valued neural networks

Abstract

Complex-valued neural networks (CVNNs) have recently shown promising empirical success, for instance for increasing the stability of recurrent neural networks and for improving the performance in tasks with complex-valued inputs, such as in MRI fingerprinting. While the overwhelming success of Deep Learning in the real-valued case is supported by a growing mathematical foundation, such a foundation is still largely lacking in the complex-valued case. We thus analyze the expressivity of CVNNs by studying their approximation properties. Our results yield the first quantitative approximation bounds for CVNNs that apply to a wide class of activation functions including the popular modReLU and complex cardioid activation functions. Precisely, our results apply to any activation function that is smooth but not polyharmonic on some non-empty open set; this is the natural generalization of the class of smooth and non-polynomial activation functions to the complex setting. Our main result shows that the error for the approximation of C k -functions scales as m -k/ (2 n ) for m →∞ where m is the number of neurons, k the smoothness of the target function and n is the (complex) input dimension. Under a natural continuity assumption, we show that this rate is optimal; we further discuss the optimality when dropping this assumption. Moreover, we prove that the problem of approximating C k -functions using continuous approximation methods unavoidably suffers from the curse of dimensionality.

1 Introduction

Deep Learning currently predominantly relies on real-valued neural networks, which have led to breakthroughs in fields like image classification or speech recognition [23, 27, 43]. However, recent work has uncovered several application areas in which the use of complex-valued neural networks (CVNNs) leads to better results than the use of their real-valued counterparts. These application areas mainly include tasks where complex numbers inherently occur as inputs of a machine learning model such as Magnetic Resonance Imaging (MRI) [16, 38, 44] and Polarimetric Synthetic Aperture Radar (PolSAR) Imaging [8, 9, 48]. Moreover, CVNNs have been used to improve the stability of recurrent neural networks [5] and have been successfully applied in various other fields [32, 37]. The mathematical theory of these complex-valued neural networks, however, is still in its infancy. There is therefore a great interest in studying CVNNs and in particular in uncovering the differences and commonalities between CVNNs and their real-valued counterparts. A prominent example highlighting the unexpected differences between both network classes is the universal approximation theorem for neural networks, whose most general real-valued version was proven in 1993 [28] (with a more restricted version appearing earlier [17]) and which was recently generalized to the case of CVNNs [45]. The two theorems describe necessary and sufficient conditions on

an activation function which guarantee that arbitrarily wide neural networks of a fixed depth can approximate any continuous function on any compact set arbitrarily well (with respect to the uniform norm). Already here it was shown that complex-valued networks behave significantly different from real-valued networks: While real-valued networks are universal if and only if the activation function is non-polynomial, complex-valued networks with a single hidden layer are universal if and only if the activation function is non-polyharmonic (see below for a definition). Furthermore, there exist continuous activation functions for which deep CVNNs are universal but shallow CVNNs are not, whereas the same cannot happen for real-valued neural networks. This example shows that it is highly relevant to study the properties of CVNNs and to examine which of the fundamental properties of real-valued networks extend to complex-valued networks.

Essentially the only existing quantitative result regarding the approximation-theoretical properties of CVNNs is [14], which provides results for approximating C k -functions by deep CVNNs using the modReLU activation function. However, for real-valued NNs it is known that already shallow NNs can approximate C k -functions at the optimal rate. Precisely, Mhaskar showed in [33] that one can approximate C k -functions on [ -1 , 1] n with an error of order m -k/n as m →∞ , where m is the number of neurons in the hidden layer. Here he assumed that the activation function is smooth on an open interval and that at some point of this interval no derivative vanishes. This is equivalent to the activation function being smooth and non-polynomial on that interval, cf. [20, p. 53].

The present paper shows that a comparable result holds in the setting of complex-valued networks, by proving that one can approximate every function in C k (Ω n ; C ) (where differentiability is understood in the sense of real variables) with an error of the order m -k/ (2 n ) (as m → ∞ ) using shallow complex-valued neural networks with m neurons in the hidden layer. Here we define the cube Ω n := [ -1 , 1] n + i [ -1 , 1] n . The result holds whenever the activation function ϕ : C → C is smooth and non-polyharmonic on some non-empty open set. This is a very natural condition, since for polyharmonic activation functions there exist C k -functions that cannot be approximated at all below some error threshold using shallow neural networks with this activation function [45].

Furthermore, the present paper studies in how far the approximation order of m -k/ (2 n ) is optimal , meaning that an order of m -( k/ 2 n ) -α (where α > 0 ) cannot be achieved. Here it turns out that the derived order of approximation is indeed optimal (even in the class of CVNNs with possibly more than one hidden layer) in the setting that the weight selection is continuous , meaning that the map that assigns to a function f ∈ C k (Ω n ; C ) the weights of the approximating network is continuous with respect to some norm on C k (Ω n ; C ) . This continuity assumption is natural since typical learning algorithms such as (stochastic) gradient descent use samples f ( x j ) of the target function and then apply continuous operations to them to update the network weights.

We investigate this optimality result further by dropping the continuity assumption and constructing two special smooth and non-polyharmonic activation functions with the first one having the property that the order of approximation can indeed be strictly improved via a discontinuous selection of

the related weights. For the second activation function we show that the order of m -k/ (2 n ) cannot be improved, even if one allows a discontinuous weight selection. This in particular shows that in the given generality of arbitrary smooth, non-polyharmonic activation functions, the upper bound O ( m -k/ (2 n ) ) cannot be improved, even for a possibly discontinuous choice of the weights. An overview of the approximation bounds proven in this paper can be found in Table 1.

Moreover, we analyze the tractability (in terms of the input dimension n ) of the considered problem of approximating C k -functions using neural networks. Theorem 5.1 shows that one necessarily needs a number of parameters exponential in n to obtain a non-trivial approximation error. To the best of our knowledge, Theorem 5.1 is the first result showing that the problem of approximating C k -functions using continuous approximation methods is intractable (in terms of the input dimension n ).

1.1 Related Work

Real-valued neural networks. By now, the approximation properties of real-valued neural networks are quite well-studied (cf. [10, 28, 33, 40, 41, 46, 47] and the references therein). We here only discuss a few papers that are most closely related to the present work.

In [33], Mhaskar analyzes the rate of approximation of shallow real-valued neural networks for target functions of regularity C k . Our results can be seen as the generalization of [33] to the complex setting. Our proofs rely on several techniques from [33]; however, significant modifications are required to make the proofs work for general smooth non-polyharmonic functions.

One of the first papers to observe that neural networks with general (smooth) activation function can be surprisingly expressive is [31] where it was shown that a neural network of constant size can be universal. One of the activation functions in Section 4 is based on a similar idea.

The importance of distinguishing between continuous and discontinuous weight selection (which in our setting is discussed in Section 4) was observed for ReLU-networks in [47].

The paper [25] shows that neural network approximation is not continuous in the following sense: The best approximating neural network Φ( f ) of a given size does not depend continuously on f ∈ C k . This result, however, is not in conflict with our results: We want to assign to any C k -function f a network ˜ Φ( f ) that approximates f below the error threshold m -k/ (2 n ) . The network ˜ Φ( f ) , however, does not have to coincide with the best approximating network Φ( f ) .

Complex-valued neural networks. When it comes to general literature about mathematical properties of complex-valued neural networks, surprisingly little work can be found. The Universal Approximation Theorem for Complex-Valued Neural Networks [45] has already been mentioned above. In particular, it has been shown that shallow CVNNs are universal if and only if the activation function ϕ is not polyharmonic. Thus, the condition assumed in the present paper (that ϕ should be smooth, but not polyharmonic) is quite natural.

Regarding quantitative approximation results for CVNNs, the only existing work of which we are aware is [14], analyzing the approximation capabilities of deep CVNNs where the modReLU is used as activation function. Since the modReLU satisfies our condition regarding the activation function, the present work can be seen as an improvement to [14]. Precisely, (i) we consider general activation functions, including but not limited to the modReLU, (ii) we improve the order of approximation by a log factor, and (iii) we show that this order of approximation can be achieved using shallow networks instead of the deep networks used in [14]. We stress that our proof techniques differ significantly from the ones applied in [14]: The arguments in [14] take their main ideas from [46] making heavy use of the specific definition of the modReLU. In contrast, since we consider quite general activation functions, we necessarily follow a much more general approach following the ideas from [33].

2 Preliminaries

Shallow complex-valued neural networks. In this paper we mainly consider so-called shallow complex-valued neural networks, meaning complex-valued neural networks with a single hidden layer. Precisely, we consider functions of the form

ϕ

(

ρ

T

1

z

+

η

1

)

ϕ

(

ρ

T

m

z

+

η

m

)

σ

T

Φ(

z

) =

∑

m

j

=1

σ

j

ϕ

(

ρ

T

j

·

z

+

η

j

)

· · ·

· · ·

n

inputs

First layer

with ρ 1 , ..., ρ m ∈ C n , σ 1 , ..., σ m , η 1 , ..., η m ∈ C and an activation function ϕ : C → C . Here, m ∈ N denotes the number of neurons of the network and we write v T for the transpose of a vector v .

To simplify the formulation of the results, we introduce the following notation: We write F ϕ n,m for the set of first layers of shallow complex-valued neural networks with activation function ϕ , with n input neurons and m hidden neurons, meaning

Hence, each shallow CVNN can be written as σ T Φ with σ ∈ C m and Φ ∈ F ϕ n,m ; see Figure 1 for a graphical representation of a shallow CVNN.

Approximation. The paper aims to analyze the approximation of C k -functions on the complex cube

using shallow CVNNs. Here, we say that a function f : Ω n → C is in C k (Ω n ; C ) if and only if f is k times continuously differentiable on Ω n , where the derivative is to be understood in the sense of real variables, i.e., in the sense of interpreting f as a function [ -1 , 1] 2 n → R 2 and taking usual real derivatives. We further define the C k -norm of a function f ∈ C k (Ω n ; C ) as

for any function g : Ω n → C . Note that we write N = { 1 , 2 , 3 , ... } and N 0 = { 0 } ∪ N . Using the previously introduced notation, we thus seek to bound the worst-case approximation error, i.e.,

Wirtinger calculus and polyharmonic functions. For a function f : C → C which is differentiable in the sense of real variables at a point z 0 ∈ C we define its Wirtinger derivatives at z 0 as

Here, ∂ ∂x and ∂ ∂y denote the usual partial derivatives in the sense of real variables. We extend this definition to multivariate functions defined on open subsets of C n by considering coordinatewise Wirtinger derivatives.

A function f : U → C , where U ⊆ C is an open set, is called smooth if it is differentiable arbitrarily many times (in the sense of real variables). We write f ∈ C ∞ ( U ; C ) in that case. Moreover, f is called polyharmonic (on U ) if it is smooth and if there exists m ∈ N 0 satisfying

-

4

-

2

0

2

4

-

5

0

5

0

2

4

6

Re(

z

)

Im(

z

)

|

σ

modReLU

,

-

1

(

z

)

|

-

4

-

2

0

2

4

-

5

0

5

0

5

10

Re(

z

)

Im(

z

)

|

card(

z

)

|

Here, ∆ := ∂ 2 ∂x 2 + ∂ 2 ∂y 2 = 4 ∂ wirt ∂ wirt denotes the usual Laplace-Operator.

The following Proposition 2.1 describes a property of non-polyharmonic functions which is crucial for proving the approximation results of this paper.

̸

Proposition 2.1. Let ∅ = U ⊆ C be an open set and let ϕ ∈ C ∞ ( U ; C ) be non-polyharmonic. Then for every M ∈ N 0 there exists a point z M ∈ U satisfying

̸

The proof of Proposition 2.1 is an application of the Baire category theorem; see Appendix B.2.

Important complex activation functions. We briefly discuss in how far two commonly used complex activation functions satisfy our assumptions: The modReLU proposed in [5] and the complex cardioid used in [44] for MRI fingerprinting where the performance could be significantly improved using complex-valued neural networks. The modReLU is defined as

where b < 0 is a fixed parameter. The complex cardioid is defined as

Here, ∢ z = θ ∈ R denotes the polar angle of a complex number z = re iθ , where we define ∢ 0 := 0 ; see Figure 2 for plots of the absolute value of the two functions.

Both functions are smooth and non-polyharmonic on a non-empty open subset of C , which is proven in Appendix A.2. Furthermore, they are both continuous on C . Therefore, our approximation bounds established in Theorems 3.1 and 3.2 in particular apply to those two functions.

3 Main results

In this section we state the main results of this paper and provide proof sketches for them. Detailed proofs of the two statements can be found in Appendices B.3 and C.3.

First we show in Theorem 3.1 that it is possible to approximate any complex polynomial in z and z arbitrarily well using shallow CVNNs with the size of the networks only depending on the degree of the polynomial (not on the desired approximation accuracy). Using this result we can prove the main approximation bound, Theorem 3.2, by first approximating a given C k -function using a polynomial in z and z and then approximating this polynomial using Theorem 3.1.

For m,n ∈ N let

denote the space of all complex polynomials on C n in z and z of componentwise degree at most m . Here, we are summing over all multi-indices m , ℓ ∈ N n 0 with m j , ℓ j ≤ m for every j ∈ { 1 , ..., n } and use the notation

The space P n m is finite-dimensional; hence, it makes sense to talk about bounded subsets of P n m without specifying a norm.

̸

Theorem 3.1. Let m,n ∈ N , ε > 0 and ϕ : C → C be smooth and non-polyharmonic on an open set ∅ = U ⊆ C . Let P ′ ⊆ P n m be bounded and set N := (4 m +1) 2 n . Then there exists a first layer Φ ∈ F ϕ n,N with the following property: For each polynomial p ∈ P ′ there exists σ ∈ C N , such that

Sketch of proof. For any multi-indices m , ℓ ∈ N n 0 an inductive argument shows for every fixed z ∈ Ω n and b ∈ C that

Here, ∂ m wirt and ∂ ℓ wirt denote the multivariate Wirtinger derivatives with respect to w according to the multi-indices m and ℓ , respectively. Evaluating this at w = 0 and taking b ∈ C such that none of the mixed Wirtinger derivatives of ϕ at b up to a sufficiently high order vanish (where such a b exists by Proposition 2.1) shows that we can rewrite

The mixed Wirtinger derivatives can by definition be expressed as linear combinations of usual partial derivatives. Those partial derivatives can be approximated using a generalized version of divided differences : If g ∈ C k (( -r, r ) s ; R ) and p ∈ N s 0 with | p | ≤ k we have

See Appendix B.1 for a proof of this approximation. Note that when one takes g ( w ) = ϕ ( w T z + b ) , the right-hand side of (3.2) is a shallow neural network, as a function of z .

Combining (3.1) and (3.2) yields the desired result; see Appendix B.3 for the details.

It is crucial that the size of the networks considered in Theorem 3.1 is independent of the approximation accuracy ε . Moreover, the first layer Φ can be chosen independently of the particular polynomial p . Only the weights σ connecting hidden layer and output neuron have to be adapted to p .

The final approximation result is as follows. Its full proof can be found in Appendix C.3.

̸

Theorem 3.2. Let n, k ∈ N . Then there exists a constant c = c ( n, k ) > 0 with the following property: For any activation function ϕ : C → C that is smooth and non-polyharmonic on an open set ∅ = U ⊆ C and for any m ∈ N there exists a first layer Φ ∈ F ϕ n,m with the following property: For any f ∈ C k (Ω n ; C ) there exist coefficients σ = σ ( f ) ∈ C m , such that

Furthermore, the map f ↦→ σ ( f ) is a continuous linear operator with respect to the L ∞ -norm.

Sketch of proof. By splitting f into real and imaginary part we may assume that f is real-valued. Fourier-analytical results (recalled in Appendix C.1) imply that each C k -function f can be well approximated by a linear combination of (multivariate) Chebyshev polynomials . Precisely, we have

where P is given via the formula

Here, the functions T k are multivariate versions of Chebyshev polynomials and V m k ( f ) are continuous linear functionals in f . The constant c > 0 only depends on n and k . See Appendix C.1 for a rigorous proof of this approximation property. Approximating the polynomials T k by neural networks according to Theorem 3.1 yields the desired claim; see Appendix C.3 for the details.

Remark 3.3 . Theorem 3.1 can not only be used to derive approximation rates for the approximation of C k -functions but can be applied to any function class that is well approximable by algebraic polynomials. For example, it can be used to prove an order of approximation of ν -m 1 / (2 n ) / 17 for the approximation of functions f : Ω n → C that can be holomorphically extended onto some polyellipse in C 2 n . The parameter ν > 1 specifies the size of this polyellipse. We refer the interested reader to Appendix D for detailed definitions, statements and proofs for this fact.

4 Optimality of the derived approximation rate

In this section we discuss the optimality of the approximation rate proven in Theorem 3.2. We first deduce from general results by DeVore et al. [19] that the rate is optimal in the setting that the map which assigns to a function f ∈ C k (Ω n ; C ) the weights of the approximating network is continuous, as is the case in Theorem 3.2. However, it might be possible to achieve a better degree of approximation if this map is not required to be continuous, which is discussed in the second part of this section. Proofs for all the statements in this section are given in Appendices E.1, F.2 and F.3.

Continuous weight selection. We begin by considering the case of continuous weight selection. As mentioned in the introduction, this is a natural assumption, since in classical training algorithms such as (S)GD, continuous operations based on samples f ( x j ) are used to adjust the weights.

In [19, Theorem 4.2] a lower bound of m -k/s is established for the rate of approximating functions f of Sobolev regularity W k, ∞ in the following very general setting: The set of functions that is used for approximation can be parametrized using m (real) parameters and the map that assigns to f the parameters of the approximating function is continuous with respect to some norm on W k, ∞ ([ -1 , 1] s ; R ) . A detailed version of the proof of that statement (for C k instead of W k, ∞ ) is contained in Appendix E.1. A careful transfer of this result to the complex-valued setting yields the following theorem (see also Appendix E.1).

Theorem 4.1. Let n, k ∈ N . Then there exists a constant c = c ( n, k ) > 0 with the following property: For any m ∈ N , any map a : C k (Ω n ; C ) → C m that is continuous with respect to some norm on C k (Ω n ; C ) and any map M : C m → C (Ω n ; C ) we have

The interpretation of Theorem 4.1 is as follows: If one approximates C k -functions on Ω n using a set of functions that can be parametrized using m (complex) parameters and one assumes that the weight assignment is continuous, one cannot achieve a better rate of approximation than m -k/ (2 n ) . As a special case it can be deduced that the approximation rate is at most m -k/ (2 n ) when approximating C k -functions using shallow CVNNs with m parameters under continuous weight assignment (see Corollary E.3). One can show that this even holds for deep CVNNs. Hence, for continuous weight selection the rate proven in Theorem 3.2 is optimal, even in the class of (possibly) deep networks.

Discontinuous weight selection. When we drop the continuity assumption on the selection of the weights, the behavior of the optimal approximation rate is more subtle. Precisely, we show that there are activation functions for which the rate of approximation can be improved to m -k/ (2 n -1) . On the other hand, we also show that there are activation functions for which an improvement of the approximation rate (up to logarithmic factors) is not possible.

Theorem 4.2. There exists a function ϕ ∈ C ∞ ( C ; C ) which is non-polyharmonic with the following additional property: For every k ∈ N and n ∈ N there exists a constant c = c ( n, k ) > 0 such that

-2

-1

1

2

3

4

5

6

7

-2

-1

1

e

Re(

z

)

u

1

u

2

Re

Im

for any m ∈ N and f ∈ C k (Ω n ; C ) there is a first layer Φ ∈ F ϕ n,m and a vector σ ∈ C m such that

Sketch of proof. The function ϕ is constructed in the following way: Take a countable dense subset { u ℓ } ℓ ∈ N of C (Ω 1 ; C ) , for instance the set of all polynomials in z and z with coefficients in Q + i Q . Define ϕ in a way such that

for every z ∈ Ω 1 and ℓ ∈ N . Furthermore, to ensure that ϕ is non-polyharmonic, let ϕ ( z ) = e Re( z ) for every z ∈ Ω 1 . The smoothness of ϕ can be accomplished by multiplying with a smooth bump function ; see Lemma F.4 for details. The construction of ϕ is illustrated in Figure 3.

Let then f ∈ C k (Ω n ; C ) be arbitrary. General results from the theory of ridge functions [22] show that there exist b 1 , ..., b m ∈ C n and g 1 , ..., g m ∈ C (Ω 1 ; C ) such that

see Proposition F.3 and Appendix F.1 for a detailed proof of this fact. Approximating the functions g j by suitable functions u ℓ j and expressing those functions via ϕ ( · +3 ℓ j ) yields the claim.

The preceding theorem showed that there exists an activation function for which the rate in Theorem 3.2 can be strictly improved, if one allows a discontinuous weight selection. In contrast, the following theorem shows for a certain (quite natural) activation function that the rate m -k/ (2 n ) cannot be improved (up to logarithmic factors), even if one allows a discontinuous weight assignment.

Theorem 4.3. Let n, k ∈ N and

Then ϕ is smooth but non-polyharmonic. Furthermore, there exists a constant c = c ( n, k ) > 0 with the following property: For any m ∈ N ≥ 2 there exists a function f ∈ C k (Ω n ; C ) with ∥ f ∥ C k (Ω n ; C ) ≤ 1 , such that for every Φ ∈ F ϕ n,m and σ ∈ C m we have

Sketch of proof. The idea of the proof is based on that of [46, Theorem 4] but instead of the bound for the VC dimension of ReLU networks used in [46], we will employ a bound for the VC dimension stated in [4, Theorem 8.11] using the real sigmoid function. For a detailed introduction to the concept of the VC dimension and related topics, see for instance [39, Chapter 6].

A technical reduction from the complex to the real case (see Appendix F.3) shows that it suffices to show the following: If ε ∈ (0 , 1 2 ) and m ∈ N are such that for every f ∈ C k ([ -1 , 1] n ; R ) with ∥ f ∥ C k ([ -1 , 1] n ; R ) ≤ 1 there exists a real-valued shallow network N with γ ( x ) = 1 1+ e -x as activation function satisfying ∥ f -N∥ L ∞ ([ -1 , 1] n ; R ) ≤ ε , then necessarily

where the constant c only depends on n and k .

To show that the latter claim holds, we assume that ε and m have the property stated above. Take N ∈ N such that N -k ≍ ε and consider the grid

For every α ∈ G we pick a number z α ∈ { 0 , 1 } arbitrarily and construct a map f ∈ C ∞ ([ -1 , 1] n ; R ) satisfying f ( α ) = z α for very α ∈ G . Scaling of f to ˜ f ensures ∥ ˜ f ∥ C k ([ -1 , 1] n ; R ) ≤ 1 , but then ˜ f ( α ) = c 0 · z α · N -k where c 0 = c 0 ( n, k ) > 0 . By assumption, we can infer the existence of a shallow real-valued neural network N with γ as activation function and m hidden neurons satisfying ∥ ˜ f -N∥ L ∞ ([ -1 , 1] n ; R ) ≤ ε . But this shows

with a constant ˜ c = ˜ c ( n, k ) > 0 . Since the z α are arbitrary, it follows that the set

✶ shatters the whole grid G . This yields VC( H ) ≥ |G| = (2 N +1) n . On the other hand, the bound provided by [4, Theorem 8.11] for linear threshold networks yields VC( H ) ≲ m · ln( N ) . Combining the two bounds and using N -k ≍ ε yields the claim.

5 Tractability of the considered problem in terms of the input dimension

In this section we discuss the tractability (in terms of the input dimension n ) of the considered problem, i.e., the dependence of the approximation error on n . We show a novel result stating that, assuming a continuous weight selection, the problem of approximating C k -functions is intractable , i.e., that the number of neurons that is required to achieve a non-trivial approximation error is necessarily exponential in n . In the literature this is referred to as the curse of dimensionality . The proof of the theorem combines ideas from [19] and [35] and is contained in Appendix E.2.

Theorem 5.1. Let s ∈ N . With ∥ · ∥ C k ([ -1 , 1] s ; R ) defined similarly to (2.1) , we write

for any function f ∈ C ∞ ([ -1 , 1] s ; R ) and denote by C ∞ , ∗ ,s the set of all f ∈ C ∞ ([ -1 , 1] s ; R ) for which this expression is finite. Let a : C ∞ , ∗ ,s → R 2 s -1 be continuous with respect to some norm on C ∞ , ∗ ,s and moreover, let M : R 2 s -1 → C ([ -1 , 1] s ; R ) be an arbitrary map. Then it holds

Note that Theorem 5.1 is formulated for real-valued functions but can be transferred to the complexvalued setting (see Corollary E.6). We decided to include the real-valued statement because it is expected to be of greater interest in the community than the complex-valued analog. Moreover, we stress that Theorem 5.1 is not limited to the class of shallow neural networks but refers to any function class that is parametrizable using finitely many parameters (in particular, e.g., the class of neural networks with possibly more than one hidden layer).

We now examine in what way the constant c appearing in Theorem 3.2 suffers from the curse of dimensionality. To this end, it is convenient to rewrite the result from Theorem 3.2 as

where the constant ˜ c = ˜ c ( n, k ) > 0 is independent of m . Writing the result in that way, one sees immediately that, if one seeks to have a worst-case approximation error of less than ε > 0 , it is

sufficient to take m = ⌈ 1 ˜ c · ε -(2 n ) /k ⌉ neurons in the hidden layer of the network. Corollary E.7 shows that it necessarily holds ˜ c ≤ 16 · 2 -n and therefore, the constant ˜ c unavoidably suffers from the curse of dimensionality. An analysis of the constant (where we refer to Appendices C.1 to C.3 for the details) shows that in our case we can establish the bound ˜ c ( n, k ) ≥ exp( -C · n 2 ) · k -4 n with an absolute constant C > 0 . We remark that, since the constant suffers from the curse of dimensionality in any case, we have not put much effort into optimizing the constant; there is therefore probably ample room for improvement.

6 Limitations

To conduct a comprehensive evaluation of machine learning algorithms, one must analyze the questions of approximation, generalization, and optimization through training algorithms. The present paper, however, only focuses on the aspect of approximation. Analyzing if the proven approximation rate can be attained with learning algorithms such as (stochastic) gradient descent falls outside the scope of this paper. Furthermore, the examination of approximation rates under possibly discontinuous weight assignment is not yet fully resolved by our results. It is an open question which rate is optimally achievable in that case, depending on the choice of the activation function, and specifically in distinguishing between shallow and deep networks. We want to mention the two following points which indicate that this is a quite subtle question:

For deep NNs (with more than two hidden layers) with general smooth activation function, it is not possible to derive any non-trivial lower bounds in the setting of unrestricted weight assignment, since there exists an activation function with the property that NNs of constant size using this activation function can approximate any continuous function to arbitrary precision (see [31, Theorem 4]). Note that [31] considers real-valued NNs, but the results can be transferred to CVNNs with a suitable choice of the activation function.

In the real-valued case, fully general lower bounds for the approximation capabilities of shallow NNs have been derived by using results from [22] regarding the approximation properties of so-called ridge functions, i.e., functions of the form ∑ m j =1 ϕ j ( ⟨ a j , x ⟩ ) with a j ∈ R d and each ϕ j : R → R . It is an interesting problem to generalize these results to higher-dimensional ridge functions of the form ∑ m j =1 ϕ j ( A j x ) , where each ϕ j : R s → R and A j ∈ R s × d . This would imply lower bounds for shallow CVNNs. However, such a generalization seems to be highly non-trivial and is outside the scope of the present paper.

7 Conclusion

This paper analyzes error bounds for the approximation of complex-valued C k -functions by means of complex-valued neural networks with smooth and non-polyharmonic activation functions. It is demonstrated that complex-valued neural networks with these activation functions achieve the identical approximation rate as real-valued networks that employ smooth and non-polynomial activation functions. This is an important theoretical finding, since CVNNs are on the one hand more restrictive than real-valued neural networks (since the mappings between layers should be C -linear and not just R -linear), but on the other hand more versatile, since the activation function is a mapping from C to C (i.e., from R 2 → R 2 ) rather than from R to R as in the real case. Additionally, it is established that the proven approximation rate is optimal if one assumes a continuous weight selection. In summary, if one focuses on the approximation rate for C k -functions, CVNNs have the same excellent approximation properties as real-valued networks.

The behavior of the approximation rate for unrestricted weight selection is more subtle. It is shown that a rate of m -k/ (2 n -1) can be achieved for certain activation functions (Theorem 4.2) but in general, one cannot improve on the rate that is attainable for continuous weight selection (Theorem 4.3).

While the proven approximation rate is optimal under the assumption of continuous weight selection, the involved constants suffer from the curse of dimensionality. Section 5, however, shows that this is inevitable in the given setting.

Such theoretical approximation results contribute to the mathematical understanding of Deep Learning. The remarkable approximation-theoretical properties of neural networks can be seen as one reason why neural networks provide outstanding results in many applications.

Acknowledgments. PG and FV acknowledge support by the German Science Foundation (DFG) in the context of the Emmy Noether junior research group VO 2594/1-1. FV acknowledges support by the Hightech Agenda Bavaria.

References

M. Abramowitz and I. A. Stegun, eds. Handbook of mathematical functions: with formulas, graphs, and mathematical tables . 9. Dover print. Dover books on mathematics. New York, NY: Dover Publ, 2013. ISBN: 978-0-486-61272-0.

B. Adcock, S. Brugiapaglia, and C. G. Webster. Sparse polynomial approximation of high-dimensional functions . Computational science and engineering 25. Philadelphia: Society for Industrial and Applied Mathematics, 2022. 292 pp. ISBN: 978-1-61197-687-8.

C. D. Aliprantis and K. C. Border. Infinite dimensional analysis: a hitchhiker's guide . 3rd ed. Berlin ; New York: Springer, 2006. ISBN: 978-3-540-29586-0.

M. Anthony and P. L. Bartlett. Neural network learning: theoretical foundations . Cambridge ; New York, NY: Cambridge University Press, 1999. ISBN: 978-0-521-57353-5.

M. Arjovsky, A. Shah, and Y. Bengio. 'Unitary evolution recurrent neural networks'. In: International conference on machine learning . PMLR. 2016, pp. 1120-1128. URL: ❤/a116/a116♣ ✿ ✴ ✴ ♣/a114♦❝❡❡❞✐♥❣/a115✳♠❧/a114✳ ♣/a114❡/a115/a115✴✈✹✽✴❛/a114❥♦✈/a115❦②✶✻✳♣❞❢ .

K. E. Atkinson. An introduction to numerical analysis . 2nd ed. New York: Wiley, 1989. ISBN: 978-0471-62489-9.

M. B. Balk. Polyanalytic functions . Mathematical research 63. Berlin: Akademie Verlag, 1991. ISBN: 978-3-05-501292-1.

J. A. Barrachina, C. Ren, C. Morisseau, G. Vieillard, and J.-P. Ovarlez. 'Complex-valued vs. real-valued neural networks for classification perspectives: An example on non-circular data'. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE. 2021, pp. 2990-2994.

J. A. Barrachina, C. Ren, C. Morisseau, G. Vieillard, and J.-P. Ovarlez. 'Comparison between equivalent architectures of complex-valued and real-valued neural networks - Application on polarimetric SAR image segmentation'. In: Journal of Signal Processing Systems 95.1 (2023), pp. 57-66.

A. R. Barron. 'Universal approximation bounds for superpositions of a sigmoidal function'. In: IEEE Transactions on Information theory 39.3 (1993), pp. 930-945. DOI: ✶✵✳✶✶✵✾✴✶✽✳✷✺✻✺✵✵ .

H. Bauer. Measure and integration theory . De Gruyter studies in mathematics 26. Berlin ; New York: W. de Gruyter, 2001. 230 pp. ISBN: 978-3-11-016719-1.

A. Benjamin and J. J. Quinn. Proofs that really count: the art of combinatorial proof . Dolciani mathematical expositions. Washington, DC: Mathematical Association of America, 2003. ISBN: 978-088385-333-7.

D. Berend and T. Tassa. 'Improved bounds on Bell numbers and on moments of sums of random variables'. In: Probability and Mathematical Statistics 30.2 (2010), pp. 185-205.

A. Caragea, D. G. Lee, J. Maly, G. Pfander, and F. Voigtlaender. 'Quantitative Approximation Results for Complex-Valued Neural Networks'. In: SIAM Journal on Mathematics of Data Science 4.2 (2022), pp. 553-580. DOI: ✶✵✳✶✶✸✼✴✷✶▼✶✹✷✾✺✹✵ .

H. H. Çevik, Y. E. Acar, and M. Çunka¸ s. 'Day Ahead Wind Power Forecasting Using Complex Valued Neural Network'. In: 2018 International Conference on Smart Energy Systems and Technologies (SEST) . 2018, pp. 1-6. DOI: ✶✵✳✶✶✵✾✴❙❊❙❚✳✷✵✶✽✳✽✹✾✺✻✸✼ .

E. Cole, J. Cheng, J. Pauly, and S. Vasanawala. 'Analysis of deep complex-valued convolutional neural networks for MRI reconstruction and phase-focused applications'. In: Magnetic resonance in medicine 86.2 (2021), pp. 1093-1109.

G. Cybenko. 'Approximation by superpositions of a sigmoidal function'. In: Mathematics of Control, Signals, and Systems 2.4 (Dec. 1989), pp. 303-314. DOI: ✶✵✳✶✵✵✼✴❇❋✵✷✺✺✶✷✼✹ .

K. Deimling. Nonlinear Functional Analysis . Dover books on mathematics. Dover Publications, 2013. ISBN: 978-3-662-00549-1.

R. A. DeVore, R. Howard, and C. Micchelli. 'Optimal nonlinear approximation'. In: Manuscripta Mathematica 63.4 (Dec. 1989), pp. 469-478. DOI: ✶✵✳✶✵✵✼✴❇❋✵✶✶✼✶✼✺✾ .

W. F. Donoghue. Distributions and Fourier transforms . OCLC: 45151. New York: Academic Press, 1969. ISBN: 978-0-12-220650-4.

G. B. Folland. Real analysis: modern techniques and their applications . 2nd ed. Pure and applied mathematics. New York: Wiley, 1999. ISBN: 978-0-471-31716-6.

Y. Gordon, V. Maiorov, M. Meyer, and S. Reisner. 'On the Best Approximation by Ridge Functions in the Uniform Norm'. In: Constructive Approximation 18.1 (Jan. 1, 2001), pp. 61-85. DOI: ✶✵✳✶✵✵✼✴/a115✵✵✸✻✺✲ ✵✵✶✲✵✵✵✾✲✺ .

A. Graves, A.-R. Mohamed, and G. Hinton. 'Speech recognition with deep recurrent neural networks'. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing . 2013, pp. 6645-6649. DOI: ✶✵✳✶✶✵✾✴■❈❆❙❙/a80✳✷✵✶✸✳✻✻✸✽✾✹✼ .

W. P. Johnson. 'The Curious History of Faa di Bruno's Formula'. In: The American Mathematical Monthly 109.3 (Mar. 2002), p. 217. DOI: ✶✵✳✷✸✵✼✴✷✻✾✺✸✺✷ .

P. C. Kainen, V. K˚ urková, and A. Vogt. 'Approximation by neural networks is not continuous'. In: Neurocomputing 29.1 (1999), pp. 47-56. DOI: ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✶✻✴❙✵✾✷✺✲✷✸✶✷✭✾✾✮✵✵✶✶✶✲ ✸ .

L. Kaup and B. Kaup. Holomorphic Functions of Several Variables: An Introduction to the Fundamental Theory . De Gruyter, Dec. 1983. ISBN: 978-3-11-004150-7. DOI: ✶✵✳✶✺✶✺✴✾✼✽✸✶✶✵✽✸✽✸✺✵ .

A. Krizhevsky, I. Sutskever, and G. E. Hinton. 'ImageNet classification with deep convolutional neural networks'. In: Communications of the ACM 60.6 (May 2017), pp. 84-90. DOI: ✶✵✳✶✶✹✺✴✸✵✻✺✸✽✻ .

M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken. 'Multilayer feedforward networks with a nonpolynomial activation function can approximate any function'. In: Neural Networks 6.6 (Jan. 1993), pp. 861-867. DOI: ✶✵✳✶✵✶✻✴❙✵✽✾✸✲✻✵✽✵✭✵✺✮✽✵✶✸✶✲✺ .

G. Lorentz. Approximation of Functions . Approximation of Functions. Holt, Rinehart and Winston, 1966.

V. Maiorov. 'On Best Approximation by Ridge Functions'. In: Journal of Approximation Theory 99.1 (July 1999), pp. 68-94. DOI: ✶✵✳✶✵✵✻✴❥❛/a116❤✳✶✾✾✽✳✸✸✵✹ .

V. Maiorov and A. Pinkus. 'Lower bounds for approximation by MLP neural networks'. In: Neurocomputing 25.1 (1999), pp. 81-91. DOI: ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✶✻✴❙✵✾✷✺✲✷✸✶✷✭✾✽✮✵✵✶✶✶✲ ✽ .

A. Marseet and F. Sahin. 'Application of complex-valued convolutional neural network for next generation wireless networks'. In: 2017 IEEE Western New York Image and Signal Processing Workshop (WNYISPW) . 2017, pp. 1-5. DOI: ✶✵✳✶✶✵✾✴❲◆❨■/a80❲✳✷✵✶✼✳✽✸✺✻✷✻✵ .

H. N. Mhaskar. 'Neural Networks for Optimal Approximation of Smooth and Analytic Functions'. In: Neural Computation 8.1 (Jan. 1996), pp. 164-177. DOI: ✶✵✳✶✶✻✷✴♥❡❝♦✳✶✾✾✻✳✽✳✶✳✶✻✹ .

C. Muscalu and W. Schlag. Classical and multilinear harmonic analysis. Volume 1 . OCLC: 887742576. Cambridge, UK: Cambridge University Press, 2013. ISBN: 978-0-521-88245-3.

E. Novak and H. Wo´ zniakowski. 'Approximation of infinitely differentiable multivariate functions is intractable'. In: Journal of Complexity 25.4 (2009), pp. 398-404. DOI: ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✶✻✴❥✳ ❥❝♦✳✷✵✵✽✳✶✶✳✵✵✷ .

A. Pinkus. Ridge functions . Cambridge: Cambridge University Press, 2016. ISBN: 978-1-107-12439-4.

Y.-D. Qu, R.-Q. Zhang, S.-Q. Shen, J. Yu, and M. Li. 'Entanglement Detection with Complex-Valued Neural Networks'. In: International Journal of Theoretical Physics 62.9 (2023), p. 206.

H. El-Rewaidy, U. Neisius, J. Mancio, S. Kucukseymen, J. Rodriguez, A. Paskavitz, B. Menze, and R. Nezafat. 'Deep complex convolutional network for fast reconstruction of 3D late gadolinium enhancement cardiac MRI'. In: NMR in Biomedicine 33.7 (2020), e4312.

S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: from theory to algorithms . New York, NY, USA: Cambridge University Press, 2014. 397 pp. ISBN: 978-1-107-05713-5.

Z. Shen, H. Yang, and S. Zhang. 'Deep Network Approximation Characterized by Number of Neurons'. In: Communications in Computational Physics 28.5 (2020), pp. 1768-1811. DOI: ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴ ✶✵✳✹✷✵✽✴❝✐❝♣✳❖❆✲✷✵✷✵✲✵✶✹✾ .

J. W. Siegel and J. Xu. 'Approximation rates for neural networks with general activation functions'. In: Neural Networks 128 (2020), pp. 313-321. DOI: ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✶✻✴❥✳♥❡✉♥❡/a116✳✷✵✷✵✳✵✺✳ ✵✶✾ .

P. Stein. 'A Note on the Volume of a Simplex'. In: The American Mathematical Monthly 73.3 (Mar. 1966), p. 299. DOI: ✶✵✳✷✸✵✼✴✷✸✶✺✸✺✸ .

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 'Attention is All you Need'. In: Advances in Neural Information Processing Systems . Ed. by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. Vol. 30. Curran Associates, Inc., 2017. URL: ❤/a116/a116♣/a115 ✿ ✴✴♣/a114♦❝❡❡❞✐♥❣/a115✳♥❡✉/a114✐♣/a115✳❝❝✴♣❛♣❡/a114❴❢✐❧❡/a115✴♣❛♣❡/a114✴✷✵✶✼✴ ❢✐❧❡✴✸❢✺❡❡✷✹✸✺✹✼❞❡❡✾✶❢❜❞✵✺✸❝✶❝✹❛✽✹✺❛❛✲/a80❛♣❡/a114✳♣❞❢ .

P. Virtue, S. X. Yu, and M. Lustig. 'Better than real: Complex-valued neural nets for MRI fingerprinting'. In: 2017 IEEE International Conference on Image Processing (ICIP) . Beijing: IEEE, Sept. 2017, pp. 3953-3957. ISBN: 978-1-5090-2175-8. DOI: ✶✵✳✶✶✵✾✴■❈■/a80✳✷✵✶✼✳✽✷✾✼✵✷✹ .

F. Voigtlaender. 'The universal approximation theorem for complex-valued neural networks'. In: Applied and Computational Harmonic Analysis (Dec. 2022), S1063520322001014. DOI: ✶✵✳✶✵✶✻✴❥✳❛❝❤❛✳ ✷✵✷✷✳✶✷✳✵✵✷ .

D. Yarotsky. 'Error bounds for approximations with deep ReLU networks'. In: Neural Networks 94 (Oct. 2017), pp. 103-114. DOI: ✶✵✳✶✵✶✻✴❥✳♥❡✉♥❡/a116✳✷✵✶✼✳✵✼✳✵✵✷ .

D. Yarotsky and A. Zhevnerchuk. 'The phase diagram of approximation rates for deep neural networks'. In: Advances in neural information processing systems 33 (2020), pp. 13005-13015. URL: ❤/a116/a116♣/a115✿✴✴ ♣/a114♦❝❡❡❞✐♥❣/a115✳♥❡✉/a114✐♣/a115✳❝❝✴♣❛♣❡/a114✴✷✵✷✵✴❢✐❧❡✴✾✼✾❛✸❢✶✹❜❛❡✺✷✸❞❝✺✶✵✶❝✺✷✶✷✵❝✺✸✺❡✾✲/a80❛♣❡/a114✳ ♣❞❢ .

Z. Zhang, H. Wang, F. Xu, and Y.-Q. Jin. 'Complex-Valued Convolutional Neural Network and Its Application in Polarimetric SAR Image Classification'. In: IEEE Transactions on Geoscience and Remote Sensing 55.12 (2017), pp. 7177-7188. DOI: ✶✵✳✶✶✵✾✴❚●❘❙✳✷✵✶✼✳✷✼✹✸✷✷✷ .

A Notation, Wirtinger derivatives and special activation functions

A.1 Notation and Wirtinger derivatives

Throughout the paper, multi-indices (i.e., elements of N n 0 ) are denoted using boldface. For m ∈ N n 0 we have the usual notation | m | = ∑ n j =1 m j . For a number m ∈ N 0 and another multi-index p ∈ N n 0 we write m ≤ m iff m j ≤ m for every j ∈ { 1 , ..., n } and m ≤ p iff m j ≤ p j for every j ∈ { 1 , ..., n } . Furthermore we write

for two multi-indices p , r ∈ N n 0 with r ≤ p . For a complex vector z ∈ C n we write

For a point x ∈ R n and r > 0 we define

with ∥ · ∥ denoting the usual Euclidean distance. This definition is analogously transferred to C n .

C n is canonically isomorphic to R 2 n by virtue of the isomorphism

The Wirtinger derivatives defined in Section 2 have the following properties that we are going to use, which can be found for example in [26, E.1a]. Here, we assume that U ⊆ C is open and f ∈ C 1 ( U ; C ) .

∂ wirt and ∂ wirt are both C -linear operators on the set C 1 ( U ; C ) .

f is complex-differentiable in z ∈ U iff ∂ f ( z ) = 0

wirt and in this case the equality )

holds true, with f ′ ( z ) denoting the complex derivative of f at z .

We have the conjugation rules

If g ∈ C 1 ( U ; C ) , the following product rules for Wirtinger derivatives hold for every z ∈ U :

This product rule is not explicitly stated in [26] but follows easily from the product rule for ∂ ∂x and ∂ ∂y .

If V ⊆ C is an open set and g ∈ C 1 ( V ; C ) with g ( V ) ⊆ U , then the following chain rules for Wirtinger derivatives hold true:

If f ∈ C 2 ( U ; C ) then we have

for every z ∈ U , with ∆ denoting the usual Laplace-Operator ∆ = ∂ (2 , 0) + ∂ (0 , 2) (cf. [7, equation (1.7)]).

For an open set U ⊆ C n , a function f ∈ C k ( U ; C ) and a multi-index ℓ ∈ N n 0 with | ℓ | ≤ k we write ∂ ℓ wirt f and ∂ ℓ wirt f for the iterated Wirtinger derivatives according to the multi-index ℓ .

Proposition A.1. Let U ⊆ C n be an open set and f ∈ C k ( U ; C ) . Then, identifying f with the function f ◦ φ n with φ n as in (A.1) , for any multi-indices m , ℓ ∈ N n 0 with | m + ℓ | ≤ k we have the representation

with complex numbers b p ∈ C only depending on p , m and ℓ and writing p = ( p ′ , p ′′ ) for the concatenation of the multi-indices p ′ and p ′′ ∈ N n 0 . In particular, the coefficients do not depend on f .

Proof. The proof is by induction over m and ℓ . We first assume m = 0 and show the claim for all ℓ ∈ N n 0 with | ℓ | ≤ k . In the case ℓ = 0 there is nothing to show, so we assume the claim to be true for fixed ℓ ∈ N n 0 with | ℓ | < k and take j ∈ { 1 , ..., n } arbitrarily. Then we get

as was to be shown.

Since we have shown the case m = 0 we may assume the claim to be true for fixed m , ℓ ∈ N n 0 with | m + ℓ | < k . Then we deduce

Hence the claim follows using the principle of mathematical induction.

Proposition A.1 is technical but crucial: In the course of the paper we will need to approximate Wirtinger derivatives of certain functions. In fact, however, we will approximate real derivatives and take advantage of the fact that Wirtinger derivatives are linear combinations of these.

A.2 Concrete examples of activation functions

In this section we analyze concrete activation functions that are commonly used in practical applications of complex-valued neural networks. We are going to show that those activation functions are smooth and non-polyharmonic on some non-empty open subset of C . Our first result analyzes a family of 'real activation functions interpreted as complex activation functions'.

Proposition A.2. Let ρ ∈ C ( R ; R ) be non-polynomial and let ψ : C → C be defined as

Then ψ is smooth and non-polyharmonic.

Proof. Since ψ depends only on the real-, resp. imaginary part of the input, we see directly from the definition of the Wirtinger derivatives that

Hence we see for arbitrary m,ℓ ∈ N 0 that

̸

Since ρ is non-polynomial we can choose a real number x , such that ρ ( n ) ( x ) = 0 for all n ∈ N 0 (cf. for instance [20, p. 53]). Thus, ∂ m wirt ∂ ℓ wirt ψ ( z ) = 0 for all m,ℓ ∈ N 0 , whenever z ∈ C with Re( z ) = x , or Im( z ) = y , respectively.

̸

Next, we consider the modReLU which was defined in Section 2.

Theorem A.3. Let b ∈ ( -∞ , 0) . Writing σ = σ modReLU ,b , one has for every z ∈ C with | z | + b > 0 the identity

for every m ∈ N 0 and ℓ ∈ N 0 . Here, the numbers q m,ℓ are non-zero and rational. Furthermore, note that all cases for choices of m and ℓ are covered, by observing that we can either have the case m ≥ ℓ +1 (where either m ≥ 2 or m = 1 , ℓ = 0 ) or m ≤ ℓ +1 , where the latter case is again split into ℓ = 0 and ℓ ≥ 1 .

̸

Proof. We first calculate certain Wirtinger derivatives for z = 0 . First note

and similarly

for any m ∈ N . Using the product rule for Wirtinger derivatives (see Appendix A.1), we see

for any m ∈ N and ℓ ∈ N 0 and furthermore

for m,ℓ ∈ N , and finally

for m ∈ N and ℓ ∈ N 0 .

Having proven those three identities, we can start with the actual computation. We first fix m = 0 and prove the claimed identity by induction over ℓ . The case ℓ = 0 follows directly from the definition of the modReLU and for ℓ = 1 , we note

which is the claimed form. Then, using induction, we compute

so that the case m = 0 is complete.

Now we deal with the case m ≤ ℓ +1 . The case ℓ = 0 and m = 1 is proven by computing

so we can assume ℓ > 0 . Since we already dealt with the case m = 0 , we can inductively assume the claim to be true for a fixed m ≤ ℓ . Then we compute

which is of the desired form. Note that (A.3) is indeed applicable because ℓ -m +1 ≥ 1 .

Finally, we consider the case where m ≥ ℓ +1 and m ≥ 2 . The case m = ℓ +1 has already been shown. Using induction, we see

so the proof is complete.

From Theorem A.3 we can now deduce that the modReLU is smooth and non-polyharmonic on some non-empty open subset of C .

Corollary A.4. Let b ∈ ( -∞ , 0) and z ∈ C with | z | > -b . Then we have

̸

for all m,ℓ ∈ N 0 . In particular, σ modReLU ,b is smooth and non-polyharmonic on the set

Furthermore, σ modReLU ,b is continuous on the entire complex plane.

̸

Proof. The first part follows from Theorem A.3 by noting that if | z | > -b we have in particular | z | > -b/ 2 and | z | > 0 . For the second part, we first note that σ modReLU ,b is trivially continuous in z ∈ C if | z | = -b . Hence, we assume | z | = -b . Take any sequence ( z j ) j ∈ N with z j → z as j →∞ and note

This shows the continuity of σ modReLU ,b .

Next we analyze the complex cardioid, which was defined in Section 2.

̸

Theorem A.5. For any z ∈ C with z = 0 and any m,ℓ ∈ N 0 we have

̸

̸

Here, the numbers a m,ℓ and b m,ℓ are non-zero and rational. Furthermore, note that all cases for possible choices of m and ℓ are covered: The case m ≤ ℓ is split into ℓ = 0 and ℓ = 0 . The case m = ℓ +1 is split into m = 1 and m> 1 . Then, the case m ≥ ℓ +2 remains.

̸

Proof. For the following we always assume z ∈ C with z = 0 . Then we can apply the identity cos( ∢ z ) = Re( z ) | z | , so we can rewrite

This establishes the case m = ℓ = 0 . Next, we compute

Using induction, we derive

so the claim has been shown if m = 0 . If we now fix any ℓ ∈ N and assume that the claim holds true for some m ∈ N 0 with m<ℓ , we get

so the claim holds true if m ≤ ℓ .

The case m = ℓ +1 is split into the case m = 1 and m> 1 . If m = 1 , then ℓ = 0 and we compute

and similarly

We deduce

If m> 1 , we get

Next, we deal with the case m = ℓ +2 : Here, we see

If we assume the claim to be true for a fixed m ≥ ℓ +2 , we get

Hence, using induction, we have proven the claimed identity.

The statement regarding the non-polyharmonicity of the complex cardioid is formulated in the following corollary.

Corollary A.6. For every z ∈ C with z / ∈ R ∪ i R and every m,ℓ ∈ N 0 we have

̸

In particular, card is smooth and non-polyharmonic on C \ ( R ∪ i R ) . Futhermore, card is continuous on the entire complex plane.

̸

Proof. We start with the first part of the corollary. For the following, let z ∈ C with z / ∈ R ∪ i R , i.e., Re( z ) = 0 and Im( z ) = 0 . Using the definition of card , we see

̸

̸

and thus, card( z ) = 0 since Im( z ) = 0 . In the case m ≤ ℓ = 0 we use Theorem A.5 to get the following chain of implications:

̸

̸

̸

holds, and thus ∂ m wirt ∂ ℓ wirt card( z ) = 0 for z / ∈ R ∪ i R .

For the case m = ℓ +1 = 1 , note

̸

and thus ∂ wirt card( z ) = 0 since z / ∈ R .

For m = ℓ +1 > 1 , we see by considering the real- and imaginary parts that

̸

̸

and hence ∂ ℓ +1 wirt ∂ ℓ wirt card( z ) = 0 , since a m,ℓ = 0 = b m,ℓ by Theorem A.5.

̸

̸

Therefore, it remains to consider the case m ≥ ℓ +2 . But here we easily see

̸

and hence ∂ m wirt ∂ ℓ wirt card( z ) = 0 . Since all cases have been considered, the claim follows.

Regarding the second part of the corollary, first note that card is trivially continuous on C \ { 0 } Further note that

as z → 0 , showing the continuity of card on the entire complex plane C .

B Postponed proofs concerning the approximation of polynomials

B.1 Divided Differences

Divided differences are well-known in numerical mathematics as they are for example used to calculate the coefficients of an interpolation polynomial in its Newton representation. In our case, we are interested in divided differences since they can be used to obtain a generalization of the classical mean-value theorem for differentiable functions: Given an interval I ⊆ R and n +1 pairwise distinct data points x 0 , ..., x n ∈ I as well as an n -times differentiable real-valued function f : I → R , there exists ξ ∈ ( min { x 0 , ..., x n } , max { x 0 , ..., x n } ) such that

where the left-hand side is a divided difference of f (defined below). The classical mean-value theorem is obtained in the case n = 1 . Our goal in this section is to generalize this result to a multivariate setting by considering divided differences in multiple variables. Such a generalization is probably well-known, but since we could not locate a convenient reference and to make the paper more self-contained, we provide a proof.

Let us first define divided differences formally. Given n +1 data points ( x 0 , y 0 ) , ..., ( x n , y n ) ∈ R × R with pairwise distinct x k , we define the associated divided differences recursively via

If the data points are defined by a function f (i.e. y k = f ( x k ) for all k ∈ { 0 , ..., n } ), we write

We first consider an alternative representation of divided differences, the so-called Hermite-GenocchiFormula . To state it, we introduce the notation Σ k to denote a certain k -dimensional simplex.

Definition B.1. Let s ∈ N . Then we define

We further set Σ 0 := { 0 } ⊆ R . Denoting by λ s the s -dimensional Lebesgue-measure (and by λ 0 the counting measure), the identity λ s (Σ s ) = 1 s ! holds true; see for instance [42] and note that the case s = 0 can be seen directly.

In the following, we consider integrals over the sets Σ s . These integrals are always formed with respect to the measure λ s . However, we only write, e.g., ∫ Σ s f ( x ) dx , with the implicit understanding that the integral is formed with respect to the measure λ s . Using this convention, we can now consider the alternative representation of divided differences.

.

Lemma B.2 (Hermite-Genocchi-Formula) . Let k ∈ N 0 . For real numbers a, b ∈ R with a < b , a function f ∈ C k ([ a, b ]; R ) (where C 0 ([ a, b ]; R ) denotes the set of continuous functions) and pairwise distinct x 0 , ..., x k ∈ [ a, b ] , the divided difference of f at the data points x 0 , ..., x k satisfies the identity

Proof. The case k = 0 follows directly from [ x 0 ] f = f ( x 0 ) . For the case k > 0 , see [6, Theorem 3.3].

We will make use of the above formula by combining it with the following generalization of the mean-value theorem for integrals.

Lemma B.3. Let D be a connected and compact topological space. Let A be some σ -algebra over D and let µ : A → [0 , ∞ ) be a finite measure on D with µ ( D ) > 0 . Let f : D → R be a continuous function with respect to the standard topology on R and the topology on D . Moreover, let f be measurable with respect to A and the Borel σ -algebra on R . Then there exists ξ ∈ D such that

Proof. Since D is compact and connected and f continuous, it follows that f ( D ) ⊂ R is compact and connected, and hence f ( D ) is a compact interval in R . Therefore, there exist x min ∈ D and x max ∈ D satisfying

for all x ∈ D . Thus, one gets

The claim now follows from the fact that f ( D ) is an interval.

We also get a convenient representation of divided differences for the case of equidistant data points.

Lemma B.4. Let f : R → R , x 0 ∈ R , k ∈ N 0 and h > 0 . We consider the case of equidistant data points, meaning x j := x 0 + jh for all j = 1 , ..., k . In this case, we have the formula

Proof. We prove the result via induction over the number j of considered data points, meaning the following: For all j ∈ { 0 , ..., k } we have

for all ℓ ∈ { 0 , ..., k } satisfying ℓ + j ≤ k . The case j = 0 is trivial. Therefore, we assume the claim to be true for a fixed j ∈ { 0 , ..., k -1 } , and let ℓ ∈ { 0 , ..., k } be arbitrary with ℓ + j +1 ≤ k . We then get

Using an index shift, we deduce

which yields the claim.

The final result for divided differences, which is the result that is actually used in the proof of Theorem 3.1 in Appendix B.3, reads as follows:

Theorem B.5. Let f : R s → R and k ∈ N 0 , r > 0 , such that f ∣ ∣ ( -r,r ) s ∈ C k (( -r, r ) s ; R ) . For p ∈ N s 0 with | p | ≤ k and h > 0 let

Let m := max j p j . Then, for 0 < h < r max { 1 ,m } there exists ξ ∈ h [ -m,m ] s satisfying

Proof. We may assume m> 0 , since m = 0 implies p = 0 and thus f p ,h = f (0) , so that the claim holds for ξ = 0 .

We prove via induction over s ∈ N that the formula

holds for all p ∈ N s 0 with 1 ≤ | p | ≤ k and all 0 < h < r m . The case s = 1 is exactly the Hermite-Genocchi-Formula (B.1), combined with (B.2) applied to the data points

By induction, assume that the claim holds for some s ∈ N . For a fixed point y ∈ ( -r, r ) , let

For p ∈ N s +1 0 with | p | ≤ k and p ′ := ( p 1 , ..., p s ) , we define

Using the induction hypothesis, we get

Γ(

y

)

for all y ∈ ( -r, r ) . Furthermore, we calculate

On the other hand, we get

Interchanging the order of integration and derivative is possible, since we integrate on compact sets and only consider continuously differentiable functions (see, e.g., [11, Lemma 16.2]).

We have thus proven (B.3) using the principle of induction. The claim of the theorem then follows directly using the mean-value theorem for integrals (Lemma B.3) applied to the topological space

equipped with the product topology where each factor is endowed with the standard topology on Σ p ℓ (where the standard topology on Σ 0 is the discrete topology), the measure

defined on the product of the Borel σ -algebras on Σ p ℓ (and the σ -algebra of { 0 } is its power set) and to the function

Moreover, note that all the simplices Σ p ℓ are compact and connected (in fact convex) with

see Definition B.1. Therefore, Lemma B.3 yields the existence of a certain ( ξ (1) , ..., ξ ( s +1) ) ∈ D with

Hence, the claim follows letting

B.2 Proof of Proposition 2.1

̸

Proof of Proposition 2.1. Let M ∈ N 0 . Since ϕ is not polyharmonic we can pick z ∈ U with ∆ M ϕ ( z ) = 0 . By continuity we can choose δ > 0 with B δ ( z ) ⊆ U and ∆ M ϕ ( w ) = 0 for all w ∈ B δ ( z ) . For all m,ℓ ∈ N 0 let

and assume towards a contradiction that

By [3, Corollary 3.35], B δ ( z ) with its usual topology is completely metrizable. By continuity of ∂ m wirt ∂ ℓ wirt ϕ the sets A m,ℓ are closed in B δ ( z ) . Hence, using the Baire category theorem [3, Theorems 3.46 and 3.47], there are m,ℓ ∈ N 0 with m,ℓ ≤ M , z ′ ∈ A m,ℓ and ε > 0 such that

But thanks to ∆ M ϕ = 4 M ∂ M wirt ∂ M wirt ϕ = 4 M ∂ M -ℓ wirt ∂ M -m wirt ∂ ℓ wirt ∂ m wirt ϕ (see property 6 in Appendix A.1), this directly implies ∆ M ϕ ≡ 0 on B ε ( z ′ ) in contradiction to the choice of B δ ( z ) .

B.3 Proof of Theorem 3.1

The following section is dedicated to proving Theorem 3.1. We are going to show that it is possible to approximate complex polynomials in z and z arbitrarily well on Ω n using shallow complex-valued neural networks. To do so, we follow the proof sketch given after the statement of Theorem 3.1, starting with the following lemma.

Lemma B.6. Let ϕ : C → C and δ > 0 , b ∈ C , k ∈ N 0 , such that ϕ ∣ ∣ B δ ( b ) ∈ C k ( B δ ( b ); C ) . For fixed z ∈ Ω n , where we recall that Ω n = [ -1 , 1] n + i [ -1 , 1] n , we consider the map

which is in C k since for w ∈ B δ √ 2 n (0) ⊆ C n we have

For all multi-indices m , ℓ ∈ N n 0 with | m + ℓ | ≤ k we have

for all w ∈ B δ √ 2 n (0) .

Proof. First we prove the statement

by induction over 0 ≤ | ℓ | ≤ k . The case ℓ = 0 is trivial. Assuming that (B.4) holds for fixed ℓ ∈ N n 0 with | ℓ | < k , we want to show

for all w ∈ B δ √ 2 n (0) , where j ∈ { 1 , ..., n } is chosen arbitrarily. To this end, first note

̸

Applying the chain rule for Wirtinger derivatives and using that

since w ↦→ w T z + b is holomorphic in every variable, we see

using the fact that w j ↦→ w T z + b is holomorphic and hence

Thus, we have proven (B.5) and induction yields (B.4).

It remains to show the full claim. We use induction over | m | and note that the case m = 0 has just been shown. We assume that the claim holds true for fixed m ∈ N n 0 with | m + ℓ | < k and choose j ∈ { 1 , ..., n } . Thus, we get

Using the chain rule again, we calculate

By induction, this proves the claim.

As the last preparation for the proof of Theorem 3.1, we need the following lemma.

Lemma B.7. Let ϕ : C → C and δ > 0 , b ∈ C , k ∈ N 0 , such that ϕ ∣ ∣ B δ ( b ) ∈ C k ( B δ ( b ); C ) . Let m,n ∈ N and ε > 0 . For p ∈ N 2 n 0 , h > 0 and z ∈ Ω n we write

where ϕ z is as introduced in Lemma B.6 and φ n is as in (A.1) . Furthermore, let

Then there exists h ∗ > 0 such that

for all p ∈ N 2 n 0 with | p | ≤ k and p ≤ m and h ∈ (0 , h ∗ ) .

Proof. Fix p ∈ N 2 n 0 with | p | ≤ k and p ≤ m . The map

is in C k since

Therefore, the map

is continuous and in particular uniformly continuous on the compact set

Here, we employed √ 2 n +1 < 3 n for every n ∈ N . Hence, there exists h p ∈ (0 , δ 3 n · √ 2 n · m ) , such that

for all ξ ∈ φ n ( h · [ -m,m ] 2 n ) , h ∈ (0 , h p ) and z ∈ Ω n . Now fix such an h ∈ (0 , h p ) and z ∈ Ω n . Applying Theorem B.5 to both components of ( φ -1 1 ◦ Φ p ,h ) ( z ) and φ -1 1 ◦ ϕ z ◦ φ n ∣ ∣ ( -δ 3 n , δ 3 n ) 2 n separately yields the existence of two real vectors ξ 1 , ξ 2 ∈ h · [ -m,m ] 2 n , such that

Rewriting this yields

Using this property, we deduce

and analogously for the imaginary part. Since z ∈ Ω n and h ∈ (0 , h p ) have been chosen arbitrarily we get the claim by choosing

Using the previous two lemmas and the results from Appendix B.1 and Appendix B.2, we can now prove Theorem 3.1.

Proof of Theorem 3.1. Let b ∈ U satisfy

̸

Such a point b exists according to Proposition 2.1. Let p ∈ P ′ and fix m , ℓ ∈ N n 0 with m , ℓ ≤ m . For each z ∈ Ω n , using Lemma B.6, we then have

with suitably chosen complex coefficients b p ′ , p ′′ ∈ C depending only on p ′ , p ′′ , m and ℓ . Here we used that | m | , | ℓ | ≤ mn . Since P ′ ⊆ P n m is bounded and p ∈ P ′ , we can write

with | a m , ℓ | ≤ c for some constant c = c ( P ′ ) > 0 . In combination with (B.6), this easily implies that we can rewrite p as

with coefficients c p ( p ) ∈ C satisfying | c p ( p ) | ≤ c ′ for some constant c ′ = c ′ ( ϕ, b, P ′ , m, n ) . By Lemma B.7, we choose h ∗ > 0 , such that

for all z ∈ Ω n and p ∈ N 2 n 0 with p ≤ 2 m . Furthermore, we can rewrite each function Φ p ,h ∗ as

with suitable coefficients λ α , p ∈ C . Since the cardinality of the set

is (4 m +1) 2 n , this can be converted to

For p as in (B.7), we then define and note

Since the coefficients ρ j have been chosen independently of the polynomial p , we can rewrite θ in the desired form.

C Postponed proofs for the approximation of C k -functions

C.1 Prerequisites from Fourier Analysis

This section is dedicated to reviewing some notations and results from Fourier Analysis. In the end, a quantitative result for the approximation of C k ([ -1 , 1] s ; R ) -functions using linear combinations of multivariate Chebyshev polynomials is derived; see Theorem C.15.

We start by recalling several notations and concepts from Fourier Analysis.

Definition C.1. Let s ∈ N and k ∈ N 0 . We define

and C 2 π ( R s ; C ) := C 0 2 π ( R s ; C ) . For a function f ∈ C k 2 π ( R s ; C ) we write

Moreover, we set ∥ f ∥ L ∞ ([ -π,π ] s ; R ) := ∥ f ∥ C 0 ([ -π,π ] s ; C ) .

Definition C.2. For any s ∈ N and k ∈ Z s , we write

where ⟨· , ·⟩ denotes the usual inner product of two vectors. For any f ∈ C 2 π ( R s ; C ) we define the k -th Fourier coefficient of f to be

Definition C.3. For two functions f, g ∈ C 2 π ( R s ; C ) , we define their convolution as

In the following we define several so-called kernels.

Definition C.4. Let m ∈ N 0 be arbitrary.

The m -th one-dimensional Dirichlet-kernel is defined as

The m -th one-dimensional Fejèr-kernel is defined as

The m -th one-dimensional de-la-Vallée-Poussin-kernel is defined as

Let s ∈ N . We extend the above definitions to dimension s by letting

We need the following property of the multivariate extension of the de-la-Vallée-Poussin-kernel.

Proposition C.5. Let m,s ∈ N . Then one has ∥ V s m ∥ L 1 ([ -π,π ] s ; C ) ≤ 3 s

Proof. From [34, Exercise 1.3 and Lemma 1.4] it follows ∥ F m ∥ L 1 ([ -π,π ]; C ) = 1 and hence using the triangle inequality ∥ V m ∥ L 1 ([ -π,π ]; C ) ≤ 3 . The claim then follows using Tonelli's theorem.

The following definition introduces the term of trigonometric polynomial.

Definition C.6. For any s ∈ N and m ∈ N 0 we call a function of the form

with coefficients a k ∈ C a trigonometric polynomial of coordinatewise degree at most m and denote the space of all those functions with H s m . Here, we consider the sum over all k ∈ Z s with -m ≤ k j ≤ m for all j ∈ { 1 , ..., s } . We then write

for any function f ∈ C 2 π ( R s ; C ) .

The following proposition shows that convolving with the Fejèr kernel produces a trigonometric polynomial of degree at most 2 m -1 , while reproducing trigonometric polynomials of degree m . Furthermore, the norm of the convolution operator is bounded uniformly in m . These properties will be useful for our proof of Theorem C.15.

Proposition C.7. Let s, m ∈ N and k ∈ N 0 . The map

is well-defined and satisfies

Furthermore, there exists a constant c = c ( s ) > 0 (independent of m ), such that

In fact, it holds c ( s ) ≤ exp( C · s ) with an absolute constant C > 0 .

Proof. A direct computation shows that f ∗ e k = ˆ f ( k ) · e k . This implies that v m is well-defined since V s m is a trigonometric polynomial of coordinatewise degree at most 2 m -1 .

The operator is bounded on C k 2 π ( R s ; C ) and C 2 π ( R s ; C ) with norm at most c = 3 s , as follows from Young's inequality [34, Lemma 1.1 (ii)], Proposition C.5, and the fact that one has for all k ∈ N s 0 with | k | ≤ k the identity

It remains to show that v m is the identity on H s m . We first prove that

holds for all k ∈ Z with | k | ≤ m . First note that

We then compute

Similarly, we get

and

Adding up those three identities yields (C.4).

To finally prove (C.2), it clearly suffices to show that

for all k ∈ Z s with -m ≤ k ≤ m . But for such k , using e k ( x ) = ∏ s j =1 e k j ( x j ) , one obtains

for any x ∈ R s , as was to be shown.

The following result follows from a theorem in [29].

Proposition C.8. Let s, k ∈ N . Then there exists a constant c = c ( s, k ) > 0 , such that, for E s m as defined in (C.1) ,

for all m ∈ N and f ∈ C k 2 π ( R s ; R ) . In fact, it holds c ( s, k ) ≤ exp( C · ks ) · k k with an absolute constant C > 0 .

Proof. We apply [29, Theorem 6.6] with n i = m and p i = k , which yields the existence of a constant c 1 = c 1 ( s, k ) > 0 , such that

for all m ∈ N and f ∈ C k 2 π ( R s ; R ) , where ω ℓ ( f, · ) denotes the modulus of continuity of ∂ k f ∂x k ℓ with respect to x ℓ , where we have the trivial bound

so the claim follows by choosing c := 2 s · c 1 .

We refer to Appendix C.2 (see Theorem C.18) for a proof of the claimed bound on the constant c ( s, k ) .

The above proposition bounds the best possible error of approximating f by trigonometric polynomials of coordinatewise degree at most m , but this is in general non-constructive. Our next result shows that a similar bound holds for approximating f by v m ( f ) .

Theorem C.9. Let s ∈ N . Then there exists a constant c = c ( s ) > 0 , such that the operator v m from Proposition C.7 satisfies

s

for any m ∈ N and f ∈ C 2 π ( R ; C ) . In fact, it holds c ( s ) ≤ exp( C · s ) with an absolute constant C > 0

Proof. For any T ∈ H s m one has

Taking the infimum over all T ∈ H m yields the claim.

By combining Proposition C.8 and Theorem C.9, we immediately get the following bound.

Hence, we get

.

Corollary C.10. Let s, k ∈ N 0 . Then there exists a constant c = c ( s, k ) > 0 , such that

for every m ∈ N and f ∈ C k 2 π ( R s ; R ) .

In fact, we have c ( s, k ) ≤ exp( C · ks ) · k k with an absolute constant C > 0 .

Up to now, we have studied the approximation of periodic functions by trigonometric polynomials, but our actual goal is to approximate non-periodic functions by algebraic polynomials. The next lemma establishes a connection between the two settings.

Lemma C.11. Let k ∈ N 0 and s ∈ N . For any function f ∈ C k ([ -1 , 1] s ; C ) , we define the corresponding periodic function via

and note f ∗ ∈ C k 2 π ( R s ; C ) . The map

is a continuous linear operator with respect to the C k -norms on C k ([ -1 , 1] s ; C ) and C k 2 π ( R s ; C ) . The operator norm can be bounded from above by k k .

Proof. The map is well-defined since cos is a smooth function and 2 π -periodic. The linearity of the operator is obvious, so it remains to show its continuity.

The goal is to apply the closed graph theorem [21, Theorem 5.12]. By definition of f ∗ , and since cos : [ -π, π ] → [ -1 , 1] is surjective, we have the equality ∥ f ∥ L ∞ ([ -1 , 1] s ; C ) = ∥ f ∗ ∥ L ∞ ([ -π,π ] s ; C ) . Let then ( f n ) n ∈ N be a sequence of functions f n ∈ C k ([ -1 , 1] s ; C ) and g ∗ ∈ C k 2 π ( R s ; C ) such that f n → f in C k ([ -1 , 1] s ; C ) and f ∗ n → g ∗ in C k 2 π ( R s ; C ) . We then have

It follows f ∗ = g ∗ and the closed graph theorem yields the desired continuity.

We refer to Appendix C.2 (see Theorem C.19 and Remark C.20) for a proof of the claimed bound on the operator norm.

For a function f ∈ C k ([ -1 , 1] s ; C ) we want to express v m ( f ∗ ) in a convenient way, involving a product of cosines. To this end, we make use of the following identity, which is a generalization of the well-known product-to-sum formula for cos .

Lemma C.12. Let s ∈ N . Then it holds for any x ∈ R s that

Proof. This is an inductive generalization of the product-to-sum formula

for x, y ∈ R , which can be found for instance in [1, Eq. 4.3.32]. The case s = 1 holds since cos is an even function. Assume that the claim holds for a fixed s ∈ N and take x ∈ R s +1 . Writing

x ′ = ( x 1 , ..., x s ) , we derive

as was to be shown.

The following proposition states that v m ( f ∗ ) can be expressed as a linear combination of products of cosines. This representation is useful since these cosines can be interpolated by Chebyshev polynomials which in the end leads to the desired approximation result.

Proposition C.13. Let s ∈ N and k ∈ N 0 . For any f ∈ C k ([ -1 , 1] s ; C ) and m ∈ N the de-laVallée-Poussin operator given as f ↦→ v m ( f ∗ ) with v m as in Proposition C.7 and f ↦→ f ∗ as in Lemma C.11 has a representation

for continuous linear functionals

̸

where ∥ k ∥ 0 = # { j ∈ { 1 , ..., s } : k j = 0 } and a m k = ̂ V s m ( k ) . Furthermore, if f ∈ C k ([ -1 , 1] s ; R ) , then V m k ( f ) ∈ R for every k ∈ N s 0 with k ≤ 2 m -1 .

Proof. First of all, it is easy to see that v m ( f ∗ ) is even in each variable, which follows directly from the fact that f ∗ and V s m are both even in each variable. Furthermore, if we write

with appropriately chosen coefficients a m k ∈ R , we easily see

Using Euler's identity and the fact that v m ( f ∗ ) is an even function, we get the representation

for all x ∈ R s . Using ⊙ to denote the componentwise product of two vectors of the same size, i.e., x ⊙ y = ( x i · y i ) i , and using the identity ⟨ k , σ ⊙ x ⟩ = ⟨ σ, k ⊙ x ⟩ , we see since v m ( f ∗ ) is even in

every variable that

with

̸

In the last step we again used that cos is an even function and that

for all σ ∈ {-1 , 1 } s , which also follows easily since f ∗ and V s m are even in every component. Letting

we have the desired form. The fact that V m k is a continuous linear functional on C k 2 π ([ -1 , 1] s ; C ) follows directly since f ↦→ ̂ f ∗ ( k ) is a continuous linear functional for every k . If f is real-valued, so is ̂ f ∗ ( k ) for every k ∈ N s 0 with k ≤ 2 m -1 , since f ∗ is real-valued and even in every component.

Our main approximation result involves linear combinations of Chebyshev polynomials where the coefficients in this linear combination are given as V m k ( f ) . It is therefore important to be able to bound the sum of the absolute values |V m k ( f ) | .

Lemma C.14. Let s ∈ N . There exists a constant c = c ( s ) > 0 , such that the inequality

holds for all m ∈ N and f ∈ C ([ -1 , 1] s ; C ) , where V m k is as in Proposition C.13. In fact, we have c ( s ) ≤ exp( C · s ) with an absolute constant C > 0 .

Proof. Let f ∈ C ([ -1 , 1] s ; C ) and m ∈ N . For any multi-index ℓ ∈ N s 0 , it follows from Proposition C.13 that

with

Now, a calculation using Fubini's theorem and using

̸

for any number k ∈ N 0 shows

Therefore, we have the bound |V m ℓ ( f ) | ≤ 2 s · ∣ ∣ ∣ ̂ v m ( f ∗ )( ℓ ) ∣ ∣ ∣ for ℓ ∈ N s 0 with | ℓ | ≤ 2 m -1 . Using the Cauchy-Schwarz and the Parseval inequality, we therefore see

Using Proposition C.7, we get a constant c 2 ( s ) ≤ exp( C 0 · s ) such that

as claimed.

For any natural number ℓ ∈ N 0 , we denote by T ℓ the ℓ -th Chebyshev polynomial , satisfying

One can show that T ℓ is in fact a polynomial of degree ℓ . For a multi-index k ∈ N s 0 , we define

We then get the following approximation result about approximating (non-periodic) C k -functions by linear combinations of Chebyshev polynomials.

Theorem C.15. Let k, s, m ∈ N . Then there exists a constant c = c ( s, k ) > 0 with the following property: For any f ∈ C k ([ -1 , 1] s ; R ) the polynomial P defined as

with V m k as in Proposition C.13, satisfies

are continuous and linear functionals with respect to the L ∞ -norm. Furthermore, there exists a constant ˜ c = ˜ c ( s ) > 0 , such that the inequality

holds for all f ∈ C ([ -1 , 1] s ; R ) . Moreover, we have c ( s, k ) ≤ exp( C · ks ) · k 2 k and ˜ c ( s ) ≤ exp( C · s ) C > 0 .

with an absolute constant

Here, the maps

Proof. We choose the constant c 0 = c 0 ( s, k ) according to Corollary C.10. Let f ∈ C k ([ -1 , 1] s ; R ) be arbitrary. Then we define the corresponding function f ∗ ∈ C k 2 π ( R s ; R ) as above. Let P be defined as in the statement of the theorem. Then it follows from the definition of the Chebyshev polynomials T k , the definition of P , and the formula for v m ( f ∗ ) from Proposition C.13 that

is satisfied, where P ∗ is the corresponding function to P defined similarly to f ∗ . Overall, we get the bound

The first claim then follows using the continuity of the map f ↦→ f ∗ as proven in Lemma C.11. The second part of the theorem has already been proven in Lemma C.14.

C.2 Details on bounding the constant c in Theorem 3.2

In this appendix we provide details on the bound of the constant c that appears in the formulation of Theorem 3.2. Specifically, we perform a careful investigation of several results from [29] to get an upper bound for the constant appearing in Proposition C.8. Moreover, we analyze the operator norm of the operator

appearing in Lemma C.11 and show that it is bounded from above by k k .

We start with the analysis of some bounds in [29, Chapter 4.3]. Here, a generalization of Jackson's kernel is defined for any m,r ∈ N as

where λ m,r is chosen such that

The first two important bounds are provided in the following proposition.

Proposition C.16. Let m,r ∈ N . Then it holds

for any k ≤ 2 r -2 , with an absolute constant C > 0 .

Proof. Since L m,r ≥ 0 and since sin( t/ 2) ≤ t/ 2 for t ∈ [0 , π ] , we get

Here, we employed the inequality sin( u ) ≥ 2 π u for u ∈ [0 , π/ 2] in the penultimate step. 1 This shows the first part of the claim.

For the second part, we again use the estimate sin( u ) ≥ 2 π u for u ∈ [0 , π/ 2] to compute

with an absolute constant C 1 > 0 . Here, we employed the first part of this proposition. It remains to bound the integral. This is done via

with an absolute constant C 2 > 0 . This proves the claim.

The proof in [29] proceeds by defining

Proposition C.16 shows for k ≤ 2 r -2 that

Since m ′ ≥ m r we infer

with an absolute constant C > 0 .

We can now quantify the constant appearing in [29, Theorem 4.3].

Theorem C.17 (cf. [29, Theorem 4.3]) . Let k, m ∈ N and f ∈ C k 2 π ( R ; R ) . Let

Then it holds

Here, we recall that E 1 m ( f ) denotes the best possible approximation error when approximating f using trigonometric polynomials of degree m ; see Equation (C.1) .

Proof. We follow the proof of [29, Theorem 4.3]. Take r = k +1 and define

Then it is shown in the proof of [29, Theorem 4.3] that I m is a trigonometric polynomial of degree at most m and that

Here, ω k +1 ( f, 1 /m ) denotes the modulus of smoothness of f as defined on [29, p. 47]. The integral can be bounded via

with absolute constants C, C 1 > 0 . Since ω k +1 ( f, 1 /m ) ≤ m -k · ω ( f ( k ) , 1 /m ) follows from [29, Equation 3.6(5)], the claim is shown.

Therefore, we can bound the constant appearing in [29, Theorem 4.3] by exp( C · k ) · k k . It remains to deal with the approximation of multivariate periodic functions by multivariate trigonometric polynomials which is contained in [29, Theorem 6.6].

Theorem C.18 (cf. [29, Theorem 6.6]) . Let s, k ∈ N and f ∈ C k 2 π ( R s ; R ) . Let ω j denote the modulus of continuity of ∂ k f ∂x k j for j = 1 , ..., s . Then, with E s m as introduced in Equation (C.1) , it holds

with an absolute constant C > 0 .

Proof. We follow the proof of [29, Theorem 6.6] with p j = k and n j = m for every index j = 1 , ..., s . For j = 1 , ..., s +1 define the set T j consisting of all functions g ∈ C 2 π ( R s ; R ) that are a trigonometric polynomial in x ℓ of degree at most m for ℓ < j ; in x ℓ for ℓ ≥ j they should have continuous partial derivatives ∂ p g ∂x p ℓ for 0 ≤ p ≤ k ; the modulus of continuity of ∂ k g ∂x k ℓ should not exceed 2 K j ω j , where

Then it is shown that if j ∈ { 1 , ..., s } and f j ∈ T j there exists a function f j +1 ∈ T j +1 for which ∥ f j -f j +1 ∥ L ∞ ( R s ; R ) ≤ exp( C 1 · k ) · k k · m -k · 2 2 ks · ω j (1 /m ) ≤ exp( C 2 · ks ) · k k · m -k · ω j (1 /m ) for absolute constants C 1 , C 2 > 0 . This is an application of Theorem C.17. Hence, defining f 1 := f , we see

Therefore, we have shown that the constant appearing in [29, Theorem 6.6] can be bounded from above by exp( C · ks ) · k k .

In the rest of this section, we discuss the operator norm of the operator defined in Lemma C.11. In Lemma C.11 the closed graph theorem is used to show that the operator is bounded. However, the closed graph theorem does not provide any bound on the norm of the operator. Therefore, in order to quantify the operator norm, we need to apply a different technique, which is Faa di Bruno's formula . This formula is a generalization of the chain rule to higher order derivatives.

Theorem C.19. Let s, k ∈ N . We define the operator

T : C k ([ -1 , 1] s ; C ) → C k 2 π ([ -π, π ] s ; C ) , ( Tf )( x 1 , ..., x s ) := f (cos( x 1 ) , ..., cos( x s )) . Let α ∈ N s 0 with | α | ≤ k . Then, for any f ∈ C k ([ -1 , 1] s ; C ) , we have

Proof. The proof is by induction over s . The case s = 1 is an application of Faa di Bruno's formula: We can write Tf = f ◦ g with g ( x ) = cos( x ) . We then take ℓ ∈ N 0 with ℓ ≤ k and some x ∈ [ -π, π ] . The set partition version of Faa di Bruno's formula (see for instance [24, p. 219]) then yields

Here, Π ℓ denotes the set of all partitions of the set { 1 , ..., ℓ } . Since all derivatives of g are bounded by 1 in absolute value and | π | ≤ ℓ for every partition π ∈ Π ℓ we get

The number | Π ℓ | is the number of possible partitions of the set { 1 , ..., ℓ } and is the so-called ℓ -th Bell number . It can be bounded from above by ℓ ℓ (see [13, Theorem 2.1]). This proves the case s = 1 .

We now assume that the claim holds for an arbitrary but fixed s ∈ N . Take α ∈ N s +1 0 with | α | ≤ k . We decompose α = ( α ′ , α s +1 ) with α ′ ∈ N s 0 . For a fixed variable y s +1 ∈ [ -1 , 1] , we define

We denote g ( x 1 , ..., x s +1 ) := (cos( x 1 ) , ..., cos( x s +1 )) , g s ( x 1 , ..., x s ) := (cos( x 1 ) , ..., cos( x s )) and θ ( x s +1 ) := cos( x s +1 ) . For every ( x 1 , ..., x s +1 ) ∈ [ -π, π ] s +1 it then holds

We now differentiate f ◦ g with respect to the multiindex α and get

where we define h x 1 ,...,x s ( y s +1 ) := ∂ α ′ ( f y s +1 ◦ g s ) ( x 1 , ..., x s ) for ( x 1 , ..., x s ) ∈ [ -π, π ] s and y s +1 ∈ [ -1 , 1] . Using the case s = 1 , we get

It remains to bound ∥ h x 1 ,...,x s ∥ C α s +1 ([ -1 , 1]; C ) . To this end, we fix ℓ ∈ N 0 with ℓ ≤ α s +1 . We further denote

For arbitrary ( x 1 , ..., x s ) ∈ [ -π, π ] s and y s +1 ∈ [ -1 , 1] we then see

where

Hence, we see by induction that

as was to be shown.

Remark C.20 . For a multiindex α ∈ N s 0 with | α | ≤ k we see

Hence, the norm of the operator introduced in Lemma C.11 can be bounded from above by k k .

C.3 Proof of Theorem 3.2

For any natural number ℓ ∈ N 0 , we denote by T ℓ the ℓ -th Chebyshev polynomial, satisfying

For a multi-index k ∈ N s 0 we define

The proof of Theorem 3.2 relies on the fact that C k -functions can by approximated at a certain rate using linear combinations of the T k (see Theorem C.15). We also refer to Figure 4 for an illustration of the overall proof strategy of Theorem 3.2.

Proof of Theorem 3.2. Choose M ∈ N as the largest integer for which (16 M -7) 2 n ≤ m , where we assume without loss of generality that 9 2 n ≤ m , which can be done by choosing σ j = 0 for all j ∈ { 1 , ..., m } for m< 9 2 n , at the cost of possibly enlarging c . First we note that by the choice of M the inequality

holds true. Since 16 M +9 ≤ 25 M , we get m ≤ 25 2 n · M 2 n or equivalently

According to Theorem C.15 we choose a constant c 1 = c 1 ( n, k ) with the property that for any function f ∈ C k ( [ -1 , 1] 2 n ; R ) there exists a polynomial

of coordinatewise degree at most 2 M -1 satisfying

Furthermore, according to Theorem C.15, we choose a constant c 2 = c 2 ( n ) , such that the inequality

holds for all f ∈ C k ( [ -1 , 1] 2 n ; R ) . The final constant is then defined to be

Fix k ≤ 2 M -1 . Since T k is a polynomial of componentwise degree less or equal to 2 M -1 with φ n as in (A.1), we have a representation

with suitably chosen coefficients a k ℓ 1 , ℓ 2 ∈ C . By using the identities Re( z t ) = 1 2 ( z t + z t ) and also Im( z t ) = 1 2 i ( z t -z t ) we can rewrite T k ◦ φ -1 n into a complex polynomial in z and z , i.e.,

with complex coefficients b k ℓ 1 , ℓ 2 ∈ C . Using Theorem 3.1, we choose ρ 1 , ..., ρ m ∈ C n and b ∈ C , such that for any polynomial P ∈ { T k ◦ φ -1 n : k ≤ 2 M -1 } ⊆ P n 4 M -2 there are coefficients σ 1 ( P ) , ..., σ m ( P ) ∈ C , such that

where

Note that here we implicitly use the bound (4 · (4 M -2) + 1) 2 n ≤ m . We are now going to show that the chosen constant and the chosen vectors ρ t have the desired property.

Let f ∈ C k (Ω n ; C ) . By splitting f into real and imaginary part, we write f = f 1 + i · f 2 with f 1 , f 2 ∈ C k (Ω n ; R ) . For the following, fix j ∈ { 1 , 2 } and note that f j ◦ φ n ∈ C k ( [ -1 , 1] 2 n ; R ) . By choice of c 1 , there exists a polynomial P with the property

or equivalently

where P ◦ φ -1 n can be written in the form

We choose the function g T k ◦ φ -1 n according to (C.8). Thus, writing

we obtain

Combining (C.9) and (C.10), we see

In the end, define

Since the vectors ρ t have been chosen fixed, it is clear that, after rearranging, g has the desired form, i.e., g = σ T Φ where Φ( z ) = ( ϕ ( ρ t z + b )) m t =1 . Furthermore, one obtains the bound

Using (C.7), we see as desired.

The linearity and continuity of the maps f ↦→ σ j ( f ) (with respect to the ∥ · ∥ L ∞ -norm) follow easily from the fact that the map f ↦→V M k ( f ) is a continuous linear functional for every multiindex 0 ≤ k ≤ 2 M -1 .

Approximation of

partial derivatives

Approximation

of polynomials

Approximation

of

C

k

-functions

Divided Differences

Wirtinger derivatives

of

w

ϕ

(

w

T

z

+

b

)

Chebyshev

polynomials

D Approximation of holomorphically extendable functions

In this appendix we provide the proofs for the statements contained in Remark 3.3. The proofs mainly rely on results about sparse polynomial approximation [2].

Definition D.1 (cf. [2, Assumption 2.3]) . Let s ∈ N and ν ∈ (1 , ∞ ) s . For every j ∈ { 1 , ..., s } let

We then define the (filled-in) Bernstein polyellipse of parameter ν as

and observe that [ -1 , 1] s ⊆ E ν when we interpret [ -1 , 1] s as a subset of C s . We further define

Here, [ -1 , 1] s is again interpreted as a subset of C s . Moreover, note that such an extension ˜ f is, if existent, unique, as follows from the identity theorem for holomorphic functions. Hence, the expression

is well-defined.

Definition D.2 (cf. [2, pp. 25, 28 ff.]) . Let s ∈ N . We define a probability measure on [ -1 , 1] s via

We define the normalized Chebyshev polynomials for k ∈ N s 0 as

̸

where ∥ k ∥ 0 := # { 1 ≤ j ≤ s : k j = 0 } . Note that this definition differs slightly from the notion used in Appendices C.1 and C.3.

The following lemma is crucial for deriving the approximation rate in Theorem D.5. The proofs can be found in [2].

Lemma D.3 (cf. [2, Remark 2.15 and Theorem 3.2]) . Let s ∈ N , k ∈ N s 0 , ν ∈ (1 , ∞ ) s and f ∈ V s ( ν ) . Then it holds

̸

It is a well-known fact that the Chebyshev polynomials ˜ T k form an orthonormal basis of L 2 µ s ([ -1 , 1] s ; C ) . The following proposition states that functions from V s ( ν ) can even be approximated uniformly by linear combinations of the ˜ T k at a certain rate. The proof follows essentially by applying Lemma D.3.

Proposition D.4. Let s, m ∈ N and ν ∈ (1 , ∞ ) s . Let ν := min j =1 ,...,s ν j . Then there exists a constant c = c ( s, ν ) > 0 with the following property: For every f ∈ V s ( ν ) , defining

Proof. Let f ∈ V s ( ν ) . Since the ˜ T k form an orthonormal basis of L 2 µ s ([ -1 , 1] s ; C ) and since µ s is a probability measure, so that f ∈ C ([ -1 , 1] s ; C ) ⊆ L 2 µ 2 ([ -1 , 1] s ; C ) , it follows that

with unconditional convergence in L 2 µ s . For x ∈ [ -1 , 1] s , note that

Here, we employed Lemma D.3 at the last inequality. From ν > 1 it follows that

converges pointwise (even uniformly) and, since all the involved functions are continuous, this pointwise limit then has to coincide with the L 2 µ s -limit f . Hence, it holds

where we again used Lemma D.3. To complete the proof we compute

̸

and define c ( s, ν ) := 2 s · s · ∏ s ℓ =1 ν j ν j -1 · ν -1 .

To formulate the result for the approximation of holomorphically extendable functions using CVNNs we need to transfer the definition of V s ( ν ) to the complex setting. For ν ∈ (1 , ∞ ) 2 n and with φ n as in Equation (A.1), we hence write

it holds

For f ∈ W n ( ν ) we define where

Thus, W n ( ν ) consists of all complex-valued functions defined on Ω n that can be holomorphically extended onto some polyellipse in C 2 n , where Ω n ⊆ C n is interpreted as a subset of R 2 n and then as a subset of C 2 n . The final approximation result then reads as follows.

Theorem D.5. Let n ∈ N and ν ∈ (1 , ∞ ) 2 n . Set

Then there exists a constant c = c ( n, ν ) > 0 with the following property: For every function ϕ : C → C that is smooth and non-polyharmonic on some open set ∅ = U ⊂ C and for every m ∈ N there exists a first layer Φ ∈ F ϕ n,m with the property that for every f ∈ W n ( ν ) there exist coefficients σ = σ ( f ) ∈ C m such that

̸

Moreover, the map f ↦→ σ ( f ) is a continuous linear functional with respect to the L ∞ -norm.

Proof. Choose M ∈ N as the largest integer satisfying (8 M +1) 2 n ≤ m , where we assume without loss of generality that 9 2 n ≤ m . This can be done by choosing σ = 0 for m < 9 2 n at the cost of possibly enlarging c . Note that the maximality of M implies (8 M + 9) 2 n > m . By using 8 M +9 ≤ 17 M , this gives us

Choose the constant c 1 = c 1 (2 n, ν ) according to Proposition D.4.

Fix k ∈ N 2 n 0 with k ≤ M . Since ˜ T k is a polynomial of componentwise degree at most M , we have a representation

with suitably chosen coefficients a k ℓ 1 , ℓ 2 ∈ C . By using the identities Re( z t ) = 1 2 ( z t + z t ) and also Im( z t ) = 1 2 i ( z t -z t ) , we can rewrite ˜ T k ◦ φ -1 n into a complex polynomial in z and z , i.e.,

with complex coefficients b k ℓ 1 , ℓ 2 ∈ C . Using Theorem 3.1, we choose ρ 1 , ..., ρ m ∈ C n and b ∈ C , such that for any polynomial P ∈ { ˜ T k ◦ φ -1 n : k ≤ M } ⊆ P n 2 M there exist coefficients σ 1 ( P ) , ..., σ m ( P ) ∈ C , such that

Note that here we implicitly use the bound (4 · (2 M ) + 1) 2 n ≤ m . We are now going to show that the first layer Φ ∈ F ϕ n,m defined using the ρ t and b (i.e., Φ( z ) = ( ϕ ( ρ T t z + b )) t m =1 ) has the desired property.

To this end, take an arbitrary function f ∈ W n ( ν ) . Proposition D.4 tells us that

where

We then define the approximating network

From the definition of the functions g ˜ T k ◦ φ -1 n it follows directly that g is a shallow CVNN with first layer Φ ∈ F ϕ n,m . Furthermore, it holds

Combining this estimate with Equation (D.3) and applying the triangle inequality gives us

Hence, the claim follows taking c := c 1 +2 n and using (D.1).

The continuity of the linear map f ↦→ σ ( f ) with respect to the L ∞ -norm follows directly from the fact that f ↦→⟨ f ◦ φ n , ˜ T k ⟩ µ 2 n is trivially continuous with respect to the L 2 µ 2 n -norm and hence also continuous with respect to the L ∞ -norm since µ 2 n is a probability measure.

E Postponed proofs for the optimality results in the case of continuous weight selection

In this section we provide the proofs for the optimality results derived if a continuous weight selection is assumed. Specifically, we prove Theorems 4.1 and 5.1. The proofs of both results rely on a very general result about the approximation in normed spaces by subsets that are parametrizable with m parameters [19]. We decided to include a detailed proof for this approximation result in this paper since the nature of the continuity assumption in [19] is not completely clear.

̸

Proposition E.1 ([19, Theorem 3.1]) . Let ( X, ∥ · ∥ X ) be a normed space, ∅ = K ⊆ X a subset and V ⊆ X a linear, not necessarily closed subspace of X containing K . Let m ∈ N , let a : K → R m be a map which is continuous with respect to some norm ∥ · ∥ V on V and M : R m → X some arbitrary map. Let

where the first supremum is taken over all ( m +1) -dimensional linear subspaces X m +1 of X and

Further, we set b m ( K ) X := 0 if the supremum in (E.1) is not well-defined as a quantity in [0 , ∞ ] . Then it holds

Proof. The claim is trivial if b m ( K ) X = 0 . Thus, assume b m ( K ) X > 0 . Let 0 < ϱ ≤ b m ( K ) X be any number such that there exists an ( m +1) -dimensional subspace X m +1 of X with U ϱ ( X m +1 ) ⊆ K . It follows U ϱ ( X m +1 ) ⊆ V , hence X m +1 ⊆ V , so ∥ · ∥ V defines a norm on X m +1 . Thus, the

restriction of a to ∂U ϱ ( X m +1 ) is a continuous mapping to R m with respect to ∥· ∥ V . Since all norms are equivalent on the finite-dimensional space X m +1 , the Borsuk-Ulam-Theorem [18, Corollary 4.2] yields the existence of a point x 0 ∈ ∂U ϱ ( X m +1 ) with a ( x 0 ) = a ( -x 0 ) . We then see

and hence at least one of the two summands on the right has to be larger than or equal to ϱ .

E.1 Proof of Theorem 4.1

Using Proposition E.1, we can deduce our lower bound in the context of C k -spaces. The proof is in fact almost identical to what is done in [19, Theorem 4.2]. However, we decided to include a detailed proof in this paper, since [19] considers Sobolev functions and not C k -functions.

Theorem E.2. Let s, k ∈ N . Then there exists a constant c = c ( s, k ) > 0 with the following property: For any m ∈ N and any map a : C k ([ -1 , 1] s ; R ) → R m that is continuous with respect to some norm on C k ([ -1 , 1] s ; R ) and any (possibly discontinuous) map M : R m → C ([ -1 , 1] s ; R ) , we have

Proof. The idea is to apply Proposition E.1 to X := C ([ -1 , 1] s ; R ) , V := C k ([ -1 , 1] s ; R ) and the set K := { f ∈ C k ([ -1 , 1] s ; R ) : ∥ f ∥ C k ([ -1 , 1] s ; R ) ≤ 1 } .

Assume in the beginning that m = n s with an integer n > 1 . Pick ϕ ∈ C ∞ ( R s ) with ϕ ≡ 1 on [ -3 / 4 , 3 / 4] s and ϕ ≡ 0 outside of [ -1 , 1] s . Fix c 0 = c 0 ( s, k ) > 0 with

Let Q 1 , ..., Q m be the partition (disjoint up to null-sets) of [ -1 , 1] s into closed cubes of sidelength 2 /n . For every j ∈ { 1 , ..., m } we write Q j = × s ℓ =1 [ a ( j ) ℓ -1 /n, a ( j ) ℓ +1 /n ] with an appropriately chosen vector a = ( a ( j ) 1 , ..., a ( j ) s ) ∈ [ -1 , 1] s and let

By choice of ϕ , the maps ϕ j are supported on a proper subset of Q j for every j ∈ { 1 , ..., m } and an inductive argument shows

and hence in particular

Let X m := span { ϕ 1 , ..., ϕ m } and S ∈ U 1 ( X m ) = { f ∈ X m : ∥ f ∥ L ∞ ([ -1 , 1] s ; R ) ≤ 1 } . Then we can write S in the form S = ∑ m j =1 c j ϕ j with real numbers c 1 , ..., c m ∈ R . Suppose there exists j ∗ ∈ { 1 , ..., m } with | c j ∗ | > 1 . Then we have

since the functions ϕ j have disjoint support and ϕ j ( a ( j ) ) = 1 . This is a contradiction to S ∈ U 1 ( X m ) and we can thus infer that max j | c j | ≤ 1 . Furthermore, we see again because the functions ϕ j have disjoint support that

for every k ∈ N s 0 with | k | ≤ k and hence

Thus, letting ϱ := c -1 0 · m -k/s yields U ϱ ( X m ) ⊆ K , so we see by Proposition E.1 that

with c 1 = c -1 0 for every map a : X → R m -1 which is continuous with respect to some norm on V and any map M m -1 : R m -1 → X . Using the inequality m ≤ 2( m -1) (note m> 1 ) we get

with c 2 = c 1 · 2 -k/s . Hence, the claim has been shown for all numbers m of the form n s -1 with an integer n > 1 .

In the end, let m ∈ N be arbitrary and pick n ∈ N with n s ≤ m < ( n + 1) s . For given maps a : V → R m and M : R m → X with a continuous with respect to some norm on V , let

where x ∈ R m , y ∈ R ( n +1) s -1 -m . Then we get

with c 3 = c 2 · 2 -k . Here we used the bound ( n +1) s -1 ≤ (2 n ) s . This proves the full claim.

Using this theorem, we can now prove Theorem 4.1.

Proof of Theorem 4.1. Let a : C k (Ω n ; C ) → C m be any map that is continuous with respect to some norm ∥ · ∥ V on C k (Ω n ; C ) , and let M : C m → C (Ω n ; C ) be arbitrary. With φ n , φ m defined as in Equation (A.1), let

Clearly, ˜ a is continuous on C k ([ -1 , 1] 2 n ; R ) with respect to the norm ∥ · ∥ ˜ V on C k ([ -1 , 1] 2 n ; R ) defined as

Let

Then it holds

with a constant ˜ c = ˜ c ( n, k ) provided by Theorem E.2. Hence, the claim follows by choosing c = c ( n, k ) := 2 -k/ (2 n ) · ˜ c .

As a corollary, we formulate a special case of Theorem 4.1 for the case of shallow complex-valued neural networks.

Corollary E.3. Let n, k ∈ N . Then there exists a constant c = c ( n, k ) > 0 with the following property: For any m ∈ N , ϕ ∈ C ( C ; C ) and any map

which is continuous with respect to some norm on C k (Ω n ; C ) , there exists f ∈ C k (Ω n ; C ) satisfying ∥ f ∥ C k (Ω n ; C ) ≤ 1 and

where Ψ( f ) ∈ C (Ω n ; C ) is given by

Proof. Using Theorem 4.1, we deduce that there exists f ∈ C k (Ω n ; C ) satisfying ∥ f ∥ C k (Ω n ; C ) ≤ 1 and

for a constant c ′ = c ′ ( n, k ) > 0 . Hence, the claim follows by letting c := c ′ · ( n +2) -k/ (2 n ) .

E.2 Proof of Theorem 5.1

We can use Proposition E.1 not only to show that the rate of convergence established in this paper is optimal (which is done in Appendix E.1) but also to show that the problem of approximating C k -functions using a set of functions that can be parametrized with finitely many parameters is intractable in the sense that it suffers from the curse of dimensionality, provided that the map which assigns to each C k -function the parameters of the approximating function is continuous. This is the subject of this section.

In [35] a certain space of polynomials was used to show the intractability in the case of linear approximation methods. We are also going to use this class of polynomials, but combine it with Proposition E.1 to infer intractability in the case of continuous approximation methods. We start with a lemma discussing an important property of this space of polynomials. This property is stated as part of a proof in [35], but no complete proof is provided.

Lemma E.4. Let s ∈ N and consider a function f ∈ C ∞ ([ -1 , 1] s ; R ) which is given via

with coefficients a k ∈ R for every k ∈ { 0 , 1 } s . Then it holds

for every k ∈ N .

Proof. The proof is by induction over s . We start with the case s = 1 and note that we can write f ( x ) = ax + b with a, b ∈ R in that case. Switching to -f if necessary, we can assume a ≥ 0 . Clearly, ∥ f ∥ L ∞ ([ -1 , 1]; R ) ≤ | a | + | b | . Conversely, if b ≥ 0 then | f (1) | = | a + b | = | a | + | b | . If otherwise b < 0 then | f ( -1) | = | b -a | = | a -b | = | a | + | b | . Thus, ∥ f ∥ L ∞ ([ -1 , 1]; R ) = | a | + | b | . For the derivatives, we have ∥ f ′ ∥ L ∞ ([ -1 , 1]; R ) = | a | and ∥ f ( k ) ∥ L ∞ ([ -1 , 1]; R ) = 0 for k ≥ 2 . This proves the claim in the case s = 1 .

We now assume that the claim holds for some arbitrary but fixed s ∈ N . We further let α ∈ N s +1 0 and fix a point ( x 1 , ..., x s +1 ) ∈ [ -1 , 1] s +1 . We decompose α = ( α ′ , α s +1 ) with α ′ ∈ N s 0 . Let

and note

Note that f is affine-linear with respect to each variable (with all other variables hold fixed). Hence, ˜ f is an affine function and we can thus apply the case s = 1 to ˜ f and get

Putting this together, we infer

We now fix an arbitrary point y s +1 ∈ [ -1 , 1] and consider

Then it holds

Applying the induction hypothesis to ̂ f (which is easily seen to be of the form (E.3)) we get

Combining (E.4) and (E.5) yields

Since α ∈ N s +1 0 was arbitrary, we get the claim by noting that

holds trivially for every k ∈ N .

Using the above lemma, we can now deduce that the approximation of smooth functions using continuous approximation methods is intractable in terms of the input dimension.

Proof of Theorem 5.1. We apply Proposition E.1 to X := C ([ -1 , 1] s ; R ) , V := C ∞ , ∗ ,s and to the set K := { f ∈ C ∞ , ∗ ,s : ∥ f ∥ C ∞ ([ -1 , 1] s ; R ) ≤ 1 } and m := 2 s -1 . The space

consisting of all functions considered in the previous Lemma E.4 is an ( m + 1) -dimensional subspace of C ([ -1 , 1] s ; R ) . For every f ∈ X m +1 with ∥ f ∥ L ∞ ([ -1 , 1] s ; R ) ≤ 1 , Lemma E.4 tells us ∥ f ∥ C ∞ ([ -1 , 1] s ; R ) ≤ 1 . Hence, U 1 ( X m +1 ) ⊆ K and Proposition E.1 then yields the claim.

Remark E.5 . The statement of Theorem 5.1 also holds if the functions satisfy a : C ∞ , ∗ ,s → R m and M : R m → C ([ -1 , 1] s ; R ) with m ≤ 2 s -1 . This can be seen by defining

and with a ∈ R m and b ∈ R 2 s -1 -m .

The following Corollary E.6 transfers Theorem 5.1 to the complex-valued setting.

Corollary E.6. Let n ∈ N . For any function f ∈ C ∞ (Ω n ; C ) we write

and let C ∞ , ∗ ,n C denote the space consisting of all functions for which this expression is finite. Let a : C ∞ , ∗ ,n C → C 2 2 n -1 -1 be continuous with respect to some norm on C ∞ , ∗ ,n C and moreover, let M : C 2 2 n -1 -1 → C (Ω n ; C ) be an arbitrary map. Then it holds

Proof. The transfer to the complex-valued setting works in the same manner as the proof of Theorem 4.1 (see Appendix E.1). We write m := 2 2 n -1 -1 and note 2 m = 2 2 n -2 ≤ 2 2 n -1 . We define ˜ a : C ∞ , ∗ , 2 n → R 2 m and ˜ M : R 2 m → C ([ -1 , 1] 2 n ; R ) in the same way as in the proof of Theorem 4.1. Using again the same technique as in the proof of Theorem 4.1, we get

applying Theorem 5.1 in the last inequality, using 2 m ≤ 2 2 n -1 .

We conclude this appendix by adding a note on the constant appearing in our main approximation bound.

Corollary E.7. Let n ∈ N with n ≥ 2 and α > 0 and let ϕ ∈ C ( C ; C ) . Let ˜ c = ˜ c ( n, α ) > 0 be such that for every m ∈ N there exists a mapping

that is continuous with respect to any norm on C ∞ , ∗ ,n C and such that

for every f ∈ C ∞ , ∗ ,n C . Here, Ψ( f ) ∈ C (Ω n ; C ) is given by

Then it necessarily holds ˜ c ≤ 16 · 2 -n .

Proof. We first assume n ≥ 4 . We take m = ⌊ 2 2 n -1 -1 n +2 ⌋ and note that then m ( n +2) ≤ 2 2 n -1 -1 . Therefore, Corollary E.6 applies and we infer that for each ε ∈ (0 , 1) , there exists f = f ε ∈ C ∞ , ∗ ,n C with ∥ f ∥ C ∞ (Ω n ; C ) ≤ 1 and such that

This then necessarily implies ˜ c · m ≤ 1 or equivalently ˜ c ≤ 1 /m . It therefore suffices to derive a lower bound for m . Firstly, we note

where we applied Bernoulli's inequality. Because of n ≥ 4 ≥ 3 , this yields

Hence, we get

Here, we used n ≥ 4 in the last inequality. An explicit computation shows that the same bounds also holds in the cases n = 2 and n = 3 . This proves the claim.

F Postponed proofs for the optimality results in the case of unrestricted weight selection

F.1 Approximation using Ridge Functions

In this section we prove for s ∈ N ≥ 2 that every function in C k ([ -1 , 1] s ; R ) can be uniformly approximated with an error of the order m -k/ ( s -1) using a linear combination of m so-called ridge functions . In fact, we only consider ridge polynomials , meaning functions of the form

for a fixed vector a ∈ R s and a polynomial p : R → R . Note that this result has already been obtained in a slightly different form in [30, Theorem 1]; namely, it is shown there that the rate of approximation m -k/ ( s -1) can be achieved by functions of the form ∑ m j =1 f j ( a T j x ) with a j ∈ R s and f j ∈ L 1 loc ( R s ) . We will need the fact that the f j can actually be chosen as polynomials and that the vectors a 1 , ..., a m can be chosen independently from the particular function f . This is shown in the proof of [30], but not stated explicitly. For this reason, and in order to clarify the proof itself and to make the paper more self-contained, we decided to present the proof in this appendix.

Lemma F.1. Let m,s ∈ N . Then we denote by

the set of real polynomials of degree at most m . The subset of homogeneous polynomials of degree m is defined as

Then there exists a constant c = c ( s ) > 0 satisfying

Proof. It is immediate that the set

This quantity clearly equals the number of possibilities for drawing m times from a set with s elements with replacement. Hence, we see

see for instance [12, Identity 143]. A further estimation shows

Hence, the claim follows with c ( s ) = 2 s -1 .

A combination of results from [36] together with the fact that it is possible to approximate C k -functions using polynomials of degree at most m with an error of the order m -k , as shown in Theorem C.15, yields the desired result.

Theorem F.2. Let s, k ∈ N with s ≥ 2 and r > 0 . Then there exists a constant c = c ( s, k ) > 0 with the following property: For every m ∈ N there exist a 1 , ..., a m ∈ R s \ { 0 } with ∥ a j ∥ 2 = r , such that for every function f ∈ C k ([ -1 , 1] s ; R ) there exist polynomials p 1 , ..., p m ∈ P 1 m satisfying

forms a basis of H s m , hence

Proof. We first pick the constant c 1 = c 1 ( s ) according to Lemma F.1. Then we define the constant c 2 = c 2 ( s ) := (2 s ) s -1 · c 1 ( s ) and let M ∈ N be the largest integer satisfying

Here, we assume without loss of generality that m ≥ c 2 , which can be justified by choosing p j = 0 for every j ∈ { 1 , ..., m } if m < c 2 , at the cost of possibly enlarging c . Note that the choice of M implies c 2 · (2 M ) s -1 ≥ c 2 · ( M +1) s -1 > m , and thus

with c 3 = c 3 ( s ) := 1 / 2 · c -1 / ( s -1) 2 .

Using [36, Proposition 5.9] and Lemma F.1 we can pick a 1 , ..., a m ∈ R s \ { 0 } satisfying

where we used that

Here we can assume ∥ a j ∥ 2 = r for every j ∈ { 1 , ..., m } since multiplying each a j with a positive constant does not change the span in (F.2). From [36, Corollary 5.12] we infer that

Let f ∈ C k ([ -1 , 1] s ; R ) . Then, according to Theorem C.15, there exists a polynomial P : R s → R of coordinatewise degree at most 2 M -1 satisfying

where c 4 = c 4 ( s, k ) > 0 . Note that by construction it holds P ∈ P s s (2 M -1) . Using (F.3) we deduce the existence of polynomials p 1 , ..., p m : R → R such that

Combining the previously shown bounds, we get

as desired. Here, we defined c = c ( s, k ) := c 4 · c -k 3 .

F.2 Proof of Theorem 4.2

Using Theorem F.2, we can prove the following statement for complex-valued C k -functions, which will play an important role in the proof of Theorem 4.2.

Proposition F.3. Let n, k ∈ N . Then there exists a constant c = c ( n, k ) > 0 with the following property: For any m ∈ N there exist complex vectors b 1 , ..., b m ∈ C n with ∥ b j ∥ 2 = 1 / √ 2 n for j = 1 , ..., m and with the property that for any function f ∈ C k (Ω n ; C ) there exist functions g 1 , ..., g m ∈ C (Ω 1 ; C ) such that

Note that the vectors b 1 , ...b m can be chosen independently from the considered function f , whereas g 1 , ..., g m do depend on f .

Proof. Theorem F.2 yields the existence of a constant c 1 = c 1 ( n, k ) > 0 with the property that for any m ∈ N there exist real vectors a 1 , ..., a m ∈ R 2 n with ∥ a j ∥ 2 = 1 / √ 2 n such that for any function ˜ f ∈ C k ( [ -1 , 1] 2 n ; R ) there exist functions ˜ g 1 , ..., ˜ g m ∈ C ([ -1 , 1]; R ) satisfying

We then define the vectors b 1 , ..., b m ∈ C n componentwise via

First we see ∥ b j ∥ 2 = ∥ a j ∥ 2 = 1 / √ 2 n . We first consider real-valued functions, i.e., f ∈ C k (Ω n ; R ) . Let φ n be defined as in (A.1). By the choice of the constant c 1 we can find continuous functions ˜ g 1 , ..., ˜ g m ∈ C ([ -1 , 1]; R ) such that

We then define g j ∈ C (Ω 1 ; R ) by g j ( z ) := ˜ g j (Re( z )) for any j ∈ { 1 , ..., m } . For z ∈ Ω n we then have

Therefore,

By the above, for f ∈ C k (Ω n ; C ) we can pick functions g Re 1 , ..., g Re m , g Im 1 , ..., g Im m ∈ C (Ω 1 ; R ) satisfying

Defining g j := g Re j + i · g Im j yields

completing the proof.

The special activation function that yields the improved approximation rate of m -k/ (2 n -1) (see Theorem 4.2) is constructed in the following lemma.

Lemma F.4. Let { u ℓ } ∞ ℓ =1 be an enumeration of the set of complex polynomials in z and z with coefficients in Q + i Q . Then there exists a function ϕ ∈ C ∞ ( C ; C ) with the following properties:

For every ℓ ∈ N and z ∈ Ω 1 one has

2. ϕ is non-polyharmonic.

Proof. Let ψ ∈ C ∞ ( C ; R ) with 0 ≤ ψ ≤ 1 and

where ˜ Ω := { z ∈ C : | Re( z ) | , | Im( z ) | < 3 2 } . We then define

where f ( z ) = e Re( z ) . Note that ϕ is smooth since it is a locally finite sum of smooth functions. Furthermore, ϕ is non-polyharmonic on the interior of Ω 1 , since the calculation in the proof of Proposition A.2 shows for z in the interior of Ω 1 and ρ : R → R , t ↦→ e t that

for arbitrary m,ℓ ∈ N 0 . Finally, property (1) follows directly by construction of ϕ because

̸

for ℓ = ℓ ′ .

Using the properties of the special activation function constructed in Lemma F.4 and applying the approximation result from Proposition F.3 we can now prove Theorem 4.2.

Proof of Theorem 4.2. Let ϕ be the activation function constructed in Lemma F.4. We choose the constant c according to Proposition F.3. Let m ∈ N and f ∈ C k (Ω n ; C ) . We can without loss of generality assume that f ̸≡ 0 . Again, according to Proposition F.3, we can choose ρ 1 , ..., ρ m ∈ C n with ∥ 2 ρ j ∥ = 1 / √ 2 n and g 1 , ..., g m ∈ C (Ω; C ) with the property

Recall from Lemma F.4 that { u ℓ } ∞ ℓ =1 is an enumeration of the set of complex polynomials in z and z . Hence, using the complex version of the Stone-Weierstraß-Theorem (see for instance [21, Theorem 4.51]), we can pick ℓ 1 , ..., ℓ m ∈ N such that

for every j ∈ { 1 , ..., m } . Since ϕ ( · +3 ℓ ) = u ℓ on Ω 1 for each ℓ ∈ N , and since ρ T j z ∈ Ω 1 for j ∈ { 1 , ..., m } and z ∈ Ω n , we estimate

F.3 Proof of Theorem 4.3

As a preparation for the proof of Theorem 4.3, we first prove a similar result in the real-valued setting. We remark that the proof idea is inspired by the proof of [46, Theorem 4].

Theorem F.5. Let n, k ∈ N and

be the sigmoid function. Then there exists a constant c = c ( n, k ) > 0 with the following property: If the numbers ε ∈ (0 , 1 2 ) and m ∈ N are such that for every function f ∈ C k ([ -1 , 1] n ; R ) with ∥ f ∥ C k ([ -1 , 1] n ; R ) ≤ 1 there exist coefficients ρ 1 , ..., ρ m ∈ R n , η 1 , ..., η m ∈ R and σ 1 , ..., σ m ∈ R satisfying

Proof. We first pick a function ψ ∈ C ∞ ( R n ; R ) with the property that ψ (0) = 1 and ψ ( x ) = 0 for every x ∈ R n with ∥ x ∥ 2 > 1 4 . We then choose

Now, let ε ∈ (0 , 1 2 ) and m ∈ N be arbitrary with the property stated in the formulation of the theorem. If ε > c 1 2 · 1 6 k , then m ≥ c · ε -n/k ln(1 /ε ) trivially holds (as long as c = c ( n, k ) > 0 is sufficiently small). Hence, we can assume that ε ≤ c 1 2 · 1 6 k . Now, let N be the smallest integer with N ≥ 2 , for which

and hence N ≥ 3 , whence N -1 ≥ 2 . Therefore, by minimality of N , and since N 2 ≤ N -1 because of N ≥ 2 , it follows that

then necessarily

Note that this implies

Now, for every α ∈ {-N,..., N } n pick z α ∈ { 0 , 1 } arbitrary and let y α := z α c 1 N -k . Define the function

Clearly, f ∈ C ∞ ( R n ; R ) . Furthermore, since the supports of the functions ψ ( · -α ) , α ∈ Z n are pairwise disjoint, we see for any multi-index k ∈ N n 0 with | k | ≤ k that

so we conclude that ∥ f ∥ C k ([ -1 , 1] n ; R ) ≤ 1 . Additionally, for any fixed β ∈ {-N,..., N } n we see

by choice of ψ . See also Figure 5 for an illustration of the function f .

By assumption, we can choose suitable coefficients ρ 1 , ..., ρ m ∈ R n , η 1 , ..., η m ∈ R and furthermore σ 1 , ..., σ m ∈ R such that

we see for every α ∈ {-N,..., N } n that

Therefore, we get ✶ ( ˜ g > ( c 1 / 2) N -k ) ( α ) = z α for any α ∈ {-N,..., N } n . Since the choice of z α has been arbitrary, it follows that the set shatters the whole set {-N,..., N } n . Therefore, we conclude that

On the other hand, [4, Theorem 8.11] shows that

with a suitably chosen constant c 3 = c 3 ( n ) . Here we used that N ≥ 3 so that ln( N ) ≥ ln(3) > 0 . Combining those two inequalities yields

Using that N ≥ c 4 · ε -1 /k with c 4 := c 4 ( n, k ) = ( c 1 2 k +1 ) 1 /k and N ≤ c 5 · ε -1 /k with the definition c 5 := c 5 ( n, k ) = ( c 1 2 ) 1 /k , we see that

with c 6 = c 6 ( n, k ) > 0 chosen appropriately.

for

Letting

then necessarily

As a corollary, we get a similar result for complex-valued neural networks.

Corollary F.6. Let n, k ∈ N and

Then there exists a constant c = c ( n, k ) > 0 with the following property: If ε ∈ (0 , 1 2 ) and m ∈ N are such that for every function f ∈ C k (Ω n ; C ) with ∥ f ∥ C k (Ω n ; C ) ≤ 1 there exist coefficients ρ 1 , ..., ρ m ∈ C n , η 1 , ..., η m ∈ C and σ 1 , ..., σ m ∈ C satisfying

Proof. We choose the constant c = c (2 n, k ) according to the previous Theorem F.5 and let φ n as in (A.1). Then, let ε ∈ (0 , 1 2 ) and m ∈ N with the properties assumed in the statement of the corollary. If we then take an arbitrary function f ∈ C k ( [ -1 , 1] 2 n ; R ) with ∥ f ∥ C k ([ -1 , 1] 2 n ; R ) ≤ 1 , we deduce the existence of ρ 1 , ..., ρ m ∈ C n , η 1 , ..., η m ∈ C and σ 1 , ..., σ m ∈ C , such that

In the next step, we show that

is a real-valued shallow neural network with m neurons in the hidden layer and the real sigmoid function as activation function. Then the claim follows using Theorem F.5.

For every j ∈ { 1 , ..., m } we pick a matrix ˜ ρ j ∈ R 2 n × 2 with the property that one has

for every z ∈ C n . This is possible, since this is equivalent to

for all v ∈ R 2 n , where the right-hand side is an R -linear map R 2 n → R 2 . Denoting the first column of ˜ ρ j by ̂ ρ j , we get

Writing γ for the classical real-valued sigmoid function (i.e. γ ( x ) = 1 1+ e -x ), we deduce for arbitrary x ∈ R 2 n that

where in the last step we used that γ is real-valued. As noted above, this completes the proof.

Now, we can finally prove Theorem 4.3.

Proof of Theorem 4.3. Let α = 2 n k and choose c 2 = c 2 ( α ) = c 2 ( n, k ) > 0 such that the inequality ln( x ) ≤ c 2 · x α/ 2 holds for all x ≥ 1 . Furthermore, let c 1 = c 1 ( n, k ) > 0 as in Corollary F.6. By choosing c = c ( n, k ) > 0 sufficiently small , we can ensure that ε m := c · ( m · ln( m )) -k/ (2 n ) satisfies

By further shrinking c = c ( n, k ) > 0 if necessary, we may assume

and hence c · ( m · ln( m )) -k/ (2 n ) < 1 2 for all m ∈ N ≥ 2 . Finally, setting c 3 := α 4 and shrinking c even further, we can arrange that c α < c 1 · c 3 . Now, assume towards a contradiction that for every f ∈ C k (Ω n ; C ) with ∥ f ∥ C k (Ω n ; C ) ≤ 1 there are coefficients ρ 1 , ..., ρ m ∈ C n , σ 1 , ..., σ m ∈ C and η 1 , ..., η m ∈ C such that

Applying Corollary F.6, we then get which implies

Overall we then get

Since we chose c such that c α < c 1 · c 3 we get the desired contradiction.

with ε = c · ( m · ln m ) -k/ (2 n ) ∈ (0 , 1 2 ) and c 1 = c 1 ( n, k ) > 0 . Recall from the beginning of the proof that α := 2 n/k and that ln( x ) ≤ c 2 x α/ 2 for every x ≥ 1 . We observe

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies approximation rates of shallow complex-valued neural networks (CVNN) with general non-polyharmonic activation functions. First, the paper establishes upper bounds for the error of approximation of polynomial and general smooth functions by CVNNs. Then, various aspects of the optimality of these bounds are discussed. First, it is shown that these bounds are tight assuming continuous weight selection. Next, a special activation function is presented for which the bound can be improved. Finally, it is shown that for a standard sigmoid-type activation function the rate bound is tight.\n\nSTRENGTHS:\n**Contribution.** The paper seems to be the first paper that establishes optimal approximation rates for CVNNs with general non-polyharmonic activation functions. Previous results were either limited to specific activations or only established the universal approximation property, without convergence rates. Moreover, the paper comprehensively analyzes the optimality of its convergence rates. It proves their tightness assuming a continuous weight selection and shows that faster rates can be achieved if this assumption is dropped and a specially designed activation is used. It should be noted, however, that the CVNN model considered in the paper does not seem to be very important, and that most results of the paper are analogs of existing results for usual real-valued neural networks - see Weaknesses below.\n\n**Quality and clarity.** The paper is very well written. The theorems are clearly stated, sketches of key ideas of the proofs are provided in the main text. Proof details are provided in a large appendix and also seem to be carefully written, though I did not study all of them closely.\n\nWEAKNESSES:\n**Questionable significance of the CVNN model.** The usual real-valued neural networks are important both mathematically and practically. They represent simple and natural non-linear models whose significance is well-established. In contrast, CVNNs do not seem to be significant from either perspective. Mathematically, they mix holomorphic linear operations with generic non-linear operations. This combination does not seem to have interesting analytic properties (except for the original observation by Voigtlaender that the condition of non-polynomiality for RVNN gets replaced by non-polyharmonicity for CVNN) or obvious practical or computational meaning. The paper does not explain why CVNNs are important, instead referring to a small number of papers from 2016-2018 where a similar structure was applied. \n\n**Limited conceptual novelty.** The paper generally adapts existing methods and results from the real-valued to the complex-valued setting - admittedly with many extra new twists. The established convergence rate is the same as for the respective real network with doubled real dimension.\n\n**High technicality for a conference.** This is quite a technical paper with 40 pages of proofs in the appendix. It is unlikely that any NeurIPS reviewer properly checks all these, so the paper might be more suitable for a journal with a more comprehensive review process. However, this is only a minor point, since the paper makes a good effort to present key ideas and sketches of proofs already in the main text.\n\nQUESTIONS:\nWhy specifically is the particular computational structure used in CVNNs (a layer of holomorphic linear operations + a layer of pointwise non-holomorphic non-linear operations + another layer of holomorphic linear operations) important?\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies approximation error bounds for complex-valued neural networks based. The authors relied on several techniques from the work by Mhaskar (1996) and proved several theorems in the paper. I will summarize two important results here.\n1. Given a function from $C^k$, an optimal error bound is proved under some conditions on the activation function and the hypothesis of continuous weight selection. Furthermore, the complex-valued neural network achieving this bound has a universal first layer.\n2. When the hypothesis of continuous weight selection is dropped, the authors proved that there exists an activation function allowing the complex-valued neural network to achieve a better bound. No optimality is claimed under this case. On the other hand, there is an activation function such that the bound cannot be improved up to logarithmic factors.\n\nSTRENGTHS:\nThe novelty of this paper is clear. The paper provides error bounds for complex-valued neural networks using more general activation functions and shows that the optimal bound can be achieved under the hypothesis of continuous weight selection. The results in this paper generalize the results in the previous work [9]. The problem is well-motivated, and the theorems proved are sound. Overall, the paper is well-written, and I enjoy reading the paper.\n\nWEAKNESSES:\nThe hypothesis of continuous weight selection is not a practical assumption. The optimal choice of the network weights is discontinuous in general (see the reference below). It would be better if the authors could make this clear in the paper.\n\nKainen, Paul C., Věra Kůrková, and Andrew Vogt. \"Approximation by neural networks is not continuous.\" Neurocomputing 29, no. 1-3 (1999): 47-56.\n\nUnder this assumption, the error bound can be proved to be optimal. However, no optimal results are provided if this assumption is dropped. Regarding this, it would be more convincing for the paper if the authors could provide insights into difficulties that arose in proving the statements and potential workarounds.\n\nThe results provided are similar to [9] so it would be more interesting if the authors could describe what difficulties they have overcome to prove these results. Given the high similarity of the results, I would expect more discussions in the related work section or in the descriptions of the proof sketch.\n\nQUESTIONS:\nLine 28: Would it be more precise if referencing the work by Cybenko in 1989?\n\nLine 140: What is your definition of a smooth function? Please make it precise.\n\nFrom Theorem 3.1, the complexity of an approximating complex-value network is established for a polynomial. Would it be possible to use the Stone-Weierstrass theorem to establish an approximation bound? This seems to be a natural step to apply.\n\nLIMITATIONS:\nThe authors have sufficiently addressed the limitations of the paper in Section 5. In my view, the greatest limitation of this work is the use of continuous weight selection for guaranteeing the optimality of the error bound. It would be more convincing if the authors can give some insights into this in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper studies the approximation power of complex-valued neural networks (CVNNs). They derive that the approximation error is with the order $m^{-k/2n}$, where m is the number of neurons, k is the smoothness of the target function, and n is the input dimension of the neural network. They also show that this approximation error is optimal under some mild assumptions.

STRENGTHS:
1. This paper derives the approximation bounds of CVNNs for any continuous activation functions.
2. Furthermore, they show that the approximation bounds they obtain are optimal with a natural continuity assumption.
3. For several specific activation functions, the authors derive the upper bounds without the assumption of the continuity of weight selection.

WEAKNESSES:
1. For general activation functions, the authors need to assume the continuity of weight selection to derive the optimal approximation rates. It will be more convincing if they can show results for general activation functions without the assumption of the continuity of weight selection.

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the expressive power of complex-valued neural networks. It is shown that depth 2 networks with non-polyharmonic (the complex equivalent of non-polynomial) activations can approximate any continuous function on a compact domain to arbitrary accuracy which decays polynomially with the width and exponentially with the smoothness of the target function, but nevertheless suffers from the curse of dimensionality. This upper bound is also shown to be optimal in general, and that furthermore, there are more tailored examples where one can slightly improve the approximation rate.\n\n\nPost-rebuttal:\n\nI still find the technical contribution of the paper rather fair (as my original score indicates). My main concern with the paper is that its impact feels somewhat limited as it provides a guarantee which suffers from the curse of dimensionality. While I understand that this cannot be evaded in general, obtaining a result which is tight in the worst case where the worst case is intractable is a clear limitation. While I appreciate the fact that there are other older results in the literature which are highly cited and obtain such results for the real setting, this paper's merits should be considered compared to what is already known, and from this perspective, obtaining results in the complex setting that are analogous to the real setting is of interest but not groundbreaking. I have therefore decided to keep my original score.\n\nSTRENGTHS:\n- The problem feels overall well-motivated, and CVNNs seem like an interesting model to study.\n\n- The analogies and differences between real and complex neural networks highlighted by the results in this paper are interesting.\n\nWEAKNESSES:\n- On a technical perspective, it seems like the main technical contributions here are to adapt existing technique to the CVNN setting. Most results seem to take existing proofs and just apply them to the complex setting, which doesn't feel novel enough.\n\n- The exact dependence on the rate of approximation is unclear. There's a `constant' $c(n,k)$ which hides some dependence on the input dimension and smoothness parameters, which could be potentially huge (see questions below).\n\n- The separation established in Section 4 by Theorem 4.2 and Theorem 4.3 is not particularly strong, and is only significant in cases where the input dimension $n$ is very small.\n\n- I didn't find the main approximation result in Theorem 3.2 very surprising. This is an analogous result for something which is already known for real-valued networks, so obtaining this result for complex-valued functions feels incremental. In particular, it would be of greater interest, in my opinion, to study cases where complex-valued networks attain an \\emph{efficient} approximation in the input dimension $n$, since these have stronger practical implications.\n\nQUESTIONS:\nQuestions:\n\n- \"complex-valued networks behave significantly different from real-valued networks\": I don't see how this is `significantly different'. The second property does indicate a difference, but the first characteristic is analogous (non-polynomial and non-polyhedral activations).\n\n- There's no clear motivation for the very technical choice of the $C^k$-norm in the paper. Can you provide a more intuitive explanation for it and why it is chosen? In particular, why is this choice not stylized for obtaining the main result?\n\n- \"It is crucial that the size of the networks considered in Theorem 3.1 is independent of the approximation quality $\\varepsilon$\": Why is this important?\n\n- What is the quantity $c(n,k)$ which appears in many of the theorem? The bounds in the paper are interesting since they allow us to study the dependence of the accuracy attained as a function of, say, $n$ for some fixed $k$; but the dependence of $c(n,k)$ on $n$ isn't clear. I tried to better understand it by looking at the proofs in the appendix but it's not made explicit there as well. $c(n,k)$ could, potentially, have magnitude $m^{k/(2n)}$ which would render the upper bound vacuous. Why is this not the case? Is this the same quantity in both the upper bound in Theorem 3.2 and the lower bound in Theorem 4.3? If not, it could make the either bound very loose despite appearing to provide a tight result.\n\n\nComments:\n\n- Abstract: \"the real-valued case is supported by a firm mathematical foundation\" -- I wouldn't say our mathematical understanding of deep learning is firm.\n\n- The abstract doesn't explicitly state what are the functions the main result applies to.\n\n- Line 27: The universal approximation theorem dates back to 1989. See \"Approximation by Superpositions of a Sigmoidal Function\" by G. Cybenko\n\n- \"CVNNs have the same excellent approximation properties as real-valued networks.\": Why is this conclusion reached? The results in the paper hold for a very broad class of smooth function and therefore suffer from the curse of dimensionality in the worst case.\n\nLIMITATIONS:\n- I think that the lack of clarity in the theorem statements regarding $c(n,k)$ is problematic, yet this is not discussed in the paper. I urge the authors to clarify this and discuss this more clearly.\n\n- The limitations of the provided results which suffer from the curse of dimensionality should be clearly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}