Title: Structure of universal formulas
Venue: NeurIPS 2023
Accepted: yes
Keywords: VC-dimension

neural networks

activation functions

approximation

polynomials

algebraic functions
Abstract: By universal formulas we understand parameterized analytic expressions that have a fixed complexity, but nevertheless can approximate any continuous function on a compact set. There exist various examples of such formulas, including some in the form of neural networks. In this paper we analyze the essential structural elements of these highly expressive models. We introduce a hierarchy of expressiveness classes connecting the global approximability property to the weaker property of infinite VC dimension, and prove a series of classification results for several increasingly complex functional families. In particular, we introduce a general family of polynomially-exponentially-algebraic functions that, as we prove, is subject to polynomial constraints. As a consequence, we show that fixed-size neural networks with not more than one layer of neurons having transcendental activations (e.g., sine or standard sigmoid) cannot in general approximate functions on arbitrary finite sets. On the other hand, we give examples of functional families, including two-hidden-layer neural networks, that approximate functions on arbitrary finite sets, but fail to do that  on the whole domain of definition.

Structure of universal formulas

Abstract

By universal formulas we understand parameterized analytic expressions that have a fixed complexity, but nevertheless can approximate any continuous function on a compact set. There exist various examples of such formulas, including some in the form of neural networks. In this paper we analyze the essential structural elements of these highly expressive models. We introduce a hierarchy of expressiveness classes connecting the global approximability property to the weaker property of infinite VC dimension, and prove a series of classification results for several increasingly complex functional families. In particular, we introduce a general family of polynomially-exponentially-algebraic functions that, as we prove, is subject to polynomial constraints. As a consequence, we show that fixed-size neural networks with not more than one layer of neurons having transcendental activations (e.g., sine or standard sigmoid) cannot in general approximate functions on arbitrary finite sets. On the other hand, we give examples of functional families, including two-hidden-layer neural networks, that approximate functions on arbitrary finite sets, but fail to do that on the whole domain of definition.

1 Introduction

By universal formulas we broadly (informally) understand parameterized explicit analytic expressions that have a fixed complexity (as expressions) but nevertheless can approximate any continuous function on a compact set. An example of such formula, given by Boshernitzan [1], is

where d > 0 , a, b and c are parameters. [1] proves that, by varying the parameters, functions (1) can uniformly approximate any continuous function on any compact interval in R .

Expression (1) involves integration. Laczkovich and Ruzsa [13] prove existence of universal formulas representable as elementary functions without integration (moreover, their formulas can uniformly approximate continuous functions on the whole R under a mild growth assumption).

One can further restrict the types of operations and show that universal formulas can be realized by fixed-size classical neural networks with suitable activation functions [29]. By a classical neural network we mean a computational model consisting of units ('hidden neurons'), each computing a map z 1 , . . . , z n ↦→ σ ( ∑ n k =1 w k z k + h ) , where z 1 , . . . , z n (signals) are outputs of some previous neurons, and w k and h are weights (parameters) of the neuron. In addition to hidden neurons, one distinguishes 'input neurons' that just represent the scalar components of the network input, and 'output neuron(s)' that represent the scalar component(s) of the output. The output neurons work as hidden neurons but without activations, i.e. as z 1 , . . . , z n ↦→ ∑ n k =1 w k z k + h. As defined, neural networks are fairly restrictive in terms of used operations: for example, they contain multiplications of signals z by constants (weights) w , but not multiplications between signals, z 1 z 2 . However, this and some other extra operations can be implemented with any accuracy using suitable combinations of neurons. The universal network in [29] uses a fixed architecture graph, activations sin and arcsin ,

x

f

(

x

)

σ

(

x

) =

x

σ

= sin

σ

z

1

, z

2

z

1

z

2

=

and can approximate any f ∈ C ([0 , 1]) (see Figure 1). Another fixed-size universal network had been constructed earlier in [15], but the activation was not an elementary function in that work.

The Kolmogorov(-Arnold) Superposition Theorem [10] shows that any function f ∈ C ([0 , 1] d ) can be expressed in terms of a fixed number (only depending on d ) of summations and univariate continuous functions. This implies that multivariate universal formulas can be easily constructed from univariate ones, by plugging the latter in the Kolmogorov ansatz. Accordingly, the question of multivariate universal formulas is not significantly more interesting or complicated than its counterpart for univariate formulas, and in this paper restrict our attention to the latter. We also mention in passing another interesting property of universal formulas: they naturally give rise to universal polynomial differential equations P ( x, f, f ′ , . . . , f ( n ) ) = 0 , with a polynomial P and a dense set of solutions in C ([ a, b ]) , see [21, 2, 1, 19, 13]. (In fact, one can find such a polynomial for any parameterized family of elementary functions; the density of solutions then follows from the universality of the formula.) See [22] for a connection to analog computers.

A foundational theorem on neural networks is the universal approximation theorem (UAT). It states that, for a broad class of activation functions, a neural network with a single layer of hidden neurons can approximate any continuous function on a compact set if we increase the number of neurons [4, 18]. This result is very different from, and should not be confused with the above fixed-size universal networks of [15] or [29]. In UAT, universality is achieved by increasing the number of parameters, and accordingly UAT imposes much weaker requirements on the activation functions (e.g., it holds for any continuous non-polynomial activation, see [14, 18]).

On the other hand, the fixed-size universal network in Figure 1 is fairly simple. One of the two activation functions used there, sin , is actually used as an activation in practical multi-layer networks (e.g., the popular SIREN network of [23]). This raises the following natural question motivating the present paper: can practically used neural-network-type models be universal as fixed-size formulas (i.e., without increasing the number of neurons) ?

There is an important caveat here: the universality of universal formulas is, of course, an idealization that requires their parameters to have unbounded precision or magnitude. If parameters are finitely defined so as to be representable on real computer, one can give a simple entropy bound constraining feasible approximation accuracy. Specifically, suppose that a formula contains p parameters, each somehow represented using B bits (e.g., B = 32 or 64 for standard floats). Suppose we are approximating a function f at N points x 1 , . . . , x N with absolute accuracy ϵ . Suppose finally that | f | is bounded by M and f ( x k ) , k = 1 , . . . , N, can be chosen independently subject to this constraint. Then

This shows that in a practical application with parameters implemented as standard floats, universal formulas such as (1) can typically produce only crude approximations. In the sequel, we will ignore these finite-precision aspects.

Our contribution. Our goal in this paper is to understand which structural properties are critical for universal formulas. This question does not seem to be simple. As a general strategy of tackling it, we analyze a sequence of progressively more complex parametric functional families with respect to several relevant classes of model expressiveness. Our specific contributions are as follows.

We introduce a hierarchy G 0 ⊃ G 1 ⊃ G 2 of classes of model expressiveness (Section 2). Here G 0 represents the families of infinite VC-dimension, G 1 the families achieving approximation on arbitrary finite sets, and G 2 the families achieving uniform approximation on the whole segment [0 , 1] . Universal formulas correspond to the class G 2 . The class G 0 is important because elementary functions belonging to it must involve, in some form, the function sin acting on an unbounded domain (Section 3). We argue that the class G 1 is also important because, as we show, some functional families cannot belong to it due to algebraic (polynomial) constraints.

As a first result on algebraic constraints, we consider the family of fixed-size linear combinations of sine waves with unbounded weights (equivalently, fixed-size single-hidden-layer neural networks with activation sin ) and prove that it belongs to the class G 0 \ G 1 . Moreover, we give a complete description of the limit points of this family (Section 4).

As a generalization of the previous result, we introduce a general family of polynomiallyexponentially-algebraic expressions of bounded complexity, and prove that it is also subject to polynomial constraints and lies outside G 1 (Section 5).

We give an application of the previous result to branching expressions and neural networks with conventional activations (Section 6). Specifically, we prove that if the network has a bounded complexity and possibly multiple layers, but only one layer with transcendental activation functions (such as sin x , Gaussian or standard sigmoid), then the network is subject to polynomial constraints and lies outside G 1 . Moreover, polynomial constraints hold even if the activation functions are only piecewise analytic, as is common in practice.

The situation is drastically different for formulas involving compositions of non-polynomial analytic functions and sin : such formulas generally belong to G 1 (Section 7). It seems hard to separate the classes G 1 \ G 2 and G 2 . We give a couple of examples of families for which we can prove that they belong to G 1 \ G 2 . One of these families can be described as that of fixed-complexity two-hidden-layer networks with sin activation that have a single neuron in the second hidden layer. We conjecture that general sine networks of fixed complexity also belong to G 1 \ G 2 .

Most proofs are deferred to the appendix (the respective sections are indicated in the theorems).

2 The hierarchy of expressiveness classes

Throughout the paper, we consider various families H of functions g : [0 , 1] → R . We define for them expressiveness classes G 0 , G 1 , G 2 :

H ∈ G 1 def ⇐⇒ For any finite subset { x 1 , . . . , x N } ∈ [0 , 1] , any y 1 , . . . , y N ∈ R and any ϵ > 0 there is g ∈ H such that | g ( x k ) -y k | < ϵ for all k = 1 , . . . , N.

Here, VCdim( H f ) = ∞ means that for any N ∈ N we can find a sizeN subset X ⊂ [0 , 1] shattered by H thresholded at 0, i.e. such that for any S ⊂ X there is g ∈ H for which g ( x ) > 0 if x ∈ S and g ( x ) ≤ 0 if x ∈ X \ S .

Clearly,

The classes G 1 and G 2 reflect approximability on arbitrary finite sets and on the whole interval [0 , 1] , respectively. Universal formulas as discussed in the Introduction correspond to the class G 2 . One can

say that G 1 corresponds to universality in a weaker sense, which is particularly relevant if we try to learn the parameters using a finite training set { ( x k , y k ) } .

We will consider several families H that we will distinguish by superscripts denoting or enumerating their types (e.g. H f or H (1) , H (2) , . . . ) and by subscripts reflecting complexity within the type (e.g., H (2) N will denote linear combinations of N sine waves). Most of these families admit natural interpretations in terms of conventional neural networks - we will mention them along the way.

While the remainder of this work focuses on the mathematical analysis of the relation between various functional families and the classes G k , let us briefly clarify the role of these classes from the general perspective of approximation and learning.

Most conventional learning models (e.g., neural networks) can be said to be defined using (piecewise) elementary functions. If the model is assumed to have a bounded complexity (e.g., a bounded number of neurons) and its weights and signals are bounded, this model falls outside even the broadest class G 0 (see Section 3 below). This does not apply, however, to models containing the function sin acting on an unbounded domain.

For a fixed-complexity model, being outside the classes G k and thus not having the full approximation power is not a practically negative property: this just means that the complexity of the model needs to increase to achieve a better fit.

The situation when a fixed-complexity model belongs to the class G 2 (i.e., is a universal formula) is mathematically interesting but somewhat exotic from the perspective of conventional computation, since the respective model parameters need to contain a potentially unbounded amount of information.

The class G 1 \ G 2 seems especially dangerous from the generalization point of view: given a hypothesis space H from this class, we can fit a model arbitrarily well to a generic continuous ground truth on any finite training set, but can hardly ever uniformly approximate this ground truth on the whole domain. This cannot be said about the class G 2 . In Section 7 we give examples of functional families (neural networks) provably belonging to this dangerous class G 1 \ G 2 .

3 Pfaffian functions

There is an important broad class of parameterized functions f ( x, w ) for which one can guarantee that the family H f obtained by fixing various w has a finite VC-dimension. This class is most naturally described in terms of Pfaffian functions introduced by [9]. Pfaffian functions can be defined using function sequences in which, at each step, the next function satisfies polynomial first-order differential equations involving itself and previous functions (see [9, 5, 30] for details). In particular, Pfaffian functions include all elementary functions when the latter are defined on suitable domains.

Importantly, in contrast to functions like ln , exp and polynomials, which are Pfaffian on their whole natural domain of definition, the function sin is not Pfaffian on the whole set R , though it is Pfaffian on any bounded interval ( a, b ) (moreover, the representation of sin as a Pfaffian function depends on ( a, b ) and becomes more complex as b -a increases). For this reason, elementary functions are Pfaffian only when they involve sin (or a closely related function, like cos ) restricted to bounded intervals.

The fundamental theorem of [9] gives an explicit finite upper bound on the number of zeros of Pfaffian functions, which implies, in particular, that the respective parametric families have finite VC-dimension. We state this result specialized to elementary functions and in the form suitable for our purposes; see [9, 5, 30, 29] for details.

Theorem 1. Suppose that a formula y = f ( x, w ) is constructed from the variables x and w 1 , . . . w d using real numbers, standard arithmetic operations ( + , -, × , / ), elementary functions ln , exp , sin , arcsin , and compositions. Suppose that there is a nonempty subset W ⊂ R d such that for any w = ( w 1 , . . . , w d ) ∈ W the function f ( · , w ) : [0 , 1] → R ) is well-defined, i.e., ln and arcsin are applied on the intervals (0 , ∞ ) and ( -1 , 1) , respectively, and there is no division by 0. Moreover, suppose that there is a bounded interval ( a, b ) to which the arguments of sin always belong for all w ∈ W . Then the family H f = { f ( · , w ) } w ∈ W has a finite VC-dimension.

This theorem shows that an elementary function has chances to be universal only if it includes sin (or some related function) acting on an unbounded domain. Therefore, we will focus on these latter functions in the sequel.

It is crucial for Theorem 1 that all the computations in the formula are real-valued and not complexvalued (the proof eventually boils down to Rolle's theorem, which does not hold for C -valued functions). In particular, we cannot bypass the domain restriction of sin by simply writing sin x = ( e ix -e -ix ) / 2 i . In contrast, many results in the next sections will be essentially algebraic and will hold for complex-valued operations.

We remark that Sontag [25] considers a class of formulas closely related to Pfaffian functions and proves that shattering k -point sets in general position requires at least ( k -1) / 2 parameters in such formulas.

4 Linear combinations of sines

A simple example of a family with an infinite VC-dimension is H (1) = { c sin( ωx ) } c,ω ∈ R . This is usually proved (see e.g. [28], Section 3.6) by giving a particular example of an arbitrarily large finite set shattered by H (1) . For our purposes, however, it is instructive to describe a general collection of finite sets on which H (1) has universal approximability. We say that numbers x 1 , x 2 , . . . are rationally independent if they are linearly independent over the field Q of rational numbers.

Theorem 2. Let X = { x 1 , . . . , x N } be a finite subset of [0 , 1] such that the values x 1 , . . . , x N are rationally independent. Then the family H (1) can approximate any R -valued function on X .

Proof. Consider the torus T N = ( R / 2 π Z ) N ; its points can be identified with the points of the cube [0 , 2 π ) N . Let ⌊·⌋ : R → [0 , 1) be the standard floor function. By a classical Kronecker's theorem [11, 7], the set { (2 π ( θx 1 -⌊ θx 1 ⌋ ) , . . . , 2 π ( θx N -⌊ θx N ⌋ )) } θ ∈ R is a dense subset of T N if and only if the numbers x 1 , . . . , x N are rationally independent. Using the 2 π -periodicity of sin , this implies immediately that if x 1 , . . . , x N are rationally independent, then { (sin( ωx 1 ) , . . . , sin( ωx N )) } ω ∈ R is dense in ( -1 , 1) N and hence { ( c sin( ωx 1 ) , . . . , c sin( ωx N )) } c,ω ∈ R is dense in R N .

̸

This result shows that the family H (1) ∈ G 0 . It is also clear that H (1) / ∈ G 1 (e.g., since g (0) = 0 for any g ∈ H (1) ). In fact, the sufficient condition of rational independence in the theorem is close to also being necessary. In particular, a simple kind of sets not shattered by H (1) are all nontrivial arithmetic progressions x k = u + vk, v = 0 , of length 5. Indeed, it follows from the identity sin( u +( k -1) v ) + sin( u +( k +1) v ) = 2 sin( u + kv ) cos( v ) that the values of functions from H (1) cannot, for example, have the sign configuration +++ -+ on such progressions.

Consider now the more general family consisting of all linear combinations of a fixed number of sine waves with arbitrary frequencies and phases:

One can view this family as representing single-hidden-layer neural networks with the sin activation function ( ω n and h n are the weights and biases in the hidden layer, and c n are the weights in the output layer).

One can again easily show that H (2) N ∈ G 0 \G 1 , by observing that the values of all functions g ∈ H (2) N are constrained by common nontrivial polynomial equations:

Theorem 3.

det A is a nonzero polynomial in the variables g ( x ) , where x ∈ X = { x 0 + α k + β m | k, m = 0 , . . . , 2 N } , if and only if all α k are different and all β m are different.

Proof. 1. The matrix A is degenerate because each of its 2 N +1 rows is a linear combination of 2 N vectors ( e sω n β 1 , . . . , e sω n β N ) with n = 1 , . . . , N and s = ± i.

2. If some α k or β m are repeated, then the matrix A contains repeated columns or rows and so the polynomial det A identically vanishes. Conversely, suppose that all α k are different and all β m are different. Without loss, we can assume α 0 < . . . < α 2 N and β 0 < . . . < β 2 N . We can then expand det A = g ( x 0 + α 2 N + β 2 N ) det A ′ + . . . , where det A ′ is the top left 2 N × 2 N minor and the terms . . . do not contain the variable g ( x 0 + α 2 N + β 2 N ) . Repeating the expansion for A ′ , etc., we see that the polynomial det A contains the monomial ∏ 2 N k =0 g ( x 0 + α k + β k ) with coefficient 1, i.e. is nonzero.

This result shows that there are (infinitely many) nontrivial polynomial constraints on the values of the functions g ∈ H (2) N . Importantly, these constraints are preserved under pointwise limits, so, in particular, H (2) N ∈ G 0 \ G 1 . Taking N = 1 and α k = β k = vk, k = 0 , 1 , 2 , we obtain a nontrivial constraint for the values of g on any nontrivial length-5 arithmetic progression x k = u + vk , in agreement with an earlier observation.

In fact, thanks to the semi-linear structure of the family H (2) N , we can go further and give an explicit description of its limit points. It turns out that the limiting functions can only differ from the original combinations of sine waves by resonant factors x m :

Theorem 4 (A) . Let N be fixed. Then a function f ∗ is a limit point of H (2) N in the sense of uniform convergence on the segment [0 , 1] if and only if

where M 0 ≥ 0 , M k ≥ 1 for k = 1 , . . . , K , and ∑ K k =0 M k = N .

Resonant factors appear by taking suitable combinations of sine waves at close frequencies, e.g. x sin( ωx ) = lim ∆ x → 0 (∆ ω ) -1 (sin(( ω +∆ ω ) x ) -sin( ωx )) . We show explicitly how to construct resonances of arbitrary degrees subject to the constraint ∑ K k =0 M k = N . Note that the frequency ω = 0 corresponding to the first sum in expansion (6) is special as it is associated with a potential double-degree resonance.

The more difficult part of the proof is the 'only if' part, i.e. that any limiting function has the form (6). Our basic idea is to show that a limiting function is subject to a linear differential equation describing waves with a given set of frequencies; the general resonant form (6) would then appear from the usual solution of such an equation with nontrivial multiplicities of the frequencies. This strategy, however, is not so easy to implement rigorously, since in our setting the frequencies are unbounded. We give two alternative proofs overcoming this difficulty. The first, clustering-based proof, groups the frequencies found in a converging sequence f n ∈ H (2) N into spatially separated 'frequency clusters', and then shows that unbounded clusters can be removed without changing the limit. The second, discretization-based proof, starts with describing the discretized functions of H (2) N by suitable linear finite-difference equations which also hold for the limiting function f ∗ . Then, we consider these discretizations at different length scales and show that they can only be consistent with each other and with the continuity of the uniform limit f ∗ if representation (6) holds.

5 Polynomially-exponentially-algebraic expressions

Theorem 3 shows that the values of functions H (2) N are subject to nontrivial polynomial constraints preserved under pointwise limits. We extend this argument to a much larger family of functions, involving complex algebraic operations. This generaliztion will allow us, in particular, to prove a related result for multi-layer neural networks.

If U ⊂ C N is an open domain, we say that a holomorphic function Q : U → C is an algebraic function if there exist N -variable polynomials ϕ 0 , . . . , ϕ q , not all zero, such that

The value q is called the degree of the algebraic function Q. Algebraic functions include, in particular, all polynomials and (on their holomorphy domains) rational functions and powers z a with rational a . Arithmetic operations and compositions applied to algebraic function produce again algebraic functions.

We define the family H (3) N,q,r,d as consisting of all functions g : [0 , 1] → R of the form

where P 1 , . . . , P N are complex-valued polynomials of degree ≤ d, and Q is an algebraic function of degree ≤ q such that all polynomials ϕ k appearing in its definition have degree ≤ r . Here, the functions Q are assumed to be defined on some domains U ⊂ C N +1 that contain all the values ( x, e P 1 ( x ) , . . . , e P N ( x ) ) , x ∈ [0 , 1] . Our main result is the following extension of Theorem 3.

Theorem 5 (B) . Fix N,q,r, d . Let m = ( q +1) ( N + r +1 N +1 ) +(max(1 , d ) + 1)( N +1) . Then there exists a nonzero polynomial R of m +1 variables such that for any g ∈ H (3) N,q,r,d and any a, b ∈ [0 , 1]

As a corollary, since the polynomial R is the same for all g ∈ H (3) N,q,r,d , constraint (9) is preserved by pointwise limits, so H (3) N,q,r,d / ∈ G 1 .

Our proof of Theorem 5 is inspired by the theory of algebraic-transcendent functions [16, 17, 8] that addresses representability of functions by algebraic differential equations. Our context is rather different: we are interested in the pointwise convergence of functions (or convergence in the uniform norm), so conditions involving derivatives are not applicable in our setting. We observe, however, that some elements of this theory can be extended from the usual to discretized derivatives. This is why, in particular, the conditions in Eq. (9) are formulated for a regular grid of points a, a + h, . . . , b .

Since the polynomials P n in the definition of H (3) N,q,r,d are allowed to be complex-valued, the families H (3) N,q,r,d subsume the sine wave families H (2) N considered in the previous section:

6 Branching expressions and neural networks

We describe now an application of Theorem 5 to neural networks. We start by noting that many standard activation functions can be classified as piecewise polynomial or (piecewise) algebraicexponential 1 :

Piecewise polynomial: Binary step function σ ( x ) = 1 [ x ≥ 0] , ReLU σ ( x ) = max(0 , x ) , leaky ReLU σ ( x ) = max( ax, x ) , squared ReLU σ ( x ) = max 2 (0 , x ) [24].

(Piecewise) algebraic-exponential: sin x , tanh x , standard sigmoid σ ( x ) = (1 + e -x ) -1 , ELU

'Piecewise' here means that R is in general divided into several subsets (in fact, up to two subsets in the above examples) on which the activation is defined by different analytic formulas. We will refer to this as 'branching'. On each branch, activations of the first type are polynomials, and those of the second type can be represented in the form (8) with a polynomial or rational Q .

Define the family H (4) N of functions g : [0 , 1] → R as realization, with various weight assignment, of the following class of feedforward neural networks. Suppose that the underlying directed acyclic graph of the network contains at most N neurons. Suppose that each hidden neuron is equipped with one of the piecewise polynomial or algebraic-exponential activation functions mentioned above (possibly different at different neurons). Suppose finally that any directed path from the input to the output neuron contains at most one (piecewise) algebraic-exponential neuron (for example, this is the case if the network has a single algebraic-exponential layer, while the other layers are piecewise polynomial). Then we have

Theorem 6 (C) . Given N , there exists m and a nontrivial polynomial R of m +1 variables such that for any g ∈ H (4) N and any a, b ∈ [0 , 1]

The main obstacle in deriving this theorem from Theorem 5 is the branching in the activations that destroys the global analytic structure of functions g ∈ H (4) N . We overcome this obstacle using the well-known Van der Waerden's theorem on arithmetic progressions.

Theorem 7 ([27]; see, e.g., [26] for a short proof) . For any integers s, p ≥ 1 there exists an integer N vdW ( s, p ) ≥ 1 such that every coloring C : { 1 , ..., N vdW } → { 1 , ..., p } of { 1 , ..., N vdW } into p colors contains at least one monochromatic arithmetic progression of length s (i.e. a progression in { 1 , ..., N vdW } of cardinality s on which C is constant).

7 Compositions of sines and non-polynomial functions

The polynomially-exponentially-algebraic structure (8) is actually close to being necessary for the polynomial constraints of Theorem 5, as we show by the following example. Let σ be a real analytic function on the interval [ -1 , 1] . Let f ( x ) = c sin( ωσ ( bx ) + h ) with parameters c, b, h ∈ R and b ∈ [ -1 , 1] . Denote the respective family by H σ .

Theorem 8 (D) . H σ ∈ G 1 if and only if σ is not a polynomial.

This shows that the polynomial constraints of Theorem 5 are destroyed if we relax even slightly the requirement of polynomial arguments of the exponentials in Eq. (8). They are also destroyed if we replace the algebraic function Q by the sine function.

While H σ ∈ G 1 for non-polynomial σ, we can show that H σ is never in G 2 , by directly finding all limit points of H σ under uniform limits. Let us call the limit points of H σ not belonging to H σ nontrivial .

̸

Theorem 9 (E) . For a non-constant σ , the only nontrivial limit points of the family H σ are a 0 + a r x r , aσ ( bx ) + c, and c sin( a 0 + a r x r ) , where r = arg min( m > 0 : d m σ dx m (0) = 0) and a 0 , a r , a, b, c ∈ R . In particular, H σ / ∈ G 2 .

The families H σ with non-polynomial σ thus provide examples of elements of the class G 1 \ G 2 of finitely- but not globally-universal formulas. Consider now a more complex family H (5) N consisting of the functions

where H (2) N consists of linear combinations of N sine waves as defined earlier in Eq. (4). One can interpret the family H (5) N as a two-hidden-layer neural network with the sin activation function, N neurons in the first hidden layer, and a single neuron in the second hidden layer. Using the same approach of examining the limit points and invoking additional topological arguments, we prove that the family H (5) N belongs to the same class as H σ :

It appears that the class G 1 \ G 2 is quite generic - we conjecture that an analog of Theorem 10 holds for any multi-layer neural network with the sin activation and a bounded number of neurons. However, proving such a general result by explicitly describing the limit points as in Theorems 9, 10 is difficult. We are not aware of simple conditions allowing to separate G 1 \ G 2 from G 2 .

In this context, let us discuss the structure of universal formulas mentioned in the Introduction. Boshernitzan's formula (1) includes the factor bd 1+ d 2 -cos( bt ) which is of the form (8) and hence in G 1 , and the transcendental composition cos( e t ) . This agrees with our results showing that compositions of sines and non-polynomial functions are essential for universal formulas. However, Boshernitzan's formula also includes integration, which effectively serves to produce an approximation to a piecewise constant function. The neural network in Figure 1 and the formulas of [13] combine sines with arcsines, which serves to approximate periodic piecewise linear or piecewise constant functions. It seems that approximate piecewise constant functions are an essential element of existing universal

formulas, and so one can conjecture that arcsine or a related inverse trigonometric function is a necessary component of an elementary universal formula without integration.

In the fairly different context of recursive decidability, there is a remarkably similar sharp difference between polynomial-sine and more complex compositions [12]. Let S 1 denote the ring generated by the integers and the expressions x, sin x n and sin( x sin x n )( n = 1 , 2 , . . . ) . Let also S 2 denote the ring generated by the integers and the expressions sin x n and cos x n ( n = 1 , 2 , . . . ) . Then the predicate that there exists a real x such that f ( x ) > 0 is undecidable for f ∈ S 1 , but decidable for f ∈ S 2 .

8 Discussion

The expressiveness hierarchy. To analyze the necessary features of the structure of universal formulas, we have examined the chain of expressiveness classes G 0 ⊃ G 1 ⊃ G 2 corresponding, respectively, to the families having the infinite VC-dimension, general approximability on all finite sets, and general global uniform approximability. Membership in the class G 0 can be conveniently studied using the theory of Pfaffian functions, while, as we have shown, membership in the class G 1 can be conveniently studied using algebraic methods. In particular, our results suggest that globally universal formulas need to involve suitable compositions of non-polynomial and sine operations.

Transcendental structure of universal formulas. Our main result in Section 5 shows that combining a single layer of exponential and sine operations with polynomials and algebraic functions has a fairly weak expressiveness, in the sense that such formulas of bounded complexity are constrained by polynomial relations and do not even have general approximability on finite sets. In this family, the requirement on the operations preceding the exponential/sine layer is much stronger (polynomial) than the requirement on the subsequent operations (algebraic). As Theorem 8 shows, any non-polynomial operation preceding the sine operation and combined with linear operations restores the general finite approximability.

General methods. All results presented in this paper that rule out global universality are essentially obtained by one of the three general methods: Khovansky's analytic theory of Pfaffian functions (Theorem 1), finding algebraic constraints (Theorems 3, 5, 6), and direct computations of limit functions (Theorems 4, 9, 10). It appears that all these methods are insufficient to conveniently separate the finite approximability class G 1 \ G 2 from the global approximability class G 2 . Only the last of the three methods seems useful for that, but it seems hard to find the full set of limit points for more complex families H similarly to how we did that for the linear combinations of sines (Theorems 4) and the families H σ , H (5) N (Theorems 9, 10). It would be interesting to have a general method suitable for separating G 1 \ G 2 from G 2 .

Limit points. Theorems 4 and 9 show that the nontrivial limit points of the families H (2) N and H σ are some sort of resonances, reflecting some degeneracies of the model. The set of nontrivial limit points is small and explicit in these two cases. However, in a universal formula this set is the whole C ([0 , 1]) . It would be interesting to see how general is this dichotomy, e.g. whether the set of nontrivial limit points of some formula can be a small but not explicitly describable subset of C ([0 , 1]) .

Neural networks. In Section 6 we described a general family of neural networks lying outside the class G 1 and so non-universal. These networks can have a single layer of transcendental neurons (with activations such as sin , tanh , e -x 2 ) as well as some layers with piecewise-polynomial activations. We have shown that though the resulting functions are only piecewise analytic, they still admit polynomial constraints. It remains, however, an open question whether networks with multiple transcendental layers involving the sine activation can be universal. Theorem 8 shows that such networks belong to G 1 , while Theorem 10 shows that the two-hidden-layer sine network of bounded complexity and with a single neuron in the second hidden layer belongs to G 1 \ G 2 . We conjecture that, without arcsin or other special functions that help create periodic piecewise-linear dependencies, general sin networks of fixed complexity belong to G 1 \ G 2 .

Acknowledgments

The author thanks the anonymous reviewers for several useful suggestions. Research was supported by Russian Science Foundation, grant 21-11-00373.

References

Michael Boshernitzan. Universal formulae and universal differential equations. Annals of mathematics , 124(2):273-291, 1986.

R Creighton Buck. The solutions to a smooth PDE can be dense in C (I). Technical report, WISCONSIN UNIV-MADISON MATHEMATICS RESEARCH CENTER, 1980.

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 , 2015.

George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems , 2(4):303-314, 1989.

Andrei Gabrielov and Nicolai Vorobjov. Complexity of computations with pfaffian and noetherian functions. Normal forms, bifurcations and finiteness problems in differential equations , 137:211-250, 2004.

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 315-323, 2011.

Steven M Gonek and Hugh L Montgomery. Kronecker's approximation theorem. Indagationes Mathematicae , 27(2):506-523, 2016.

Erik Christian Hansen, Yonathan Stone, and Jesse Wolfson. Alexander Ostrowski's "On Dirichlet Series and Algebraic Differential Equations". arXiv preprint arXiv:2211.02088 , 2022.

A. G. Khovanskii. Fewnomials . Vol. 88 of Translations of Mathematical Monographs. American Mathematical Society, 1991.

Andrei Kolmogorov. On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition. In Doklady Akademii Nauk , volume 114, pages 953-956. Russian Academy of Sciences, 1957.

Leopold Kronecker. Näherungsweise ganzzahlige Auflösung linearer Gleichungen. Monats. Königl. Preuss. Akad. Wiss. Berlin , pages 1179-1193, 1271-1299, 1884.

Miklós Laczkovich. The removal of π from some undecidable problems involving elementary functions. Proceedings of the American Mathematical Society , 131(7):2235-2240, 2003.

Miklós Laczkovich and Imre Z Ruzsa. Elementary and integral-elementary functions. Illinois Journal of Mathematics , 44(1):161-182, 2000.

Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks , 6(6):861-867, 1993.

Vitaly Maiorov and Allan Pinkus. Lower bounds for approximation by mlp neural networks. Neurocomputing , 25(1-3):81-91, 1999.

Eliakim Hastings Moore. Concerning transcendentally transcendental functions. Mathematische Annalen , 48(1-2):49-74, 1896.

Alexander Ostrowski. Über dirichletsche Reihen und algebraische Differentialgleichungen. Mathematische Zeitschrift , 8(3):241-298, 1920.

Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica , 8:143-195, 1999.

Amaury Pouly and Olivier Bournez. A universal ordinary differential equation. Logical Methods in Computer Science , 16, 2020.

Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941 , 2017.

Lee A Rubel. A universal differential equation. Bulletin (new series) of the american mathematical society , 4(3):345-349, 1981.

Lee A Rubel. The extended analog computer. Advances in Applied Mathematics , 14(1):39-50, 1993.

Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems , 33:7462-7473, 2020.

David R So, Wojciech Ma´ nke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Primer: Searching for efficient transformers for language modeling. arXiv preprint arXiv:2109.08668 , 2021.

Eduardo D Sontag. Shattering all sets of 'k'points in 'general position' requires (k-1)/2 parameters. Neural Computation , 9(2):337-348, 1997.

Terence Tao. A quantitative ergodic theory proof of Szemerédi's theorem. arXiv preprint math/0405251 , 2004.

Bartel Leendert Van der Waerden. Beweis einer baudetschen Vermutung. Nieuw Arch. Wiskunde , 15:212-216, 1927.

Vladimir Vapnik. The nature of statistical learning theory . Springer science & business media, 1999.

Dmitry Yarotsky. Elementary superexpressive activations. In International Conference on Machine Learning , pages 11932-11940. PMLR, 2021.

Thierry Zell. Betti numbers of semi-pfaffian sets. Journal of Pure and Applied Algebra , 139(1-3):323-338, 1999.

Contents

A Proof of Theorem 4

Sufficiency (each function (6) is a limit point of H (2) N ). First observe that each resonant component x m sin( ωx + h ) is a uniform limit of m +1 sine waves linearly combined to match the discretized derivative d m dω m sin( ωx + h -πm/ 2) :

Note also that any finite linear combination ∑ M -1 m =0 c m sin( ωx + h m ) can be written as a single sine wave c sin( ωx + h ) with suitable amplitude c and phase h . It follows that we can obtain any linear combination ∑ M -1 m =0 c m x m sin( ωx + h m ) as a uniform ∆ ω → 0 limit of linear combinations

with suitable values of c m, ∆ ω and h m, ∆ ω . This shows in turn that the second component of expression (6),

can be obtained by limits of linear combinations of ∑ K k =1 M k sine waves with suitable phases and frequencies.

It remains to show that the first component of expression (6), ∑ 2 M 0 -1 m =0 c 0 m x m , can be obtained as a limit of linear combinations of M 0 sine waves. Observe that for any constant x 0

Since 2 M 0 -1 is odd, sin 2 M 0 -1 (∆ ωx -∆ ωx 0 ) is a linear combination of M 0 sine waves with frequencies (2 s -1)∆ ω, s = 1 , . . . , M 0 (and some phases). By a previously given argument, this implies that any linear combination of monomials ( x -x 0 ) 2 M 0 -1 can also be obtained as a limit of some linear combinations of M 0 sine waves with frequencies (2 s -1)∆ ω, s = 1 , . . . , M 0 . But the monomials ( x -x 0 ) 2 M 0 -1 linearly span the space of polynomials of degree ≤ 2 M 0 -1 . It follows that any such polynomial can be obtained as a limit of linear combinations of M 0 sine waves, as claimed.

Necessity (each limit point of H (2) N has the from (6) ). We give two alternative proofs.

Proof 1 ('clustering-based'). Suppose that a function f ∗ is the uniform limit of functions f r ∈ H (2) N . We identify the different frequency clusters associated with the sequence f r . Let 0 ≤ ω 1 ,r ≤ . . . ≤ ω N,r be the N sorted frequencies appearing in the sine wave expansion of f r ; we assume here without loss of generality that all the frequencies are nonnegative. For each n = 1 , . . . , N -1 one of the following three options holds:

either ω n +1 ,r -ω n,r r →∞ -→ ∞ ;

or sup n ( ω n +1 ,r -ω n,r ) < ∞ ;

or none of the above holds.

For each n , by taking a subsequence of f r , we can exclude option 3. By taking subsequences N times, we can then assume without loss of generality that for each n either option 1 or option 2 holds. It follows that we can partition the set { 1 , . . . , N } into contiguous subsets I 1 = { 1 , 2 , . . . , i 1 } , I 2 = { i 1 +1 , . . . , i 2 } , . . . , I s = { i s -1 +1 , . . . , N } such that for some constant Ω

The sets I t thus represent the largest clusters of frequencies that remain at a bounded distance from each other in the limit r →∞ .

Clearly, there is at most one cluster whose frequencies remain bounded in the limit r →∞ , namely I t 0 with t 0 = 1 . It will be convenient to assume that I t 0 has bounded frequencies but possibly is empty.

For any M = 1 , 2 , . . . and Ω > 0 let us denote by H (2) M, Ω the set of functions f : [0 , 1] → C of the form

where c n ∈ C are arbitrary amplitudes and ω n ∈ [ -Ω , Ω] .

Lemma 1. For any f ∈ H (2) M, Ω and n = 1 , 2 , . . .

We remark that the constant in Eq. (20) is likely suboptimal, but it must certainly grow with M , since, as the proof of the sufficiency part of Theorem 4 shows, arbitrarily narrow frequency bands are sufficient to generate any degreeM polynomial.

Proof. The function f satisfies the differential equation Df = 0 , where

with some constants b n . Consider the vector-valued function f ( x ) = ( f ( x ) , f ′ ( x ) , . . . , f ( M -1) ( x )) T . It satisfies the differential equation

with the matrix A = ( a nm ) M -1 n,m =0 , where

The solution of this equation is

If ∥ · ∥ is any norm on the C M , then

where ∥ A ∥ is the respective matrix norm. Let ∥ · ∥ ∞ be the maximum ( l ∞ ) norm, then

Now let x 0 = argmax x ∈ [0 , 1] ∥ f ( x ) ∥ ∞ and let n 0 = argmax n =0 ,...,M -1 | f ( n ) ( x 0 ) | so that ∥ f ( x 0 ) ∥ ∞ = | f ( n 0 ) ( x 0 ) | . If n 0 = 0 , then

Suppose that n 0 > 0 . Observe that if | x -x 0 | ≤ ln 2 2 (1 + Ω) -M , then

Consider the complex unit direction z 0 = f ( n 0 ) ( x 0 ) | f ( n 0 ) ( x 0 ) | and denote l = ln 2 2 (1 + Ω) -M . For all x such that | x -x 0 | ≤ l we have

It follows that we can find in [0 , 1] a subinterval of length l/ 4 on which either

or

Making another iteration, we can find in [0 , 1] a subinterval of length l/ 4 2 on which either

or

Continuing in this way, for any n = n 0 , n 0 -1 , . . . , 0 we find in [0 , 1] a subinterval of length l/ 4 n 0 -n on which either or

Taking n = 0 , this implies

and so, by definition of x 0 and n 0 , for any n = 0 , . . . , M -1

This proves the claim for n ≤ M -1 . If n ≥ M, then we use again Eq. (21) and the bound ∑ m | b m | ≤ (1 + Ω) M .

Lemma 2. The L ∞ and L 2 norms are equivalent on the set H (2) M, Ω : for any f ∈ H (2) M, Ω

Proof. Only the second inequality is nontrivial. By Lemma 1,

It x 0 = arg max x ∈ [0 , 1] | f ( x ) | , then for | x -x 0 | ≤ 1 /c we have | f ( x ) | ≥ (1 -| x -x 0 | c ) | f ( x 0 ) | and so

implying the desired result.

Consider now the expansion of f r over different frequency clusters:

where f t,r denotes the I t -component of f r . We first show that we can discard the components associated with unbounded frequencies.

Lemma 3. If f r converge to f ∗ uniformly on [0 , 1] , then f t 0 ,r also converge to f ∗ uniformly on [0 , 1] .

Proof. Step 1. Let us first prove the statement in the case f ∗ = 0 . Observe that each f t,r can be represented as

where g t,r, 1 = g t,r, -1 and g t,r,q ∈ H (2) 2 N, Ω with some Ω independent of t and r . We use this representation for all t except t = t 0 (since f t 0 ,r ∈ H (2) 2 N, Ω already):

̸

Then,

̸

Observe that a product of g, h ∈ H (2) M, Ω belongs to H (2) M 2 , 2Ω . It follows that each cross-term can be written as

̸

where ∆ ω r r →∞ -→ ∞ and g r ∈ H (2) (2 N ) 2 , 2Ω ; the function g r being a product of two functions out of the collection F = { f t 0 ,r } ∪ { ( g t,r,q ) t = t 0 ,q = ± 1 } . We have

where we used the inequality ∥ g ′ r ∥ ∞ ≤ c ∥ g r ∥ ∞ with a constants c = c ( N, Ω) as established in Lemma 1. Observe that if g r is a product of two functions from the collection F , say g, h, then we have

with another suitable constant C = C ( N, Ω) , by Lemma 2. It follows that the cross-terms in Eq. (50) are dominated by the explicitly written terms:

̸

Therefore, if ∥ f r ∥ ∞ = o (1) as r →∞ , then we also have

and similarly all

Step 2. Now we prove the result for a general f ∗ . If ∥ f r -f ∗ ∥ 2 → 0 , then the sequence f r is Cauchy:

̸

We write

̸

Assuming that r 2 = r 2 ( r 1 ) and grows sufficiently fast, the frequencies ω t,r 1 , ω t,r 2 and 0 (corresponding to the first term) will form disjoint clusters. We can then use the argument from Step 1 and conclude that all ∥ g t,r,q ∥ ∞ = o (1) , implying the desired result.

The last lemma shows that any uniform limit of functions f r ∈ H (2) N is also a uniform limit of band-limited functions f r ∈ H (2) 2 M, Ω for some Ω < ∞ . Any function f r ∈ H (2) N is a solution of the differential equation

where

If the frequencies ω nr are bounded, by compactness we can find their limit points ω n . We argue now that the limiting function f ∗ is a solution of the corresponding limiting equation Df = 0 with

Indeed, such solutions have the form where T is the left shift:

(cf. Eq. (24)), while the functions f r can be written as

with respective matrices A r . As r → ∞ , the matrices A r converge (by taking a subsequence, if necessary) to A , and, by Lemma 1, f r ( x 0 ) → f ( x 0 ) . It follows that the function f ∗ indeed is representable in the form (65), i.e. is a solution of equation Df = 0 . This immediately implies the claim of the theorem (with isolated zero limiting frequencies ω n = 0 ).

Proof 2 ('discretization-based'). Our second proof essentially relies on discretizations of the function f ∗ and will consist of two steps. In the first step we introduce the discretizations, show that they satisfy discrete difference equations, and describe general solutions of these equations. In the second step we analyze how discretizations at different length scales agree with each other and show that the continuity of f ∗ enforces representation (6) in the zero spacing limit.

Step 1: Discretization. Suppose that the function f ∗ is the uniform limit of functions f r ∈ H (2) N , r = 1 , 2 , . . . Fix some ∆ x > 0 and consider the sequences u ∗ = ( u ∗ s ) ⌊ 1 / ∆ x ⌋ s =0 and u ( r ) = ( u ( r ) s ) ⌊ 1 / ∆ x ⌋ s =0 obtained by restricting these functions to the points s ∆ x :

Since f r ∈ H (2) N , each sequence u ( r ) can be written as a linear combination

with some complex amplitudes b nr ∈ C and frequencies ω nr ∈ R (the horizontal bar denotes complex conjugation).

We want to perform now the limit r →∞ and derive a similar representation for u ∗ . To this end observe that, up to the boundary values, a sequence of the form u s = be iωs ∆ x satisfies the finite difference equation

Since each sequence u ( r ) has the form of linear combination (68) and the operators T -e iω ∆ x commute, it follows that u ( r ) satisfies the finite difference equation

This equation holds up to the 2 N boundary components u ( r ) s that are moved by the shifts T contained in D ( r ) outside the domain [0 , 1] of the functions f r .

We would like to perform the limit r →∞ in this difference equation, but we cannot directly perform it in the exponents iω nr ∆ x, since the frequencies ω nr may be unbounded and not have any limit points. However, the values e iω nr ∆ x belong to the unit circle in C , which is a compact set. Therefore, we can choose a subsequence r t such that e iω nr t ∆ x → z ∗ n as t →∞ , with some z ∗ n ∈ C , | z ∗ n | = 1 .

Performing this limit in the difference equation, we then see that the limiting sequence u ∗ satisfies a difference equation

A general solution of this equation is again a linear combination of sine waves, but with possible resonances due to repeated values z ∗ n . Suppose that the numbers z ∗ n contain K distinct nonreal values z 1 , . . . , z K with multiplicities M k ≥ 1 . Additionally, let M ± ≥ 0 be the multiplicities of the real values z ± = ± 1 among the numbers z ∗ n . We can then write the general real solution of equation (72) as

with M -+ M + + ∑ K k =1 M k = N. Equivalently, we can write

where we have introduced the vectors v ± ,m , v km , v km ∈ C ⌊ 1 / ∆ x ⌋ +1 with components

The coefficients b km and b ± ,m in expansions (73), (74) are determined uniquely as long as ∆ x is sufficiently small:

Lemma 4. Suppose that ∆ x ≤ 1 / (2 N -1) so that dim u ∗ = ⌊ 1 / ∆ x ⌋ +1 ≥ 2 N. Then the 2 N vectors v ± ,m , v km and v km are linearly independent and the coefficients b ± ,m , b km are uniquely determined.

Proof. For 1 ≤ q ≤ M k , consider the finite difference operators D k,q obtained by removing the factor ( T -z k ) q from the operator D that appears in Eq. (72):

When applied to u ∗ , this operator kills all the components of the expansions (73), (74) except those associated with z k and having degree m ≥ M k -q . In particular,

̸

with the explicit coefficient

We can then find the coefficient b k,M k -1 from Eq. (77) if there is at least one s for which it holds (recall that it is not applicable to a few boundary values of s ). The highest power of the shift T appearing in D k, 1 is T 2 N -1 , so such s exists as long as dim u ∗ ≥ 2 N .

Having found b k,M k -1 , we can subtract the corresponding component b k,M k -1 v k,M k -1 from u ∗ and apply to the remainder the operator D k, 2 , which will kill all terms except b k,M k -2 v k,M k -2 . This allows to determine the next coefficient b k,M k -2 . Continuing this process and extending it to other z k and z ± , we uniquely recover all the coefficients b ± ,m , b km . The linear independence of the vectors v ± ,m , v km and v km follows from this uniqueness.

Step 2: The limit ∆ x → 0 . We consider now discretizations with variable spacing ∆ x . Our goal is to show that expansions (73) lead to the desired representation (6) in the limit ∆ x → 0 . In the notation, we will occasionally add ∆ x to the subscripts to reflect the dependence of various quantities on the discretization spacing.

It is convenient to rewrite expansion (73) by rescaling the coefficients via b km = c km (∆ x ) m and introducing frequencies ω k and ω -such that z k = e iω k ∆ x and -1 = e iω -∆ x , so as to make the expansion less ∆ x -dependent:

The frequencies ω k , ω -are determined from the values z k, ∆ x only up to integer multiples of 2 π/ ∆ x :

Observe that expansion (79)-(81) is invariant under downscaling, in the following sense. Since s appears in it only through the combination s ∆ x , if this expansion holds for some ∆ x , then it also holds for any integer multiple ∆ x ′ = l ∆ x , with the same coefficients c ± ,m , c km and frequencies ω -, ω k .

To discuss this effect in more detail, we argue now that we can establish a correspondence between different values z k across different scales. In general, downscaling is accompanied by aliasing: if the difference equation (72) holds with the values z ∗ n, ∆ x on the length scale ∆ x, then on the integer multiple length scale l ∆ x it holds with the values

In particular, the values z k, ∆ x that are distinct on the length scale ∆ x may collide when mapped to the length scale l ∆ x . However, in our setting we can bound the number of collisions thanks to the finiteness of the set of frequencies.

Specifically, consider length scales of the form 2 -t ∆ x, with t = 0 , 1 , . . . The total number of collisions over all t is bounded, since there are at most K = N distinct values z k . It follows that at sufficiently large t there are no collisions, so that the number K = K t of distinct values z k, 2 -t ∆ x is the same for all t , and we can enumerate these values so that for t 2 > t 1

Moreover, observe that for sufficiently large t there will be no ω --component (corresponding to z -= -1 ) in the (79)-(81). Indeed, since the square roots of -1 are not real, the presence of the ω --component for some t would mean that K t +1 > K t . But such an increase is possible only for a finite number of t 's.

Similarly, the ω + -component of the expansion (79)-(81) must have the form

with the same c + ,m for all sufficiently large t , since any change of this component with t is accompanied by an increase in K t . The function f ∗ , 0 uniquely extends to a continuous function on the whole interval [0 , 1] ,

which agrees with the first component in the expansion (6).

This discussion shows that without loss of generality (by subtracting f ∗ , 0 from f ∗ if necessary), we may assume that expansion (79)-(81) contains only the last component, standing in line (81).

Assuming that transition from the length scale ∆ x to ∆ x ′ = l ∆ x occurs without collisions in z k , the downscaling invariance condition can be written as

where the sets W k, ∆ x ′ are defined by Eq. (82). By Lemma 4, the coefficients c km, ∆ x are uniquely determined as soon as ∆ x and ∆ x ′ are small enough.

Considering again the length scales of the form 2 -t ∆ x , we get a nested sequence

All the inclusions here are strict. The intersection

is either empty or consists of a single frequency { ω k } . In the latter case we can again (as previously with f ∗ , 0 ) identify the respective component of the function f ∗ :

Our goal is to prove the that the intersections W k, lim are indeed nonempty for all k . We will obtain this as a consequence of the continuity of the function f ∗ , which in turn follows from the fact that f ∗ is assumed to be a uniform limit of continuous functions f r .

̸

To this end, observe that when upscaling from 2 -t ∆ x to 2 -t -1 ∆ x, the value z k, 2 -t -1 ∆ x is one of the two square roots of z k, 2 -t ∆ x . The case W k, lim = ∅ corresponds to the case when for all sufficiently large t we take the square root with positive real part, so that

In contrast, in the case W k, lim = ∅ , for any t 0 there is t > t 0 such that z k, 2 -t -1 ∆ x has a negative real part, so that

Now consider the vector u ∗ obtained by sampling the function f ∗ at the points s ∆ x as in Eq. (67). Consider the sequence u ( ∗ ,t ) of vectors of the same dimension, obtained by sampling the function f ∗ with additional shifts 2 -t ∆ x :

Since f ∗ is continuous,

Recall that the vector u ∗ admits an expansion (74) over the vectors v ± ,m , v km , v km . As mentioned earlier, we can assume without loss of generality that there are no v ± ,m terms:

Observe now that an analogous expansion can also be written for the shifted vectors:

Moreover, the vectors v ( t ) km are either exact or approximate multiples of the vectors v km :

̸

By Lemma 4, the vectors v km and v km are linearly independent for sufficiently small ∆ x , so convergence (96) requires all those coefficients z k, 2 -t ∆ x for which b km = 0 at least for one m to converge to 1 (as in Eq. (93)). This proves that all W k, lim are non-empty for such k . If for some k we have b km = 0 for all m , then this k -component is effectively missing and can be ignored. This completes the proof of the theorem.

B Proof of Theorem 5

Our proof will consist of three steps. In the first step we show that the polynomial constraint claimed in the theorem holds for functions of the form e P ( x ) . In the second step we show that if polynomial constraints hold for the arguments of an algebraic function, then its values also admit a polynomial constraint. In the final third step we combine these results to complete the proof of the theorem.

We remark that our arguments are close to the arguments of the theory of algebraic-transcendent functions [16, 17], but we work in a discretized setting of uniformly sampled data without considering any derivatives.

Step 1: Polynomial constraints for exponentials of polynomials. Consider the function

where P is a (possibly complex-valued) polynomial of degree ≤ d . Consider operators D s h of forward discrete derivatives of order s with step h :

We have D d +1 h P = 0 for any h ∈ C and any polynomial P of degree ≤ d :

Separating the positive and negative coefficients, we can write this, equivalently, as

Exponentiating, it follows for g ( x ) given by Eq. (100) that

We can view this as a polynomial constraint of the form

with some fixed polynomial R of d +2 variables.

Step 2: Extension of polynomial constraints to algebraic functions. Lemma 5.

Suppose that Q ( y 0 , y 1 , . . . , y N ) is an algebraic function, and g 0 ( x ) , . . . , g N ( x ) are functions satisfying polynomial relations

Let g ( x ) = Q ( g 0 ( x ) , . . . , g N ( x )) and m ≥ s ( N + 1) . Then there exists a nontrivial polynomial R of m +1 variables such that

Moreover, suppose that the function Q is not fixed, and is only constrained by its degree deg Q ≤ q and the degrees deg ϕ k ≤ r of the polynomials appearing in its definition. Let m ≥ ( q +1) ( N + r +1 N +1 ) + s ( N +1) . Then there exists a nontrivial polynomial R of m +1 variables such that relation (107) holds for all such Q .

Proof. 1. Recall the defining relation (7) of the algebraic function Q :

We can use this relation to express Q q in terms of lower powers Q 0 , . . . , Q q -1 :

By iterating this relation, for any power t ≥ q we can express Q t in terms of the powers Q 0 , . . . , Q q -1 :

where ϕ t,k are suitable polynomials of degree not greater than t max k =0 ,...,q deg ϕ k . Substituting

where for brevity we have denoted g ( x ) = ( g 0 ( x ) , . . . , g N ( x )) .

Formula (111) remains valid if we replace x by x + lh with any l . Consider various monomials in the variables Q ( g ( x + lh )) for l = 0 , . . . , m :

where t = ( t 0 , . . . , t m ) , k = ( k 0 , . . . , k m ) , ˜ Φ t , k are some polynomials of degree not greater than ( m +1)(max l t l ) max k deg ϕ k , and

Our goal is to prove that for sufficiently large m , the monomials F t are linearly dependent (with some coefficients independent of x and h ), which is exactly equivalent to the existence of a polynomial R satisfying relation (107). This is done by showing a linear dependence of the expressions (113) with different t .

Specifically, observe that, regardless of t , expressions (113) contain only finitely many (namely, q m +1 ) different components Q k . Since there are infinitely many monomials F t , it follows already from this observation that there exist nontrivial vanishing linear combinations of the monomials F t if the coefficients in these linear combinations are allowed to belong to the field of all rational functions of g ( x ) , . . . , g ( x + mh ) . However, we need to show that nontrivial vanishing linear combinations can be found even with constant coefficients.

To this end, we will further reduce the combinatorial complexity of expressions (113) by removing high powers of the variables g n ( x + lh ) at the expense of introducing new rational functions of the variables g n ( x + l ′ h ) with l ′ > l . Recall that each g n is subject to polynomial relation (106). We can use these relations to transform polynomial functions of g ( x ) into rational functions of g ( x ) , g ( x + h ) , . . . , g ( x + sh ) that contain only bounded powers of g ( x ) . Specifically, isolate the variables g n ( x ) in the polynomials R n by writing equations (106) in the form

we get

with suitable polynomials ψ n,j of the remaining variables g n ( x + h ) , . . . , g n ( x + sh ) . The value d n is the degree of g n ( x ) in this representation. Then, similarly to how we did this for Q, we can express

By iterating this relation, for any t ≥ d n we get

with some polynomials ψ n,j,t of degree not greater than t max j deg ψ n,j . This relation remains valid if we replace x by x + lh with any l :

We can now transform expression (113) by first replacing there all powers t ≥ d n of g n ( x ) with powers t ≤ d n and rational functions of g n ( x + h ) , . . . , g n ( x + sh ) , then replacing powers t ≥ d n of g n ( x + h ) with powers t ≤ d n and rational functions of g n ( x +2 h ) , . . . , g n ( x +( s +1) h ) , and so on. We continue this process while the involved variables g ( x + lh ) obey the condition l ≤ m . As a result, we obtain a rational function of the variables g ( x ) , . . . , g ( x + mh ) that contains only powers of the variables g n ( x ) , . . . , g n ( x +( m -s ) h ) not exceeding d n -1 .

To establish the desired linear dependence, we want the rational functions appearing in these transformations of the monomials F t to have the same denominator for all t . It is convenient to ensure this in the following way. Choose some large T and assume that all the powers t k in the monomials F t vary between 0 and T , so that in total we have ( T +1) m +1 monomials. In expression (113), instead of the denominator ∏ m l =0 ϕ t q ( g ( x + lh )) use the common denominator

This changes the polynomials ˜ Φ t , k :

the new polynomials Φ t , k now have degrees not exceeding ( m +1) T max k deg ϕ k . Now we perform the iterative substitutions of the powers g t n ( x + km ) in the polynomials Φ t , k as described above, while keeping the same denominator for all t ; this requires multiplying the common denominator by appropriate powers of ψ n,d n ( g n ( x +( l +1) h ) , . . . , g n ( x +( l + s ) h ))) . As a result of this process we obtain expressions

Here J = ( J nl ) is a matrix with elements 0 ≤ J nl ≤ d n -1 , so summation over J is finite. The polynomials Ψ t , k ,J only depend on the remaining variables g ( x +( m -s +1) h ) , . . . , g ( x + mh ) .

Importantly, by taking into account previously observed bounds on the degrees of polynomials appearing in Eqs. (119) and (121), we observe that deg Ψ t , k ,J ≤ CT with some constant C depending on s, m, N and the degrees of the polynomials ϕ k and ψ n,j appearing in the assumed expansions (108) and (115).

Now the desired linear dependence of the monomials F t results, with a suitable choice of m and T , by comparing the number ( T +1) m +1 of different monomials F t with the number of different monomials that may appear in the expansions of the polynomials Ψ t , k ,J . Since these polynomials depend on the s ( N + 1) variables g ( x + ( m -s + 1) h ) , . . . , g ( x + mh ) and have degrees not exceeding CT, the number of different monomials in them does not exceed C 1 T s ( N +1) with some constant C 1 depending on the same parameters as C. Taking into account that the summations over k and J in expansion (122) involve, respectively, q m +1 and ∏ N n =0 d m -s +1 n terms, we see that the ( T + 1) m +1 monomials F t belong to a C 1 q m +1 ( ∏ N n =0 d m -s +1 n ) T s ( N +1) -dimensional linear functional space. Accordingly, if s ( N +1) < m +1 , then some of the monomials F t will be linearly dependent if T is large enough. This completes the proof of part 1 of the lemma.

2. The argument above produces nontrivial linear combinations of monomials F t with coefficients generally dependent on Q . We show now that, assuming appropriate degree bounds for Q and choosing a larger m , one can find nontrivial linear combinations with Q -independent coefficients. To this end, note that the main obstacle to choosing Q -independent coefficients is that the polynomials Ψ t , k ,J in expansion (122) are Q -dependent. To overcome this obstacle, we express these polynomials both in the variables g ( x +( m -s +1) h ) , . . . , g ( x + mh ) and the coefficients b of the polynomials ϕ k :

Here, the vector b includes all the coefficients of all q +1 polynomials ϕ k appearing in the defining identity (108). By retracing the steps to construct the polynomials Ψ t , k ,J , we see that their dependence on the coefficients b is indeed polynomial. Moreover, deg ˜ Ψ t , k ,J is again bounded by CT , where C is some constant possibly dependent on N,s,m,q, maxdeg ϕ k , max j deg ψ n,j , but not on T . Since we assume deg ϕ k ≤ r, we have dim b ≤ ( q +1) ( N + r +1 N +1 ) . The total number of variables in ˜ Ψ t , k ,J is thus not greater than

We repeat now the dimensionality-based linear dependence argument. The total dimensionality of polynomials ˜ Ψ t , k ,J considered at all k , J is O ( T p ) . On the other hand, the number of monomials F t is ( T +1) m +1 . If we set m ≥ p , then for sufficiently large T we can find a nontrivial vanishing linear combination of F t with coefficients independent of Q .

Step 3: Proof of Theorem 5. We apply now Lemma 5 to our setting in which g 0 ( x ) = x and g k ( x ) = e P k ( x ) , k = 1 , . . . , N . We know from Step 1 that for k = 1 , . . . , N conditions (106) hold with s = deg P k +1 . For g 0 , condition (106) holds with s = 2 :

Since we assumed that deg P k ≤ d , we can set s = max( d, 1) + 1 as a common value with which condition (106) holds for all n = 0 , . . . , N . The conclusion of Theorem 5 then follows from statement 2 of Lemma 5.

C Proof of Theorem 6

Since the network has not more than 2 N hidden neurons and each of our activation functions has at most two branches, the output g ( x ) of the network at any x belongs to one of at most 2 N branches represented by analytic expressions. Consider each of these branches as defined for all x ∈ [0 , 1] and denote the family of all branches resulting from all g ∈ H (4) N by ˜ H (4) N . Observe that ˜ H (4) N ⊂ H (3) N ′ ,q,r,d if N ′ , q, r, d are large enough. Indeed, the input of any algebraic-exponential neuron is a polynomial of degree not greater than d = 2 N (if the squared ReLU is present; otherwise, if only the step function and ReLU/leaky ReLU are present, then one can take d = 1 ). All the considered algebraic-exponential activations are rational functions of x, e x or e ± ix , and the subsequent neurons are polynomial, so any branch of the full network can be written in the form (8) with a rational Q and 2 N as the number of its exponential arguments. Rational Q 's have algebraic degree q = 1 . The maximum degree of the

polynomials ϕ k appearing in the defining relation (7) does not exceed r = 2 N (again, if the squared ReLU is present; otherwise one can take r = 1 ). Summarizing,

Theorem 5 implies now that there exists some ˜ m and a nonzero polynomial ˜ R of ˜ m +1 variables such that for any ˜ g ∈ ˜ H (4) N and any size-( ˜ m +1) arithmetic progression ˜ a, ˜ a + ˜ h, . . . , a + ˜ m ˜ h ∈ [0 , 1]

We need to obtain a similar relation (11), but for g ∈ H (4) N . Each value g ( a + kh ) in Eq. (11) belongs to one of the 2 N branches ˜ g ∈ ˜ H (4) N , which we associate with p = 2 N colors. By applying van der Waerden's theorem with p = 2 N and s = ˜ m +1 , we see that if we set m = N vdW ( ˜ m +1 , 2 N ) -1 , then the length-( m +1) progression a, a + h, . . . , a + mh appearing in Eq. (11) contains at least one length-( ˜ m +1) sub-progression ˜ a, ˜ a + ˜ h, . . . , ˜ a + ˜ m ˜ h on which the values of g belong to the same branch. We can then satisfy condition (11) by defining the polynomial R by

where s runs over all length-( ˜ m +1) progressions from the set { 0 , . . . , m } .

D Proof of Theorem 8

̸

̸

The 'only if' part follows from Theorem 5. The 'if' part is a generalization of Lemma 1 from [29]. Suppose that σ is not a polynomial; we want to prove that H σ ∈ G 1 . By absorbing h in σ , we can assume that σ (0) = 0 . Dropping then h , we specialize to f ( x ) = c sin( ωσ ( bx )) . Consider an arbitrary finite subset X = { x 1 , . . . , x N } ⊂ [0 , 1] . By Theorem 2, it is sufficient to show that there exsts b ∈ [ -1 , 1] such that the values σ ( bx 1 ) , . . . σ ( bx N ) are rationally independent. Suppose that this is not the case. For fixed coefficients λ = ( λ 1 , . . . , λ N ) , the function σ λ ( b ) = ∑ N n =1 λ n σ ( bx n ) is real analytic for b ∈ [ -1 , 1] . Since there are only countably many λ ∈ Q N , there is some λ such that σ λ vanishes at uncountably many b ∈ [ -1 , 1] . Then, by analyticity, σ λ ≡ 0 on [ -1 , 1] . Expanding this σ λ into the Taylor series at b = 0 , we get the identity ∑ N n =1 λ n x m n = 0 for each m such that d m σ dw m (0) = 0 . Suppose that x k have the natural order 0 ≤ x 1 < . . . < x N . If there are infinitely many m for which d m σ dw m (0) = 0 , then, by letting m →∞ , we first show that λ N = 0 , then λ N -1 = 0 , etc. If x 1 > 0 , then we exhaust all λ n in this way; otherwise we exhaust all except λ 1 . However, λ 1 = 0 follows then from σ (0) = 0 . Thus, either λ = 0 or σ is a polynomial, which contradicts our assumption.

E Proof of Theorem 9

Suppose that the function sequence f n ( x ) = c n sin( ω n σ ( b n x ) + h n ) converges to a nontrivial limit point f ∗ . By a compactness argument, this requires at least one of the parameters c n , ω n to be unbounded. We expect unboundedness of one parameter to be compensated by another parameter converging to 0 . Accordingly, we consider three possibilities: ω n → 0 , ω n → ω = 0 and ω n →∞ . One of these possibilities can always by ensured by choosing a subsequence of f n , if necessary (in the sequel we will without further mention repeatedly use such reductions based on compactness and/or transition to subsequences).

̸

̸

Case 1: ω n → 0 . Since σ ( b n x ) is uniformly bounded by our assumption and sin is periodic, we can assume that ω n σ ( b n x ) + h n n →∞ -→ a with some constant a ∈ [0 , 2 π ) , for all x ∈ [0 , 1] . If a = 0 , π , then the limit f ∗ can only be trivial, a constant. Suppose that a = 0 (the case a = π leads to analogous results). In this case f n ( x ) = c n ( ω n σ ( b n x ) + h n )(1 + o (1)) , so f ∗ ( x ) = lim n →∞ c n ( ω n σ ( b n x ) + h n ) . There are two possibilities: b n → b = 0 and b n → 0 . In the first case we obtain nontrivial limits

̸

̸

̸

In the second case we can Taylor expand

̸

A nontrivial f ∗ requires c n b r n to have a finite nonzero limit. In this case, by adjusting h n we obtain nontrivial limits

̸

Case 2: ω n → ω = 0 . Consider again the two possibilities: b n → b = 0 and b n → 0 . In the first case we do not get any nontrivial limits. In the second case we again have Taylor expansion (130). Arguing as in Case 1, the only nontrivial limits that we can produce are of the form (131).

̸

Case 3: ω n →∞ . In this case we must have b n → 0 , since otherwise sin( ω n σ ( b n x ) + h n ) will have an unbounded number of alternating values ± 1 , so we must have c n → 0 and accordingly the only possible limit is f ∗ = 0 . It follows that we again have expansion (130), and

To avoid triviality due to an unbounded number of alternating values ± 1 of sin , we must have ω n b r n = O (1) . We get new nontrivial limits of f n when ω n b r n has a nonzero limit:

F Proof of Theorem 10

We need to prove that H (5) N ∈ G 1 and H (5) N / ∈ G 2 .

̸

Proof of H (5) N ∈ G 1 . For N ≥ 2 , this property can be concluded from Theorem 8, since we can just set σ ( x ) = sin( x ) and h = h sin(0 · x + π/ 2) in this theorem. In the case N = 1 , observe that in the proof of Theorem 8 we only use h to redefine σ so that σ (0) = 0 ; otherwise we can set h = 0 . Then, this proof becomes applicable to the family f ( x ) = c sin( ωσ ( bx )) with σ ( x ) = sin( x + h ) and any phase h = πk, k ∈ Z ; this is a subfamily of H (5) N =1 .

Proof of H (5) N / ∈ G 2 . We will show that we cannot obtain a general continuous function as a limit of functions from H (5) N , because any such limit has a particular restrictive form.

We write f n ( x ) = c n sin( g n ( x )) + h n and consider several cases depending on the behavior of c n .

Case 1: c n → 0 . In this case the limiting function is constant: f ∗ ( x ) = h .

̸

Case 2: c n → c = 0 . In this case h n must be bounded and hence can be assumed to converge to some h (by possibly choosing a subsequence). Denote ϕ n ( y ) = c n sin( y ) + h n ; then ϕ n ( y ) converges uniformly on R to ϕ ∗ ( y ) = c sin( y ) + h . Clearly, the functions ˜ f n ( x ) = ϕ ∗ ( g n ) also uniformly converge to f ∗ .

Consider now two alternatives: either the family { g n } is uniformly equicontinuous, or not. In the first case, fix some point x 0 ∈ [0 , 1] , let r n = ⌊ g n ( x 0 ) 2 π ⌋ and consider the shifted functions ˜ g n ( x ) = g n ( x ) -2 πr n . We have ϕ ∗ ( g n ( x )) ≡ ϕ ∗ ( ˜ g n ( x )) , the family { ˜ g n } is also equicontinuous, and ˜ g n ( x 0 ) ∈ [0 , 2 π ] for all n , so that the family { ˜ g n } is additionally uniformly bounded on [0 , 1] . Then, by the Arzelà-Ascoli theorem, the family { ˜ g n } contains a uniformly convergent subsequence. It follows that f = ϕ ∗ ( g ∗ ) , where g ∗ is a uniform limit of some functions from H (2) N +1 . Theorem 4 describes all such limits.

̸

In the case of the second alternative, there exist ϵ > 0 and a subsequence g n s such that for some x s , x ′ s ∈ [0 , 1] we have | x s -x ′ s |→ 0 while | g n s ( x s ) -g n s ( x ′ s ) | ≥ ϵ for all s. Let r s = ⌊ g ns ( x s ) 2 π ⌋ and ˜ g s ( x ) = g n s ( x ) -2 πr s . Then ˜ g s ( x s ) ∈ [0 , 2 π ] and by choosing a subsequence we can assume that g s ( x s ) converges to some y ∗ ∈ [0 , 2 π ] . Using | ˜ g s ( x s ) -˜ g s ( x ′ s ) | ≥ ϵ and the continuity of ˜ g s , we can then assume that ˜ g s ( x ′ s ) converges to some y ′ ∗ = y ∗ . Also, we can assume that x s and x ′ s converge to

̸

some x ∗ ∈ [0 , 1] . Then, by the uniform convergence of ˜ f n , sup x ∈ [ x s ,x ′ s ] | ˜ f n s ( x ) -˜ f ∗ ( x ∗ ) |→ 0 as s →∞ . On the other hand, ˜ f n s ( x ) = ϕ ∗ ( g n s ( x )) = ϕ ∗ ( ˜ g s ( x )) so that, by the continuity of ˜ g s ,

̸

because ϕ ∗ in nonconstant on any interval [ y ∗ , y ′ ∗ ] . This contradiction shows that the second alternative is impossible.

Case 3: c n → ∞ . In this case the range of g n must shrink, since otherwise the range of f n would be unbounded. Taking a subsequence, we can assume that for some y 0 ∈ [0 , 2 π ] we have g n ( x ) = y 0 +2 πm n + o (1) uniformly on [0 , 1] . We can then Taylor expand

̸

If cos( y 0 ) = 0 , then the respective term dominates the higher order terms and must be bounded as n increases. We can then discard the higher order terms, and we see that f ∗ is a uniform limit of the functions c n g n -a n with some a n . These limits are among the limits of the family H (2) N +1 , again covered by Theorem 4.

If cos( y 0 ) = 0 , then the respective term vanishes and the next, second order term will dominate the higher order terms. In this case f ∗ is a limit of functions of the form ψ n = h n + a n ( g n -b n ) 2 . Substituting g n ∈ H (2) N and expanding products of sines, we see that ψ n ∈ H (2) 1+2( N +1) 2 , so, again, the limits of ψ n are restricted by Theorem 4.

Peer reviews:
comments: SUMMARY:
This paper studies function approximation in the spirit of the Kolmogorov-Arnold representation theorem.
A definition of expressivity classes is made which distinguishes pointwise and uniform approximation.
Function families with the form of neural networks are proposed and approximation results proven.

STRENGTHS:
Definitions are systematic and arguments are clear.

WEAKNESSES:
As discussed at the bottom of p. 2, these constructions and arguments rely on infinite precision arithmetic
and are badly behaved when restricted to finite precision.  This is as expected for finite dimensional families
which live in the classes G_n of section 2.  The results are of mathematical interest but I do not see the interest 
for the machine learning community.

QUESTIONS:
How could this line of work bear on or influence approximation theory which (like ref. [4]) is relevant for practical computation?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper advances the basic understanding of the expressive power of fixed-complexity function families on [0, 1]. It defines three classes of function families: $\\mathcal{G}_0$, whose members are infinite VC-dimension families, $\\mathcal{G}_1$, whose members can approximately fit any finite number of points, and $\\mathcal{G}_2$, whose members can achieve uniform approximation over the domain [0, 1]. Previous work has identified certain function families that belong to $\\mathcal{G}_2$. This paper focuses on the difference sets, namely $\\mathcal{G}_0 \\setminus \\mathcal{G}_1$ and $\\mathcal{G}_1 \\setminus \\mathcal{G}_2$.\n\nSTRENGTHS:\n[originality]\n\n- The paper explores the expressiveness of function classes that have fixed complexity and infinite VC dimension. To the best of my understanding, this research area remains largely unexplored beyond the prior results on universal formulas.\n\n[quality]\n\n- The mathematical framework is clearly defined, and the analysis is conducted rigorously.\n\n[clarity]\n\n- The paper is well-written.\n\n[significance]\n\n- The paper identifies certain specific function classes (such as $H^{(5)}$) that are seemingly complex and expressive but do not belong to $\\mathcal{G}_2$ and belong to $\\mathcal{G}_1$.\n\nWEAKNESSES:\n[originality]\n\n- None in particular\n\n[quality]\n\n- None in particular\n\n[clarity]\n\n- None in particular\n\n[significance]\n\n- This paper only focuses on one-dimensional functions, which inevitably restricts the immediate applicability of the presented results to the field of machine learning. However, I consider this limitation to be a reflection of the fact that this area of analysis is still in its early stages.\n\nQUESTIONS:\n- Should Line 327 say $H_N^{(5)}$ instead of $H_N^{(2)}$?\n\n[other minor suggestions]\n\n- Line 353 “suggests” → “suggest”\n- Line 183 “over the field Q” → “over the field Q of rational numbers” (as the field appears less frequently than R or Z in machine learning papers).\n- Line 165 “is a subset” → “is a non-empty subset”\n\nLIMITATIONS:\nThe paper acknowledges a limitation in its methodology, which is explained in the paragraph starting from Line 363. Specifically, the three methods used in the paper cannot effectively distinguish between $\\mathcal{G}_1 \\setminus \\mathcal{G}_2$ and $\\mathcal{G}_2$. As a result, the investigation in the paper is not completed and requires further research.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors isolate the study universal hypothesis subclasses of $C([0,1])$ for the compact-open and point-open topologies respectively, and they compare universality here to infinite VC-dimension.  The results are interesting, though perhaps not that surprising nor novel, but are still publishable (in my opinion).\n\nSome of the author's derivations could use more mathematical rigour and the terminology \"formulas\" is misleading as it alludes to logic/model theory, which does not appear in the paper.  The authors could also add some references to approximation theory in ML (some of which I provide below) to better tie in their results.  Details below in the Weakenesses section.\n\nSTRENGTHS:\nAs stated above, the results are interesting, though perhaps not so surprising.\n\nWEAKNESSES:\n0. Add details to all proofs.  My main issue with this paper, is that many of its proofs contain handwavy and imprecise statements such as: 1) \"Specifically, isolate the variables gn(x) in the polynomials\" (line 739-740).  This is not rigorous and vague 2) equation (94) what space is this limit taken in; to me it seems to be $\\mathbb{C}$ with its modulus as a norm, or 3) \"by a classical theorem of Kronecker's\" on lien 188; which one? (citation missing)... There are many such instances in derivation.\n\n**Requested Change 0:**  Make all proofs rigorous and remove handwaved details (clear limits, clear steps, no missing references)\n\n1. This sentence \"especially dangerous from the generalization point of view\" is purely speculative in the context of the paper, and one can doubt its accuracy [4] and [5] on benign overfitting for high-capacity models.  \n\n**Requested Change 1:** Since the authors only deal with approximation, they should remove this claim in the final version.\n\n2. The hierarchy $G_1\\subset G_2$ in (3) is misleading - The class $G_1$ is simply the set of subsets $A$ of $\\mathbb{R}^{[0,1]}$ satisfying $A$ is dense in $\\mathbb{R}^{[0,1]}$ for the point-open topology, which I denote $\\tau_{PO}$; i.e. the topology of pointwise convergence (see [1] - Section 46, page 281).  So the inclusion of $G_1$ into $G_2$ is immediate and has a simple topological meaning; since $G_2$ is the same but for the compact-open topology (see [1] - Section 46, page 285).  \n\nFrom these remarks, and say Theorem 46.7 and Theorem 46.8 - both on page 281 of [1], imply $G_1\\subset G_2$  and they give a topological interpretation of $G_1\\subset G_2$.  \n\nI point this out because, from this perspective, one has a more general phenomenon.  Let us focus only on continuous functions.  \nConsider the standard poset $(\\mathcal{T},\\lesssim)$ of topologies on $C([0,1])$ with the partial order relation  $\\tau_1\\lesssim \\tau_2$ if and only if $\\tau_2$ is at-least as fine as $\\tau_1$ (i.e.\\ $\\tau_1\\subseteq \\tau_2$).  For any $\\tau\\in \\mathcal{T}$ define the class\n$G_{\\tau}:=\\{A\\subset C([0,1]): \\bar{A}^{\\tau} = C([0,1])\\}$\nwhere $\\bar{A}^{\\tau}$ denotes the closure of a subset $A$ of $C([0,1]$ with respect to the $\\tau$ topology.  One immediately then deduces the contravariant-functorial relation: for all $\\tau,\\tilde{\\tau}\\in \\mathcal{T}$\n$\\tau \\lesssim \\tilde{\\tau} \\Rightarrow G_{\\tilde{\\tau}}\\subseteq G_{\\tau}$.\nSo this type of construction holds in greater generality.  In fact, strictly finer topologies than the compact-open topology (i.e. the topology of uniform convergence since $[0,1]$ is compact and $\\mathbb{R}$ is equipped with a metric - see Munkres Theorem 46.8) and it has been studied in the context of kernel methods [3] (see the strict topology) and non-uniform topologies on $L^1([0,1])$ were considered in [2] (see Theorems 1 and 2) in the context of deep learning.\n\n**Requested Change 2:** Please add such comments circa (3), and emphasize that there are many such \"classes\" between $G_1$ and $G_2$, as well as, below or above then in the chain of inclusions (3).  Clearly the top of the chain consists of $G_{\\tau_{disc}}$ where $\\tau_{disc}$ is the discrete topology on $C([0,1])$ and the *correct* bottom of the chain consists of $G_{\\{C([0,1]),\\emptyset\\}}$ (i.e. $\\{C([0,1]),\\emptyset\\}$ is the trivial topology on $C([0,1])$.  \n\n3. Line 183 - You are considering $\\mathbb{R}$ as an (infinite-dimensiona) $\\mathbb{Q}$-module (vector space over the rationals).  This is not clear for readers who did not have a class or two on algebra during their studies.\n\n**Requested Change 3:** Add an explanation of how $\\mathbb{R}$ is an infinite-dimensional $\\mathbb{Q}$-module.\n\n4. Theorem 2 is not rigersouly stated.  What does \"can approximate any R-valued function on X\".  In reading the theorem's proof you must mean the point-open topology (which in this case consides with the compact-open topology, trivially).\n\n**Requested Change 4:** Remove \"can approximate any R-valued function on X\" in the pointwise/point (in this case compact-open/uniform topologies.  Please write what is meant clearly, with \"the usual for all $\\epsilon$ theres exists an f satisfying .... \" type statement.\n \nWithout this modification I cannot bump my score up to a pass, since the paper must be precise.\n\n5. Missing Reference\n**Requested Change 5:** Please add a precise reference to this hand-waving: \"by classical Kronecker’s theorem\" in the proof of Theorem 2 (it is classical if one has seen such results, but not in general).\n\n6. Branching expressions terminology\n\n**Requested Change 6:** Please call these piecewise functions.\n\n---\n\nI believe the manuscript will be in publishable condition when these changes are made.\n\n**References**\n\n[1] Munkres, James. \"Topology James Munkres Second Edition.\"\n\n[2] Kratsios, Anastasis, and Behnoosh Zamanlooy. \"Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?.\" Transactions on Machine Learning Research (2022).\n\n[3] Chevyrev, Ilya, and Harald Oberhauser. \"Signature moments to characterize laws of stochastic processes.\" The Journal of Machine Learning Research 23.1 (2022): 7928-7969.\n\n[4] Bartlett, Peter L., et al. \"Benign overfitting in linear regression.\" Proceedings of the National Academy of Sciences 117.48 (2020): 30063-30070.\n\n[5] Tsigler, Alexander, and Peter L. Bartlett. \"Benign overfitting in ridge regression.\" J. Mach. Learn. Res. 24 (2023): 123-1.\n\n\n**Note** I did not use mathcal since open-review had compilation issues with it...\n\nQUESTIONS:\n- Related to requested change 2: is there a topology $\\tau$ on $C([0,1])$ such that $\\mathcal{G}_{\\tau}=\\mathcal{G}_0$?\n- In the proof of Theorem 2, does one really need to index over $\\mathbb{R}$; isn't $\\mathbb{R}\\setminus \\mathbb{Q}$ enough?\n- Why do you call these formulas?  Is there any connection to logic/model theory?  To me the correct/standard term is universal hypothesis/function classes, in the context of approximation theory and universal approximation theorems.\n\nLIMITATIONS:\nNot really, but this section doesn't really affect this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study \"universal formulas\" -- specifically, families of parametric\nmappings $(f_{\\omega})\\_{\\omega \\in \\Omega} \\subset C([0, 1])$ where $\\Omega$ is\na domain in a finite-dimensional euclidean space, such that $(f_{\\omega})$ is\nactually dense in $C([0, 1])$ with the sup-norm -- motivated in part by\napplications to neural networks. In this connection, universal formulas should\nbe contrasted with typical \"universal approximation\" results for neural\nnetworks that state that various fixed-depth neural networks can approximate\nany continuous function on $[0,1]$ by *increasing the number of neurons*\nsufficiently; universal formulas achieve this approximation with a fixed number\nof neurons by instead exploiting (at least indirectly) unbounded-magnitude\nparameters and infinite-precision arithmetic. The authors present a hierarchy\nof nested expressiveness classes -- in decreasing order, families with infinite\nVC dimension, families that achieve arbitrary approximation on arbitrary finite\nsets, and families that achieve arbitrary approximation of arbitrary continuous\nfunctions (i.e., universal formulas) -- and study obstructions to a family\nbeing in a class higher in the hierarchy, but not a class lower in the\nhierarchy. Starting from the classical example of the infinite VC-dim class\n$\\\\{c \\sin \\omega x\\\\}_{c,\\omega \\in \\mathbb{R}}$, the authors prove using\nalgebraic methods that a fairly general class of\n\"polynomially-exponentially-algebraic\" expressions, which includes neural\nnetworks with topology given by a (fixed size) directed acyclic graph with only\none \"algebraic-exponential\" activation on any path to the output (e.g., $\\sin$\nactivation, $\\tanh$ activation) and arbitrary piecewise-polynomial activations\notherwise (e.g., ReLU), cannot achieve the arbitrary finite approximation\nproperty.\nResults for networks achieving finite approximation but not being universal\nformulas are slightly less general, due to technical challenges: the authors\nshow, for example, that classes $f(x) = c \\sin (\\omega \\sigma(bx) + h)$ achieve\narbitrary finite approximation if and only if $\\sigma$ is non-polynomial, and\ndemonstrate that for a certain sub-class of $\\sigma$ maps studied earlier\ninvolving $\\sin$ activations, the resulting class does not have the universal\nformula property. This kind of mapping can be viewed as a one-hidden-layer\nneural network with one neuron on the hidden layer and $\\sin$ activations; they\nconjecture that the same separation holds for similar architectures with\nmultiple neurons in the hidden layer. \nWith these results established, they discuss known results on architectures\nthat do constitute universal formulas, and possible paths to characterize\nnecessary and sufficient conditions to be a universal formula further.\n\nSTRENGTHS:\nThe writing, both technical and expository, is clear and precise. The authors\ngo to great lengths to present a broad view of the problem and prior work in\nthe introduction, which makes the subsequent technical analysis accessible to a\nnon-expert reader.\n\nThe authors defer more technical proofs to the appendices, but always give a\nclear indication in the main submission what the types of arguments developed\nare (e.g., algebraic methods vs.\\ Pfaffian function theory). This will make the\nwork useful for follow-up work.\n\nThe discussion section (section 8) seems uncommonly interesting for a\nsubmission in this venue, in that it lists several open problems that the\nauthors' work suggests, and which should lead to interesting follow-up work.\n\nWEAKNESSES:\nIt is not a weakness per se, but as the authors themselves note, the\ninvestigation is mostly of mathematical interest as \"... the universality of\nuniversal formulas is... an idealization that requires their parameters to have\nunbounded precision or magnitude\". Nonetheless, there are neural networks\ncommonly used that have structures similar to some of the near-universal\nformulas studied (the authors mention SIREN, which uses periodic activations in\nneural radiance field learning), and these kinds of periodic constructions have\nbeen shown to be useful in studying general deep/shallow neural network\napproximability (e.g., work of Telgarsky).\n\nQUESTIONS:\n### Questions and Comments\n\nDoes the \"GeLU\" activation function ($x \\mapsto x \\Phi(x)$, where $\\Phi$ is the\nstandard normal cumulative distribution function) satisfy the\nalgebraic-exponential property? I am not sure if it would, because it is\ndefined with an integral, but if so, it might be interesting to mention here\n(this activation is used in some modern transformer architectures).\n\n\n### Minor Fixes\n\nCaption of Figure 1: typo \"uiversality\"\n\nLine 186: the $2\\pi$ is on the wrong factor in the quotient? (the reals,\n\"modulo\" $2\\pi$).\n\nLine 187: is this supposed to be the \"fractional part\" operator (based on its\nspecified domain) rather than the \"floor\" operator (which as I understand gives\nthe next-lowest integer)?\n\nLine 345: \"Then the existence of a ...\" -- should this be \"Then there exists\na...\"?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-28"}