SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dmitry Yarotsky
d.yarotsky@skoltech.ru
By universal formulas we understand parameterized analytic expressions that have a fixed complexity, but nevertheless can approximate any continuous function on a compact set. There exist various examples of such formulas, including some in the form of neural networks. In this paper we analyze the essential structural elements of these highly expressive models. We introduce a hierarchy of expressiveness classes connecting the global approximability property to the weaker property of infinite VC dimension, and prove a series of classification results for several increasingly complex functional families. In particular, we introduce a general family of polynomially-exponentially-algebraic functions that, as we prove, is subject to polynomial constraints. As a consequence, we show that fixed-size neural networks with not more than one layer of neurons having transcendental activations (e.g., sine or standard sigmoid) cannot in general approximate functions on arbitrary finite sets. On the other hand, we give examples of functional families, including two-hidden-layer neural networks, that approximate functions on arbitrary finite sets, but fail to do that on the whole domain of definition.
By universal formulas we broadly (informally) understand parameterized explicit analytic expressions that have a fixed complexity (as expressions) but nevertheless can approximate any continuous function on a compact set. An example of such formula, given by Boshernitzan [1], is
$$
y (x) = \int_ {0} ^ {x + a} \frac {b d}{1 + d ^ {2} - \cos (b t)} \cos \left(e ^ {t}\right) d t + c, \tag {1}
$$
where  $d > 0, a, b$  and  $c$  are parameters. [1] proves that, by varying the parameters, functions (1) can uniformly approximate any continuous function on any compact interval in  $\mathbb{R}$ .
Expression (1) involves integration. Laczkovich and Ruzsa [13] prove existence of universal formulas representable as elementary functions without integration (moreover, their formulas can uniformly approximate continuous functions on the whole  $\mathbb{R}$  under a mild growth assumption).
One can further restrict the types of operations and show that universal formulas can be realized by fixed-size classical neural networks with suitable activation functions [29]. By a classical neural network we mean a computational model consisting of units ("hidden neurons"), each computing a map  $z_{1},\ldots ,z_{n}\mapsto \sigma (\sum_{k = 1}^{n}w_{k}z_{k} + h)$ , where  $z_{1},\ldots ,z_{n}$  (signals) are outputs of some previous neurons, and  $w_{k}$  and  $h$  are weights (parameters) of the neuron. In addition to hidden neurons, one distinguishes "input neurons" that just represent the scalar components of the network input, and "output neuron(s)" that represent the scalar component(s) of the output. The output neurons work as hidden neurons but without activations, i.e. as  $z_{1},\ldots ,z_{n}\mapsto \sum_{k = 1}^{n}w_{k}z_{k} + h$ . As defined, neural networks are fairly restrictive in terms of used operations: for example, they contain multiplications of signals  $z$  by constants (weights)  $w$ , but not multiplications between signals,  $z_{1}z_{2}$ . However, this and some other extra operations can be implemented with any accuracy using suitable combinations of neurons. The universal network in [29] uses a fixed architecture graph, activations sin and arcsin,
Figure 1: Left: A (slightly simplified) fixed-size universal neural network approximating any  $f \in C([0,1])$  from [29]. Most hidden neurons are standard neurons  $z_{1},\ldots ,z_{n} \mapsto \sigma (\sum_{k = 1}^{n}w_{k}z_{k} + h)$  with one of the activation functions  $\sigma$  (identity, sin or arcsin). Also, there are several multiplication neurons. The parameters of the model are all the weights  $w_{k},h$  in the standard neurons. Right: A multiplication neuron can be replaced by a group of standard neurons while preserving universality.
and can approximate any  $f \in C([0,1])$  (see Figure 1). Another fixed-size universal network had been constructed earlier in [15], but the activation was not an elementary function in that work.
The Kolmogorov(-Arnold) Superposition Theorem [10] shows that any function  $f \in C([0,1]^d)$  can be expressed in terms of a fixed number (only depending on  $d$ ) of summations and univariate continuous functions. This implies that multivariate universal formulas can be easily constructed from univariate ones, by plugging the latter in the Kolmogorov ansatz. Accordingly, the question of multivariate universal formulas is not significantly more interesting or complicated than its counterpart for univariate formulas, and in this paper restrict our attention to the latter. We also mention in passing another interesting property of universal formulas: they naturally give rise to universal polynomial differential equations  $P(x,f,f^{\prime},\ldots ,f^{(n)}) = 0$ , with a polynomial  $P$  and a dense set of solutions in  $C([a,b])$ , see [21, 2, 1, 19, 13]. (In fact, one can find such a polynomial for any parameterized family of elementary functions; the density of solutions then follows from the universality of the formula.) See [22] for a connection to analog computers.
A foundational theorem on neural networks is the universal approximation theorem (UAT). It states that, for a broad class of activation functions, a neural network with a single layer of hidden neurons can approximate any continuous function on a compact set if we increase the number of neurons [4, 18]. This result is very different from, and should not be confused with the above fixed-size universal networks of [15] or [29]. In UAT, universality is achieved by increasing the number of parameters, and accordingly UAT imposes much weaker requirements on the activation functions (e.g., it holds for any continuous non-polynomial activation, see [14, 18]).
On the other hand, the fixed-size universal network in Figure 1 is fairly simple. One of the two activation functions used there, sin, is actually used as an activation in practical multi-layer networks (e.g., the popular SIREN network of [23]). This raises the following natural question motivating the present paper: can practically used neural-network-type models be universal as fixed-size formulas (i.e., without increasing the number of neurons)?
There is an important caveat here: the universality of universal formulas is, of course, an idealization that requires their parameters to have unbounded precision or magnitude. If parameters are finitely defined so as to be representable on real computer, one can give a simple entropy bound constraining feasible approximation accuracy. Specifically, suppose that a formula contains  $p$  parameters, each somehow represented using  $B$  bits (e.g.,  $B = 32$  or 64 for standard floats). Suppose we are approximating a function  $f$  at  $N$  points  $x_{1},\ldots ,x_{N}$  with absolute accuracy  $\epsilon$ . Suppose finally that  $|f|$  is bounded by  $M$  and  $f(x_{k}), k = 1,\dots ,N$ , can be chosen independently subject to this constraint. Then
$$
N \log_ {2} (M / \epsilon) \leq p B. \tag {2}
$$
This shows that in a practical application with parameters implemented as standard floats, universal formulas such as (1) can typically produce only crude approximations. In the sequel, we will ignore these finite-precision aspects.
Our contribution. Our goal in this paper is to understand which structural properties are critical for universal formulas. This question does not seem to be simple. As a general strategy of tackling it, we analyze a sequence of progressively more complex parametric functional families with respect to several relevant classes of model expressiveness. Our specific contributions are as follows.
1. We introduce a hierarchy  $\mathcal{G}_0\supset \mathcal{G}_1\supset \mathcal{G}_2$  of classes of model expressiveness (Section 2). Here  $\mathcal{G}_0$  represents the families of infinite VC-dimension,  $\mathcal{G}_1$  the families achieving approximation on arbitrary finite sets, and  $\mathcal{G}_2$  the families achieving uniform approximation on the whole segment [0, 1]. Universal formulas correspond to the class  $\mathcal{G}_2$ . The class  $\mathcal{G}_0$  is important because elementary functions belonging to it must involve, in some form, the function sin acting on an unbounded domain (Section 3). We argue that the class  $\mathcal{G}_1$  is also important because, as we show, some functional families cannot belong to it due to algebraic (polynomial) constraints.
2. As a first result on algebraic constraints, we consider the family of fixed-size linear combinations of sine waves with unbounded weights (equivalently, fixed-size single-hidden-layer neural networks with activation sin) and prove that it belongs to the class  $\mathcal{G}_0\setminus \mathcal{G}_1$ . Moreover, we give a complete description of the limit points of this family (Section 4).
3. As a generalization of the previous result, we introduce a general family of polynomially-exponentially-algebraic expressions of bounded complexity, and prove that it is also subject to polynomial constraints and lies outside  $\mathcal{G}_1$  (Section 5).
4. We give an application of the previous result to branching expressions and neural networks with conventional activations (Section 6). Specifically, we prove that if the network has a bounded complexity and possibly multiple layers, but only one layer with transcendental activation functions (such as  $\sin x$ , Gaussian or standard sigmoid), then the network is subject to polynomial constraints and lies outside  $\mathcal{G}_1$ . Moreover, polynomial constraints hold even if the activation functions are only piecewise analytic, as is common in practice.
5. The situation is drastically different for formulas involving compositions of non-polynomial analytic functions and sin: such formulas generally belong to  $\mathcal{G}_1$  (Section 7). It seems hard to separate the classes  $\mathcal{G}_1 \setminus \mathcal{G}_2$  and  $\mathcal{G}_2$ . We give a couple of examples of families for which we can prove that they belong to  $\mathcal{G}_1 \setminus \mathcal{G}_2$ . One of these families can be described as that of fixed-complexity two-hidden-layer networks with sin activation that have a single neuron in the second hidden layer. We conjecture that general sine networks of fixed complexity also belong to  $\mathcal{G}_1 \setminus \mathcal{G}_2$ .
Most proofs are deferred to the appendix (the respective sections are indicated in the theorems).
2 The hierarchy of expressiveness classes
Throughout the paper, we consider various families  $H$  of functions  $g:[0,1]\to \mathbb{R}$ . We define for them expressiveness classes  $\mathcal{G}_0,\mathcal{G}_1,\mathcal{G}_2$ :
$$
H \in \mathcal {G} _ {0} \stackrel {\text {d e f}} {\Longleftrightarrow} \operatorname {V a p n i k - C h e r v o n e n k i s} (H) = \infty ;
$$
$$
\begin{array}{l} H \in \mathcal {G} _ {1} \stackrel {{\text {d e f}}} {{\Longleftrightarrow}} \text {F o r a n y f i n i t e s u b s e t} \left\{x _ {1}, \ldots , x _ {N} \right\} \in [ 0, 1 ], \text {a n y} y _ {1}, \ldots , y _ {N} \in \mathbb {R} \text {a n d a n y} \epsilon > 0 \text {t h e r e} \\ \text {i s} g \in H \text {s u c h t h a t} | g (x _ {k}) - y _ {k} | <   \epsilon \text {f o r a l l} k = 1, \ldots , N. \end{array}
$$
$$
\begin{array}{l} H \in \mathcal {G} _ {2} \stackrel {{\text {d e f}}} {{\Longleftrightarrow}} \text {F o r a n y} f \in C ([ 0, 1 ]) \text {a n d} \epsilon > 0, \text {t h e r e i s} g \in H \text {s u c h t h a t} \| g - f \| _ {\infty} = \\ \sup  _ {x \in [ 0, 1 ]} | g (x) - f (x) | <   \epsilon . \end{array}
$$
Here,  $\mathrm{VCdim}(H_f) = \infty$  means that for any  $N\in \mathbb{N}$  we can find a size-  $N$  subset  $X\subset [0,1]$  shattered by  $H$  thresholded at 0, i.e. such that for any  $S\subset X$  there is  $g\in H$  for which  $g(x) > 0$  if  $x\in S$  and  $g(x)\leq 0$  if  $x\in X\setminus S$ .
$$
\mathcal {G} _ {0} \supseteq \mathcal {G} _ {1} \supseteq \mathcal {G} _ {2}. \tag {3}
$$
The classes  $\mathcal{G}_1$  and  $\mathcal{G}_2$  reflect approximability on arbitrary finite sets and on the whole interval  $[0, 1]$ , respectively. Universal formulas as discussed in the Introduction correspond to the class  $\mathcal{G}_2$ . One can
say that  $\mathcal{G}_1$  corresponds to universality in a weaker sense, which is particularly relevant if we try to learn the parameters using a finite training set  $\{(x_k,y_k)\}$ .
We will consider several families  $H$  that we will distinguish by superscripts denoting or enumerating their types (e.g.  $H^{f}$  or  $H^{(1)}, H^{(2)}, \ldots$ ) and by subscripts reflecting complexity within the type (e.g.,  $H_{N}^{(2)}$  will denote linear combinations of  $N$  sine waves). Most of these families admit natural interpretations in terms of conventional neural networks – we will mention them along the way.
While the remainder of this work focuses on the mathematical analysis of the relation between various functional families and the classes  $\mathcal{G}_k$ , let us briefly clarify the role of these classes from the general perspective of approximation and learning.
1. Most conventional learning models (e.g., neural networks) can be said to be defined using (piecewise) elementary functions. If the model is assumed to have a bounded complexity (e.g., a bounded number of neurons) and its weights and signals are bounded, this model falls outside even the broadest class  $\mathcal{G}_0$  (see Section 3 below). This does not apply, however, to models containing the function  $\sin$  acting on an unbounded domain.
2. For a fixed-complexity model, being outside the classes  $\mathcal{G}_k$  and thus not having the full approximation power is not a practically negative property: this just means that the complexity of the model needs to increase to achieve a better fit.
3. The situation when a fixed-complexity model belongs to the class  $\mathcal{G}_2$  (i.e., is a universal formula) is mathematically interesting but somewhat exotic from the perspective of conventional computation, since the respective model parameters need to contain a potentially unbounded amount of information.
4. The class  $\mathcal{G}_1 \setminus \mathcal{G}_2$  seems especially dangerous from the generalization point of view: given a hypothesis space  $H$  from this class, we can fit a model arbitrarily well to a generic continuous ground truth on any finite training set, but can hardly ever uniformly approximate this ground truth on the whole domain. This cannot be said about the class  $\mathcal{G}_2$ . In Section 7 we give examples of functional families (neural networks) provably belonging to this dangerous class  $\mathcal{G}_1 \setminus \mathcal{G}_2$ .
3 Pfaffian functions
There is an important broad class of parameterized functions  $f(x, \mathbf{w})$  for which one can guarantee that the family  $H^f$  obtained by fixing various  $\mathbf{w}$  has a finite VC-dimension. This class is most naturally described in terms of Pfaffian functions introduced by [9]. Pfaffian functions can be defined using function sequences in which, at each step, the next function satisfies polynomial first-order differential equations involving itself and previous functions (see [9, 5, 30] for details). In particular, Pfaffian functions include all elementary functions when the latter are defined on suitable domains.
Importantly, in contrast to functions like  $\ln$ ,  $\exp$  and polynomials, which are Pfaffian on their whole natural domain of definition, the function  $\sin$  is not Pfaffian on the whole set  $\mathbb{R}$ , though it is Pfaffian on any bounded interval  $(a,b)$  (moreover, the representation of  $\sin$  as a Pfaffian function depends on  $(a,b)$  and becomes more complex as  $b - a$  increases). For this reason, elementary functions are Pfaffian only when they involve  $\sin$  (or a closely related function, like  $\cos$ ) restricted to bounded intervals.
The fundamental theorem of [9] gives an explicit finite upper bound on the number of zeros of Pfaffian functions, which implies, in particular, that the respective parametric families have finite VC-dimension. We state this result specialized to elementary functions and in the form suitable for our purposes; see [9, 5, 30, 29] for details.
Theorem 1. Suppose that a formula  $y = f(x, \mathbf{w})$  is constructed from the variables  $x$  and  $w_1, \ldots, w_d$  using real numbers, standard arithmetic operations  $(+, -, \times, /)$ , elementary functions  $\ln$ ,  $\exp$ ,  $\sin$ ,  $\arcsin$ , and compositions. Suppose that there is a nonempty subset  $W \subset \mathbb{R}^d$  such that for any  $\mathbf{w} = (w_1, \ldots, w_d) \in W$  the function  $f(\cdot, \mathbf{w}) : [0, 1] \to \mathbb{R}$  is well-defined, i.e.,  $\ln$  and  $\arcsin$  are applied on the intervals  $(0, \infty)$  and  $(-1, 1)$ , respectively, and there is no division by 0. Moreover, suppose that there is a bounded interval  $(a, b)$  to which the arguments of  $\sin$  always belong for all  $\mathbf{w} \in W$ . Then the family  $H^f = \{f(\cdot, \mathbf{w})\}_{\mathbf{w} \in W}$  has a finite VC-dimension.
This theorem shows that an elementary function has chances to be universal only if it includes sin (or some related function) acting on an unbounded domain. Therefore, we will focus on these latter functions in the sequel.
It is crucial for Theorem 1 that all the computations in the formula are real-valued and not complex-valued (the proof eventually boils down to Rolle's theorem, which does not hold for  $\mathbb{C}$ -valued functions). In particular, we cannot bypass the domain restriction of  $\sin$  by simply writing  $\sin x = (e^{ix} - e^{-ix}) / 2i$ . In contrast, many results in the next sections will be essentially algebraic and will hold for complex-valued operations.
We remark that Sontag [25] considers a class of formulas closely related to Pfaffian functions and proves that shattering  $k$ -point sets in general position require at least  $(k - 1) / 2$  parameters in such formulas.
4 Linear combinations of sines
A simple example of a family with an infinite VC-dimension is  $H^{(1)} = \{c\sin (\omega x)\}_{c,\omega \in \mathbb{R}}$ . This is usually proved (see e.g. [28], Section 3.6) by giving a particular example of an arbitrarily large finite set shattered by  $H^{(1)}$ . For our purposes, however, it is instructive to describe a general collection of finite sets on which  $H^{(1)}$  has universal approximability. We say that numbers  $x_{1},x_{2},\ldots$  are rationally independent if they are linearly independent over the field  $\mathbb{Q}$  of rational numbers.
Theorem 2. Let  $X = \{x_{1},\ldots ,x_{N}\}$  be a finite subset of [0,1] such that the values  $x_{1},\dots,x_{N}$  are rationally independent. Then the family  $H^{(1)}$  can approximate any  $\mathbb{R}$ -valued function on  $X$ .
Proof. Consider the torus  $\mathbb{T}^N = (\mathbb{R} / 2\pi \mathbb{Z})^N$ ; its points can be identified with the points of the cube  $[0,2\pi)^N$ . Let  $\lfloor \cdot \rfloor : \mathbb{R} \to [0,1)$  be the standard floor function. By a classical Kronecker's theorem [11, 7], the set  $\{(2\pi (\theta x_1 - \lfloor \theta x_1 \rfloor), \ldots, 2\pi (\theta x_N - \lfloor \theta x_N \rfloor))\}_{\theta \in \mathbb{R}}$  is a dense subset of  $\mathbb{T}^N$  if and only if the numbers  $x_1, \ldots, x_N$  are rationally independent. Using the  $2\pi$ -periodicity of  $\sin$ , this implies immediately that if  $x_1, \ldots, x_N$  are rationally independent, then  $\{(\sin(\omega x_1), \ldots, \sin(\omega x_N))\}_{\omega \in \mathbb{R}}$  is dense in  $(-1,1)^N$  and hence  $\{(c\sin(\omega x_1), \ldots, c\sin(\omega x_N))\}_{c,\omega \in \mathbb{R}}$  is dense in  $\mathbb{R}^N$ .
This result shows that the family  $H^{(1)} \in \mathcal{G}_0$ . It is also clear that  $H^{(1)} \notin \mathcal{G}_1$  (e.g., since  $g(0) = 0$  for any  $g \in H^{(1)}$ ). In fact, the sufficient condition of rational independence in the theorem is close to also being necessary. In particular, a simple kind of sets not shattered by  $H^{(1)}$  are all nontrivial arithmetic progressions  $x_k = u + vk, v \neq 0$ , of length 5. Indeed, it follows from the identity  $\sin(u + (k - 1)v) + \sin(u + (k + 1)v) = 2\sin(u + kv)\cos(v)$  that the values of functions from  $H^{(1)}$  cannot, for example, have the sign configuration  $++ + - +$  on such progressions.
Consider now the more general family consisting of all linear combinations of a fixed number of sine waves with arbitrary frequencies and phases:
$$
H _ {N} ^ {(2)} = \left\{\sum_ {n = 1} ^ {N} c _ {n} \sin \left(\omega_ {n} x + h _ {n}\right) \right\} _ {c _ {n}, \omega_ {n}, h _ {n} \in \mathbb {R}, n = 1, \dots , N}. \tag {4}
$$
One can view this family as representing single-hidden-layer neural networks with the sin activation function  $(\omega_{n}$  and  $h_n$  are the weights and biases in the hidden layer, and  $c_{n}$  are the weights in the output layer).
One can again easily show that  $H_N^{(2)} \in \mathcal{G}_0 \setminus \mathcal{G}_1$ , by observing that the values of all functions  $g \in H_N^{(2)}$  are constrained by common nontrivial polynomial equations:
1. For any  $g \in H_N^{(2)}$ ,  $x_0 \in \mathbb{R}$  and  $\alpha_n, \beta_n \in \mathbb{R}$ ,  $n = 0, \ldots, 2N$ ,
$$
\det  A = 0, \text {w h e r e} A = \left(g \left(x _ {0} + \alpha_ {k} + \beta_ {m}\right)\right) _ {k, m = 0} ^ {2 N}. \tag {5}
$$
2.  $\det A$  is a nonzero polynomial in the variables  $g(x)$ , where  $x \in X = \{x_0 + \alpha_k + \beta_m | k, m = 0, \ldots, 2N\}$ , if and only if all  $\alpha_k$  are different and all  $\beta_m$  are different.
Proof. 1. The matrix  $A$  is degenerate because each of its  $2N + 1$  rows is a linear combination of  $2N$  vectors  $(e^{s\omega_n\beta_1}, \ldots, e^{s\omega_n\beta_N})$  with  $n = 1, \ldots, N$  and  $s = \pm i$ .
2. If some  $\alpha_{k}$  or  $\beta_{m}$  are repeated, then the matrix  $A$  contains repeated columns or rows and so the polynomial  $\operatorname{det} A$  identically vanishes. Conversely, suppose that all  $\alpha_{k}$  are different and all  $\beta_{m}$  are different. Without loss, we can assume  $\alpha_{0} < \ldots < \alpha_{2N}$  and  $\beta_{0} < \ldots < \beta_{2N}$ . We can then expand  $\operatorname{det} A = g(x_{0} + \alpha_{2N} + \beta_{2N})\operatorname{det} A' + \ldots$ , where  $\operatorname{det} A'$  is the top left  $2N \times 2N$  minor and the terms ... do not contain the variable  $g(x_{0} + \alpha_{2N} + \beta_{2N})$ . Repeating the expansion for  $A'$ , etc., we see that the polynomial  $\operatorname{det} A$  contains the monomial  $\prod_{k=0}^{2N} g(x_{0} + \alpha_{k} + \beta_{k})$  with coefficient 1, i.e. is nonzero.
This result shows that there are (infinitely many) nontrivial polynomial constraints on the values of the functions  $g \in H_N^{(2)}$ . Importantly, these constraints are preserved under pointwise limits, so, in particular,  $H_N^{(2)} \in \mathcal{G}_0 \setminus \mathcal{G}_1$ . Taking  $N = 1$  and  $\alpha_k = \beta_k = vk, k = 0,1,2$ , we obtain a nontrivial constraint for the values of  $g$  on any nontrivial length-5 arithmetic progression  $x_{k} = u + vk$ , in agreement with an earlier observation.
In fact, thanks to the semi-linear structure of the family  $H_N^{(2)}$ , we can go further and give an explicit description of its limit points. It turns out that the limiting functions can only differ from the original combinations of sine waves by resonant factors  $x^m$ :
Theorem 4 (A). Let  $N$  be fixed. Then a function  $f_*$  is a limit point of  $H_N^{(2)}$  in the sense of uniform convergence on the segment [0,1] if and only if
$$
f _ {*} (x) = \sum_ {m = 0} ^ {2 M _ {0} - 1} c _ {0 m} x ^ {m} + \sum_ {k = 1} ^ {K} \sum_ {m = 0} ^ {M _ {k} - 1} c _ {k m} x ^ {m} \sin \left(\omega_ {k} x + h _ {k m}\right), \tag {6}
$$
where  $M_0 \geq 0$ ,  $M_k \geq 1$  for  $k = 1, \ldots, K$ , and  $\sum_{k=0}^{K} M_k = N$ .