SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Resonant factors appear by taking suitable combinations of sine waves at close frequencies, e.g.  $x\sin (\omega x) = \lim_{\Delta x\to 0}(\Delta \omega)^{-1}(\sin ((\omega +\Delta \omega)x) - \sin (\omega x))$  . We show explicitly how to construct resonances of arbitrary degrees subject to the constraint  $\sum_{k = 0}^{K}M_{k} = N$  . Note that the frequency  $\omega = 0$  corresponding to the first sum in expansion (6) is special as it is associated with a potential double-degree resonance.
The more difficult part of the proof is the "only if" part, i.e. that any limiting function has the form (6). Our basic idea is to show that a limiting function is subject to a linear differential equation describing waves with a given set of frequencies; the general resonant form (6) would then appear from the usual solution of such an equation with nontrivial multiplicities of the frequencies. This strategy, however, is not so easy to implement rigorously, since in our setting the frequencies are unbounded. We give two alternative proofs overcoming this difficulty. The first, clustering-based proof, groups the frequencies found in a converging sequence  $f_{n} \in H_{N}^{(2)}$  into spatially separated "frequency clusters", and then shows that unbounded clusters can be removed without changing the limit. The second, discretization-based proof, starts with describing the discretized functions of  $H_{N}^{(2)}$  by suitable linear finite-difference equations which also hold for the limiting function  $f_{*}$ . Then, we consider these discretizations at different length scales and show that they can only be consistent with each other and with the continuity of the uniform limit  $f_{*}$  if representation (6) holds.
5 Polynomials-exponentially-algebraic expressions
Theorem 3 shows that the values of functions  $H_N^{(2)}$  are subject to nontrivial polynomial constraints preserved under pointwise limits. We extend this argument to a much larger family of functions, involving complex algebraic operations. This generalization will allow us, in particular, to prove a related result for multi-layer neural networks.
If  $U \subset \mathbb{C}^N$  is an open domain, we say that a holomorphic function  $Q: U \to \mathbb{C}$  is an algebraic function if there exist  $N$ -variable polynomials  $\phi_0, \ldots, \phi_q$ , not all zero, such that
$$
\sum_ {k = 0} ^ {q} \phi_ {k} \left(z _ {1}, \dots , z _ {N}\right) Q ^ {k} \left(z _ {1}, \dots , z _ {N}\right) = 0, \quad \forall \mathbf {z} = \left(z _ {1}, \dots , z _ {N}\right) \in U. \tag {7}
$$
The value  $q$  is called the degree of the algebraic function  $Q$ . Algebraic functions include, in particular, all polynomials and (on their holomorphy domains) rational functions and powers  $z^a$  with rational  $a$ . Arithmetic operations and compositions applied to algebraic function produce again algebraic functions.
We define the family  $H_{N,q,r,d}^{(3)}$  as consisting of all functions  $g:[0,1]\to \mathbb{R}$  of the form
$$
g (x) = Q \left(x, e ^ {P _ {1} (x)}, \dots , e ^ {P _ {N} (x)}\right), \tag {8}
$$
where  $P_{1},\ldots ,P_{N}$  are complex-valued polynomials of degree  $\leq d$  , and  $Q$  is an algebraic function of degree  $\leq q$  such that all polynomials  $\phi_{k}$  appearing in its definition have degree  $\leq r$  . Here, the functions  $Q$  are assumed to be defined on some domains  $U\subset \mathbb{C}^{N + 1}$  that contain all the values  $(x,e^{P_1(x)},\dots ,e^{P_N(x)}),x\in [0,1]$  . Our main result is the following extension of Theorem 3.
Theorem 5 (B). Fix  $N, q, r, d$ . Let  $m = (q + 1)\binom{N + r + 1}{N + 1} + (\max(1, d) + 1)(N + 1)$ . Then there exists a nonzero polynomial  $R$  of  $m + 1$  variables such that for any  $g \in H_{N,q,r,d}^{(3)}$  and any  $a, b \in [0,1]$
$$
R (g (a), g (a + h), g (a + 2 h), \dots , g (b)) = 0, \quad h = \frac {b - a}{m}. \tag {9}
$$
As a corollary, since the polynomial  $R$  is the same for all  $g \in H_{N,q,r,d}^{(3)}$ , constraint (9) is preserved by pointwise limits, so  $H_{N,q,r,d}^{(3)} \notin \mathcal{G}_1$ .
Our proof of Theorem 5 is inspired by the theory of algebraic-transcendent functions [16, 17, 8] that addresses representability of functions by algebraic differential equations. Our context is rather different: we are interested in the pointwise convergence of functions (or convergence in the uniform norm), so conditions involving derivatives are not applicable in our setting. We observe, however, that some elements of this theory can be extended from the usual to discretized derivatives. This is why, in particular, the conditions in Eq. (9) are formulated for a regular grid of points  $a, a + h, \ldots, b$ .
Since the polynomials  $P_{n}$  in the definition of  $H_{N,q,r,d}^{(3)}$  are allowed to be complex-valued, the families  $H_{N,q,r,d}^{(3)}$  subsume the sine wave families  $H_N^{(2)}$  considered in the previous section:
$$
H _ {N} ^ {(2)} \subset H _ {2 N, 1, 1, 1} ^ {(3)}. \tag {10}
$$
6 Branching expressions and neural networks
We describe now an application of Theorem 5 to neural networks. We start by noting that many standard activation functions can be classified as piecewise polynomial or (piecewise) algebraic-exponential<sup>1</sup>:
Piecewise polynomial: Binary step function  $\sigma(x) = \mathbf{1}[x \geq 0]$ , ReLU  $\sigma(x) = \max(0, x)$ , leaky ReLU  $\sigma(x) = \max(ax, x)$ , squared ReLU  $\sigma(x) = \max^2(0, x)$  [24].
(Piecewise) algebraic-exponential:  $\sin x$ ,  $\tanh x$ , standard sigmoid  $\sigma(x) = (1 + e^{-x})^{-1}$ , ELU
$$
\sigma (x) = \left\{ \begin{array}{l l} a (e ^ {x} - 1), & x <   0 \\ x, & x \geq 0 \end{array} \right. [ 3 ], \text {s w i s h} \sigma (x) = x / (1 + e ^ {- x}) [ 2 0 ], \text {G a u s s i a n} \sigma (x) = e ^ {- x ^ {2}}.
$$
"Piecewise" here means that  $\mathbb{R}$  is in general divided into several subsets (in fact, up to two subsets in the above examples) on which the activation is defined by different analytic formulas. We will refer to this as "branching". On each branch, activations of the first type are polynomials, and those of the second type can be represented in the form (8) with a polynomial or rational  $Q$ .
Define the family  $H_N^{(4)}$  of functions  $g:[0,1]\to \mathbb{R}$  as realization, with various weight assignment, of the following class of feedforward neural networks. Suppose that the underlying directed acyclic graph of the network contains at most  $N$  neurons. Suppose that each hidden neuron is equipped with one of the piecewise polynomial or algebraic-exponential activation functions mentioned above (possibly different at different neurons). Suppose finally that any directed path from the input to the output neuron contains at most one (piecewise) algebraic-exponential neuron (for example, this is the case if the network has a single algebraic-exponential layer, while the other layers are piecewise polynomial). Then we have
<sup>1</sup>Of course, there also exist standard activations not of these two types, e.g. softplus  $\sigma(x) = \ln(1 + e^x)$  [6].
Theorem 6 (C). Given  $N$ , there exists  $m$  and a nontrivial polynomial  $R$  of  $m + 1$  variables such that for any  $g \in H_N^{(4)}$  and any  $a, b \in [0,1]$
$$
R (g (a), g (a + h), g (a + 2 h), \dots , g (b)) = 0, \quad h = \frac {b - a}{m}. \tag {11}
$$
The main obstacle in deriving this theorem from Theorem 5 is the branching in the activations that destroys the global analytic structure of functions  $g \in H_N^{(4)}$ . We overcome this obstacle using the well-known Van der Waerden's theorem on arithmetic progressions.
Theorem 7 ([27]; see, e.g., [26] for a short proof). For any integers  $s, p \geq 1$  there exists an integer  $N_{vdW}(s,p) \geq 1$  such that every coloring  $C: \{1,\dots,N_{vdW}\} \to \{1,\dots,p\}$  of  $\{1,\dots,N_{vdW}\}$  into  $p$  colors contains at least one monochromatic arithmetic progression of length  $s$  (i.e. a progression in  $\{1,\dots,N_{vdW}\}$  of cardinality  $s$  on which  $C$  is constant).
7 Compositions of sines and non-polynomial functions
The polynomially-exponentially-algebraic structure (8) is actually close to being necessary for the polynomial constraints of Theorem 5, as we show by the following example. Let  $\sigma$  be a real analytic function on the interval  $[-1,1]$ . Let  $f(x) = c\sin (\omega \sigma (bx) + h)$  with parameters  $c,b,h\in \mathbb{R}$  and  $b\in [-1,1]$ . Denote the respective family by  $H^{\sigma}$ .
Theorem 8 (D).  $H^{\sigma}\in \mathcal{G}_1$  if and only if  $\sigma$  is not a polynomial.
This shows that the polynomial constraints of Theorem 5 are destroyed if we relax even slightly the requirement of polynomial arguments of the exponentials in Eq. (8). They are also destroyed if we replace the algebraic function  $Q$  by the sine function.
While  $H^{\sigma} \in \mathcal{G}_1$  for non-polynomial  $\sigma$ , we can show that  $H^{\sigma}$  is never in  $\mathcal{G}_2$ , by directly finding all limit points of  $H^{\sigma}$  under uniform limits. Let us call the limit points of  $H^{\sigma}$  not belonging to  $H^{\sigma}$  nontrivial.
Theorem 9 (E). For a non-constant  $\sigma$ , the only nontrivial limit points of the family  $H^{\sigma}$  are  $a_0 + a_r x^r, a\sigma(bx) + c$ , and  $c \sin(a_0 + a_r x^r)$ , where  $r = \arg \min(m > 0 : \frac{d^m \sigma}{dx^m}(0) \neq 0)$  and  $a_0, a_r, a, b, c \in \mathbb{R}$ . In particular,  $H^{\sigma} \notin \mathcal{G}_2$ .
The families  $H^{\sigma}$  with non-polynomial  $\sigma$  thus provide examples of elements of the class  $\mathcal{G}_1 \setminus \mathcal{G}_2$  of finitely- but not globally-universal formulas. Consider now a more complex family  $H_N^{(5)}$  consisting of the functions
$$
f (x) = c \sin (g (x)) + h, \quad g \in H _ {N} ^ {(2)}, \tag {12}
$$
where  $H_N^{(2)}$  consists of linear combinations of  $N$  sine waves as defined earlier in Eq. (4). One can interpret the family  $H_N^{(5)}$  as a two-hidden-layer neural network with the sin activation function,  $N$  neurons in the first hidden layer, and a single neuron in the second hidden layer. Using the same approach of examining the limit points and invoking additional topological arguments, we prove that the family  $H_N^{(5)}$  belongs to the same class as  $H^{\sigma}$ :
Theorem 10 (F). For any  $N \geq 1$ ,  $H_N^{(5)} \in \mathcal{G}_1 \setminus \mathcal{G}_2$ .
It appears that the class  $\mathcal{G}_1 \setminus \mathcal{G}_2$  is quite generic – we conjecture that an analog of Theorem 10 holds for any multi-layer neural network with the sin activation and a bounded number of neurons. However, proving such a general result by explicitly describing the limit points as in Theorems 9, 10 is difficult. We are not aware of simple conditions allowing to separate  $\mathcal{G}_1 \setminus \mathcal{G}_2$  from  $\mathcal{G}_2$ .
In this context, let us discuss the structure of universal formulas mentioned in the Introduction. Boshernitzan's formula (1) includes the factor  $\frac{bd}{1 + d^2 - \cos(bt)}$  which is of the form (8) and hence in  $\mathcal{G}_1$ , and the transcendental composition  $\cos(e^t)$ . This agrees with our results showing that compositions of sines and non-polynomial functions are essential for universal formulas. However, Boshernitzan's formula also includes integration, which effectively serves to produce an approximation to a piecewise constant function. The neural network in Figure 1 and the formulas of [13] combine sines with arcsines, which serves to approximate periodic piecewise linear or piecewise constant functions. It seems that approximate piecewise constant functions are an essential element of existing universal
formulas, and so one can conjecture that arcsine or a related inverse trigonometric function is a necessary component of an elementary universal formula without integration.
In the fairly different context of recursive decidability, there is a remarkably similar sharp difference between polynomial-sine and more complex compositions [12]. Let  $S_{1}$  denote the ring generated by the integers and the expressions  $x, \sin x^{n}$  and  $\sin (x\sin x^n)(n = 1,2,\ldots)$ . Let also  $S_{2}$  denote the ring generated by the integers and the expressions  $\sin x^{n}$  and  $\cos x^{n}(n = 1,2,\dots)$ . Then the predicate that there exists a real  $x$  such that  $f(x) > 0$  is undecidable for  $f\in S_1$ , but decidable for  $f\in S_{2}$ .
The expressiveness hierarchy. To analyze the necessary features of the structure of universal formulas, we have examined the chain of expressiveness classes  $\mathcal{G}_0 \supset \mathcal{G}_1 \supset \mathcal{G}_2$  corresponding, respectively, to the families having the infinite VC-dimension, general approximability on all finite sets, and general global uniform approximability. Membership in the class  $\mathcal{G}_0$  can be conveniently studied using the theory of Pfaffian functions, while, as we have shown, membership in the class  $\mathcal{G}_1$  can be conveniently studied using algebraic methods. In particular, our results suggest that globally universal formulas need to involve suitable compositions of non-polynomial and sine operations.
Transcendental structure of universal formulas. Our main result in Section 5 shows that combining a single layer of exponential and sine operations with polynomials and algebraic functions has a fairly weak expressiveness, in the sense that such formulas of bounded complexity are constrained by polynomial relations and do not even have general approximability on finite sets. In this family, the requirement on the operations preceding the exponential/sine layer is much stronger (polynomial) than the requirement on the subsequent operations (algebraic). As Theorem 8 shows, any non-polynomial operation preceding the sine operation and combined with linear operations restores the general finite approximability.
General methods. All results presented in this paper that rule out global universality are essentially obtained by one of the three general methods: Khovansky's analytic theory of Pfaffian functions (Theorem 1), finding algebraic constraints (Theorems 3, 5, 6), and direct computations of limit functions (Theorems 4, 9, 10). It appears that all these methods are insufficient to conveniently separate the finite approximability class  $\mathcal{G}_1 \setminus \mathcal{G}_2$  from the global approximability class  $\mathcal{G}_2$ . Only the last of the three methods seems useful for that, but it seems hard to find the full set of limit points for more complex families  $H$  similarly to how we did that for the linear combinations of sines (Theorems 4) and the families  $H^\sigma, H_N^{(5)}$  (Theorems 9, 10). It would be interesting to have a general method suitable for separating  $\mathcal{G}_1 \setminus \mathcal{G}_2$  from  $\mathcal{G}_2$ .
Limit points. Theorems 4 and 9 show that the nontrivial limit points of the families  $H_N^{(2)}$  and  $H^\sigma$  are some sort of resonances, reflecting some degeneracies of the model. The set of nontrivial limit points is small and explicit in these two cases. However, in a universal formula this set is the whole  $C([0,1])$ . It would be interesting to see how general is this dichotomy, e.g. whether the set of nontrivial limit points of some formula can be a small but not explicitly describable subset of  $C([0,1])$ .
Neural networks. In Section 6 we described a general family of neural networks lying outside the class  $\mathcal{G}_1$  and so non-universal. These networks can have a single layer of transcendental neurons (with activations such as sin, tanh,  $e^{-x^2}$ ) as well as some layers with piecewise-polynomial activations. We have shown that though the resulting functions are only piecewise analytic, they still admit polynomial constraints. It remains, however, an open question whether networks with multiple transcendental layers involving the sine activation can be universal. Theorem 8 shows that such networks belong to  $\mathcal{G}_1$ , while Theorem 10 shows that the two-hidden-layer sine network of bounded complexity and with a single neuron in the second hidden layer belongs to  $\mathcal{G}_1 \setminus \mathcal{G}_2$ . We conjecture that, without arcsin or other special functions that help create periodic piecewise-linear dependencies, general sin networks of fixed complexity belong to  $\mathcal{G}_1 \setminus \mathcal{G}_2$ .
The author thanks the anonymous reviewers for several useful suggestions. Research was supported by Russian Science Foundation, grant 21-11-00373.
[1] Michael Boshernitzan. Universal formulae and universal differential equations. Annals of mathematics, 124(2):273-291, 1986.
[2] R Creighton Buck. The solutions to a smooth PDE can be dense in C (I). Technical report, WISCONSIN UNIV-MADISON MATHEMATICS RESEARCH CENTER, 1980.
[3] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
[4] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
[5] Andrei Gabrielov and Nicolai Vorobjov. Complexity of computations with pfaffian and noetherian functions. Normal forms, bifurcations and finiteness problems in differential equations, 137:211-250, 2004.
[6] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315-323, 2011.
[7] Steven M Gonek and Hugh L Montgomery. Kronecker's approximation theorem. Indagationes Mathematicae, 27(2):506-523, 2016.
[8] Erik Christian Hansen, Yonathan Stone, and Jesse Wolfson. Alexander Ostrowski's "On Dirichlet Series and Algebraic Differential Equations". arXiv preprint arXiv:2211.02088, 2022.
[9] A. G. Khovanskii. *Fewnomials*. Vol. 88 of Translations of Mathematical Monographs. American Mathematical Society, 1991.
[10] Andrei Kolmogorov. On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition. In Doklady Akademii Nauk, volume 114, pages 953-956. Russian Academy of Sciences, 1957.
[11] Leopold Kronecker. Näherungsweise ganzzahlige Aufösung linearer Gleichungen. Monats. Königl. Preuss. Akad. Wiss. Berlin, pages 1179-1193, 1271-1299, 1884.
[12] Miklós Laczkovich. The removal of  $\pi$  from some undecidable problems involving elementary functions. Proceedings of the American Mathematical Society, 131(7):2235-2240, 2003.
[13] Miklós Laczkovich and Imre Z Ruzsa. Elementary and integral-elementary functions. Illinois Journal of Mathematics, 44(1):161-182, 2000.
[14] Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks, 6(6):861-867, 1993.
[15] Vitaly Maiorov and Allan Pinkus. Lower bounds for approximation by mlp neural networks. Neurocomputing, 25(1-3):81-91, 1999.
[16] Eliakim Hastings Moore. Concerning transcendentially transcendental functions. Mathematische Annalen, 48(1-2):49-74, 1896.
[17] Alexander Ostrowski. Über dirichlatische Reihen und algebraische Differentialgleichungen. Mathematische Zeitschrift, 8(3):241-298, 1920.
[18] Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143-195, 1999.
[19] Amaury Pouly and Olivier Bournez. A universal ordinary differential equation. Logical Methods in Computer Science, 16, 2020.
[20] Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
[21] Lee A Rubel. A universal differential equation. Bulletin (new series) of the american mathematical society, 4(3):345-349, 1981.
[22] Lee A Rubel. The extended analog computer. Advances in Applied Mathematics, 14(1):39-50, 1993.
[23] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
[24] David R So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Primer: Searching for efficient transformers for language modeling. arXiv preprint arXiv:2109.08668, 2021.
[25] Eduardo D Sontag. Shattering all sets of 'k'points in "general position" requires (k—1)/2 parameters. Neural Computation, 9(2):337–348, 1997.
[26] Terence Tao. A quantitative ergodic theory proof of Szemerédi's theorem. arXiv preprint math/0405251, 2004.
[27] Bartel Leendert Van der Waerden. Beweis einer baudetschen Vermutung. Nieuw Arch. Wiskunde, 15:212-216, 1927.
[28] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
[29] Dmitry Yarotsky. Elementary superexpressive activations. In International Conference on Machine Learning, pages 11932-11940. PMLR, 2021.
[30] Thierry Zell. Betti numbers of semi-pfaffian sets. Journal of Pure and Applied Algebra, 139(1-3):323-338, 1999.
1 Introduction 1
2 The hierarchy of expressiveness classes 3
3 Pfaffian functions 4
4 Linear combinations of sines 5
5 Polynomials-exponentially-algebraic expressions 6
6 Branching expressions and neural networks 7
7 Compositions of sines and non-polynomial functions 8
8 Discussion 9
References 10
A Proof of Theorem 4 12
B Proof of Theorem 5 21
C Proof of Theorem 6 24
D Proof of Theorem 8 25
E Proof of Theorem 9 25
F Proof of Theorem 10 26
A Proof of Theorem 4
Sufficiency (each function (6) is a limit point of  $H_N^{(2)}$ ). First observe that each resonant component  $x^m\sin (\omega x + h)$  is a uniform limit of  $m + 1$  sine waves linearly combined to match the discretized derivative  $\frac{d^m}{d\omega^m}\sin (\omega x + h - \pi m / 2)$ :
$$
\begin{array}{l} x ^ {m} \sin (\omega x + h) = \frac {d ^ {s}}{d \omega^ {m}} \sin (\omega x + h - \pi m / 2) (13) \\ = \lim  _ {\Delta \omega \rightarrow 0} (\Delta \omega) ^ {- m} \sum_ {n = 0} ^ {m} (- 1) ^ {m - n} \binom {m} {n} \sin ((\omega + n \Delta \omega) x + h - \pi m / 2). (14) \\ \end{array}
$$
Note also that any finite linear combination  $\sum_{m=0}^{M-1} c_m \sin(\omega x + h_m)$  can be written as a single sine wave  $c \sin(\omega x + h)$  with suitable amplitude  $c$  and phase  $h$ . It follows that we can obtain any linear combination  $\sum_{m=0}^{M-1} c_m x^m \sin(\omega x + h_m)$  as a uniform  $\Delta \omega \to 0$  limit of linear combinations
$$
\sum_ {m = 0} ^ {M - 1} c _ {m, \Delta \omega} \sin ((\omega + m \Delta \omega) x + h _ {m, \Delta \omega}) \tag {15}
$$
with suitable values of  $c_{m,\Delta \omega}$  and  $h_{m,\Delta \omega}$ . This shows in turn that the second component of expression (6),
$$
\sum_ {k = 1} ^ {K} \sum_ {m = 0} ^ {M _ {k} - 1} c _ {k m} x ^ {m} \sin \left(\omega_ {k} x + h _ {k m}\right), \tag {16}
$$
can be obtained by limits of linear combinations of  $\sum_{k=1}^{K} M_k$  sine waves with suitable phases and frequencies.
It remains to show that the first component of expression (6),  $\sum_{m=0}^{2M_0-1} c_{0m} x^m$ , can be obtained as a limit of linear combinations of  $M_0$  sine waves. Observe that for any constant  $x_0$
$$
\left(x - x _ {0}\right) ^ {2 M _ {0} - 1} = \lim  _ {\Delta \omega \rightarrow 0} (\Delta \omega) ^ {2 M _ {0} - 1} \sin^ {2 M _ {0} - 1} (\Delta \omega x - \Delta \omega x _ {0}). \tag {17}
$$
Since  $2M_0 - 1$  is odd,  $\sin^{2M_0 - 1}(\Delta \omega x - \Delta \omega x_0)$  is a linear combination of  $M_0$  sine waves with frequencies  $(2s - 1)\Delta \omega$ ,  $s = 1,\dots ,M_0$  (and some phases). By a previously given argument, this implies that any linear combination of monomials  $(x - x_0)^{2M_0 - 1}$  can also be obtained as a limit of some linear combinations of  $M_0$  sine waves with frequencies  $(2s - 1)\Delta \omega$ ,  $s = 1,\dots ,M_0$ . But the monomials  $(x - x_0)^{2M_0 - 1}$  linearly span the space of polynomials of degree  $\leq 2M_0 - 1$ . It follows that any such polynomial can be obtained as a limit of linear combinations of  $M_0$  sine waves, as claimed.
Necessity (each limit point of  $H_N^{(2)}$  has the from (6)). We give two alternative proofs.