Neural Oscillators are Universal
Authors: Samuel Lanthaler
T. Konstantin Rusch
Siddhartha Mishra
Venue: NeurIPS 2023 poster (2023)
Keywords: neural ODE
universal approximation
oscillator

Abstract: Coupled oscillators are being increasingly used as the basis of machine learning (ML) architectures, for instance in sequence modeling, graph representation learning and in physical neural networks that are used in analog ML devices. We introduce an abstract class of *neural oscillators* that encompasses these architectures and prove that neural oscillators are universal, i.e, they can approximate any continuous and casual operator mapping between time-varying functions, to desired accuracy. This universality result provides theoretical justification for the use of oscillator based ML systems. The proof builds on a fundamental result of independent interest, which shows that a combination of forced harmonic oscillators with a nonlinear read-out suffices to approximate the underlying operators.

Neural Oscillators are Universal
Samuel Lanthaler California Institute Technology s slanth@caltech.edd
T. Konstantin Rusch ETH Zurich
Siddhartha Mishra ETH Zurich
Abstract
Coupled oscillators are being increasingly used as the basis of machine learn- ing (ML) architectures, for instance in sequence modeling, graph representation learning and in physical neural networks that are used in analog ML devices. We introduce an abstract class of neural oscillators that encompasses these architec- tures and prove that neural oscillators are universal, i.e, they can approximate any continuous and casual operator mapping between time-varying functions, desired accuracy. This universality result provides theoretical justification for the use oscillator based ML systems. The proof builds on fundamental result of dependent interest, which shows that combination of forced harmonic oscillators with a nonlinear read-out suffices to approximate the underlying operators.
Introduction
Oscillators are ubiquitous in the sciences and engineering 12 30 Prototypical examples include pendulums in mechanies, feedback and relaxation oscillators in electronics, business cycles in economics and heart beat and circadian rhythms in biology. Particularly relevant to our context is the fact that the neurons in our brain be thought of as oscillators on account of the periodic spiking and firing of the action potential [29 I1]. Consequently. functional brain circuits such as cortical columns are being increasingly analyzed in terms of networks of coupled oscillators (29).
Given this wide prevalence of (networks of) oscillators in nature and man-made devices, it is not surprising that oscillators have inspired various machine learning architectures in recent years. Prominent examples include the CoRNN (27] and UnICORNN (28 recurrent neural networks for sequence modeling. CoRNN is based
network of coupled, forced and damped oscillators, whereas UnICORNN is a multi-layer sequence model that stacks networks of independent undamped oscillators hidden layers within an RNN. Both these architectures were rigorously shown mitigate the exploding and vanishing gradient problem (20] that plagues RNNs. Hence, both CoRNN and UnICORNN performed very well on sequence learning tasks with long-term dependencies. Another example of the use of oscillators in machine learning is provided by GraphCON (26). a framework for designing graph neural networks (GNNs) 3. that based on coupled oscillators. GraphCON was also shown to ameliorate the oversmoothing problem (55 and allow for the deployment of multi-layer deep GNNs. Other examples include Second Order Neural ODEs (SONODES) 19,, which can be interpreted as oscillatory neural ODEs, locally coupled oscillatory recurrent networks (LocoRNN) 17]. and Oscillatory Fourier Neural Network (O-FNN) [3]
Another avenue where ML models based on oscillators arise is that of physical neural networks (PNNs) 35 i.e., physical devices that perform machine learning on analog (beyond digital) systems. Such analog systems have been proposed as alternatives or accelerators to the current paradigm of machine learning on conventional electronies, allowing us to significantly reduce the prohibitive energy costs of training state-of-the-an ML models. In [35]. the authors propose variety of physical neural networks which include a mechanical network of multi-mode oscillations on plate and electronic circuits of oscillators well as a network of nonlinear oscillators. Coupled with a novel physics aware training (PAT) algorithm, the authors of 33 demonstrated that their
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

nonlinear oscillatory PNN achieved very good performance on challenging benchmarks such as Bushion-MNISS" [36]. Moreover, other oscillatory systems such as coupled lasers and spintronic nano-oscillators have also been proposed as possible PNNs, see 33 as an example of the use of
thermally coupled vanadium dioxide oscillators for image recognition and [24 32
for the use of spin-torque nano-oscillators for speech recognition and for neuromorphic computing, respectively.
What is the rationale behind the successful use of (networks of) oscillators in many different contexts in machine learning? The authors of [27] attribute it to the inherent stability of oscillatory dynamics, as the state (and its gradients) of an oscillatory system remain within reasonable bounds throughout the time-evolution of the system. However, this is at best a partial explanation, as it does not demonstrate why oscillatory dynamics can learn (approximate) mappings between inputs and outputs rather than bias the learned states towards oscillatory functions. As an example, consider the problem of classification of MNIST [88] (or Fashion-MNISTT images. is completely unclear if the inputs (vectors of pixel values), outputs (class probabilities) and the underlying mapping possess any (periodic) oscillatory structure. Consequently, how can oscillatory RNNs (such an CoRNN and UnICORNN) or network of oscillatory PNNs learn the underlying mapping?
Our main aim in this paper is to provide an answer to this very question on the ability of neural networks, based on oscillators, to express (to approximate) arbitrary mappings. To this end.
We introduce an abstract framework of neural oscillators that encompasses both sequence models such as CoRNN and UnICORNN, as well as variants of physical neural networks as the ones proposed in [35] These neural oscillators are defined in terms of second-order versions of neural ODEs and combine nonlinear dynamics with a linear read-oul.
We prove a Universality theorem for neural oscillators by showing that they can approximate, to any given tolerance, continuous operators between appropriate function spaces.
Our proof of universality is based on a novel theoretical result of independent interest, termed the fundamental Lemma, which implies that suitable combination of linear oscillator dynamics with nonlinear read-out suffices for universality.
Such universality results, of traditional neural networks (such as multi-layer perceptrons and convolutional neural networks). Hence, our universality result establishes firm mathematical foundation for the deployment of neural networks, based on oscillators, in myriad applications. Moreover, our constructive proof provides insight into how networks of oscillators can approximate a large class of mappings.
and references therein, have underpinned the widespread use
2 Neural Oscillators
General Form of Neural Oscillators. Given u: [0, R& as an input signal, for any final time T E R4, we consider the following system of neural ODEs for the evolution of dynamic hidden variables y E R,,, coupled to a linear read-out to yield the output 2 E IR,
i(t) (Wy(t) Vu(() + b),
(2.
y(0)= = i(()= = 0,
(2.1b)
2(t) Ay(t) +c.
(2.1c)
Equation (2.1) defines an input-/output-nn maattm/puuutm ult) + 2(t), with dime-dependent output [0, + Rº. Specification of this system requires choice of the hidden variable dimension m and the activation function a. The resulting mapping + 2 depends on tunable weight matrices W E Rm V E RRmxp, A E Rexm and bias vectors E RT c R2. For simplicity of the exposition, we consider only activation functions C (RE, with (0) = 0 and o' 2((0) = 1, such as tanh or sin, although more general activation functions can be readily considered. This general second-order neural ODE system (2.1 will be referred to neural oscillator.

---

structure for W: Multi-layer neural oscillators. As a special case of neural oscillators, we consider the following much sparser class of second-order neural ODEs,
y (t) u(t). (t) a (w' V''()) +0)+ (L=1....,1))
(2.2a)
(2.2b)
(0) - j0(0)= =
(2.2c)
x(t) = (t) cc.
(2.2d)
contrast to the general neural oscillator 2.I). the above multi-layer neural oscillator (2.2 defines a hierarchical structure; The solution v/ IR Th at level solves second-order ODE with driving force yf= and the lowest level. 1 = u, is the input signal. Here, the layer dimensions mi..... my can vary across layers, the weights w E Rm V E x is a weight matrix, and given by vectors, with @ componentwise multiplication, R' the bias. Given the result of the final layer, y", the output signal is finally obtained by affine output layer =(t) (t" + In the multi- layer neural oscillator, the matrices VL, A and vectors wf, bf and represent the trainable hidden parameters. The system (.2) is special case of (2.I), since can be written in the form (2.I), with y 1 y y--1
= [b_. bb..... b²j² and (upper-diagonal
diag(w")
VL
diag(w
VL
W
V
(2.3)
O
0
diag w
V2 diag(w') (w)
Given the diako-da structure the underlying weight matrices, it is clear that the multi-layer neural oscillator (2.2) is much sparser representation of the general neural operator (2.1). Moreover, one can observe from the structure of the neural ODE (2.2) that within each layer, the individual neurons causally independent of each other.
Assuming that w # 0. for all < S my and all E < we further highlight that the multi layti-layer neural oscillator (..2) is Hamiltonian system,
OH Oij
0H Oy
8 =
(2.4)
with the layer-wise time-dependent Hamiltonian,
H jf
Σ
@ (w (V* K(),
(2.5)
with ā being the antiderivative of o,, and ||x| (x, x) denoting the Euclidean norm of the vector E RT and (---) the corresponding inner product. Hence, any symplectic discretization of the multi-layer neural oscillator (2.2) will result
fully reversible model, which can first be leveraged in the context of normalizing flows [23] and second leads to a memory-efficient training, the intermediate states (i.e., (ta). j (to), ye (t1)), (e((t),..............))..)...))) (e(t je((t) ), for some time discretization ty, ti, , ty during the backward pass. This potentially leads drastic memory saving O(N) during training.
of length N) do not need to be stored and can be reconstructed
2.1 Examples of Neural Oscillators
q and we set W = -w², for some w E R, V 1, 6 0 and the activation function be identity (xx) x. In this case, the neural ODE (2.l) reduces to the ODE modeling the dynamics of forced simple harmonic oscillator [12] of the form,
(Forced) harmonic oscillator. Let
i = - + y(0) = (0) =
(2.6)
Here, is the displacement of the oscillator, w the frequency of oscillation and u is forcing term that forces the motion of the oscillator. Note that (2.)) also a particular example of the multi-layer
oscillator 2.2 with L = 1.

---

This simple example provides justification for our terminology of neural oscillators, as in general, the hidden state y can be thought as the vector of displacements of m-coupled oscillators, which are coupled together through the weight matrix W and are forced through a forcing term u, whose effect modulated via V and a bias term b. The nonlinear activation function mediates possible nonlinear feedback to the system on account of large displacements.
CoRNN. The Coupled oscillatory RNN (CoRNN) architecture 27 given by the neural ODE:
j = (Wy + Wý + Vu + b) YY ey.
We can recover the neural oscillator (2.1 as a special case of CoRNN by setting W = 0, = =0 0: thus, a universality theorem for neural oscillators immediately implies corresponding universality result for the CoRNN architecture.
UnICORNN. The Undamped Independent Controlled Oscillatory RNN (UnICORNN) architecture of 28 eqn. 1] recovers the multi-layer neural oscillator (2.2) in the case where the fundamental frequencies of UnICORNN are automatically determined inside the weight matrix W in (2.1
Nonlinear oscillatory PNN of 35
In 35 SM, Sect. 4.A], the authors propose an analog machine learning device that simulates a network of nonlinear oscillators, for instance realized through coupled pendula. The resulting mathematical model is the so-called simplified Kontel-Kontorova model given by the ODE system,
MÒ - K sin C sin(()) +
where 0 = (fI..... ON) is the vector of angles across all coupled pendula, M = diag(u' ..... ) diagonal mass matrix, an external forcing, K diag(k' ,.... ,,)) the "spring constant" for pendula, given by = u'g/E with the pendulum length and the gravitational acceleration, and where C' CT is symmetric matrix, with
Cu =
- so that (C's)e = Σ Cee sin(())), E'gé
(2.7)
which quantifies the coupling between different pendula. We note that this simplified Frenkel-
Kontorova system also model other coupled nonlinear oscillators, such coupled lasers spintronic oscillators [35].
We can bring the above system into more familiar form by introducing the variable y according to the relationship Py = for a matrix P. Substitution of this ansatz then yields M Pij = -(K- C) sin(Py) f: choosing P = M M- (K C), find
ÿ = sin A (K + C)v) f
(2.8)
which can be written in the form j = o w) + for a = - -si(.)))) and W = M- + C). We next make particular choice of C, by choosing it in block-matrix form
C
0
0
CL.
1
C==
0
0
C
221
C
C2T
7
and with corresponding mass matrix M block-matrix form M diag(u= I, then with p² : >//pp= we have
I²I
C²/p""
0
CL. /1--1p2-1
p
:--C:= :=
0
p-1
C2
02/12
1'1

---

With the intent of ordering the masses of different layers, such that /²/p²=1 ~ e is small, we now introduce an ordering parameter > O. And we consider the following ordering 7 C pe ~ e. Assuming this ordering, it follows that p², C² masses introduces an effective me- coupling. making
fer O(1,, and C E O(e). This ordering of the
pTI VL
0
p p
I
Via
C
+ +
0
0
V2
1'I
upper triangular, up to small terms of order e. We note that the diagonal entries p² in M are determined by the off-diagonal terms through the identity (2.7. The additional degrees freedom in the (diagonal) K -matiix in (2.8) can be used to tune the diagonal weights of the resulting weight matrix W M (K- + C.).
Thus, physical systems such as the Trankel-Kontorow system of nonlinear oscillators can be approxi- mated (to leading order) by multi-layer systems of the form
f' = (w" @ of + -
(2.9)
with J external forcing, representing tunable linear transformation of the external input the system. The only formal difference between (2.9) and (2.2) is (i) the absence of a bias term in (.9 and (ii) the fact that the external foreing appears outside of the nonlinear activation function o in (2.9). A bias term could readily be introduced by measuring the angles represented by y in a suitably shifted reference frame; physically, this corresponds to tuning the initial position y (0) of the pendula, with (0) also serving as the reference value. Furthermore, in our proof of universality for (2.2) it makes very little difference whether the external forcing f is applied inside the activation function, in (2.2) resp. (2. or outside in (2.9); indeed, the first layer in our proof of universality will in fact approximate the linearized dynamics of (2.2p), i.e. forced harmonic oscillator (2.6).
Consequently, a universality result for the multi-layer neural oscillator (2.2) also implies universality of variants of nonlinear oscillator-based physical neural networks, such as those considered in 335
3 Universality of Neural Oscillators
In this section, we state and sketch the proof for our main result regarding the universality of neural oscillators (2.1) or, more specifically, multi-layer oscillators (2.2). To this end, we start with some mathematical preliminaries to the stage for the main theorem.
3.1 Setting
Input signal. We want to approximate operators + ((u), where u u(() is a time- dependent input signal over a time-interval t E TJ. and (()((t) is time-dependent output signal. We will assume that the input signal t + u(t) continuous, and that u(O) 0. To this end, we introduce the space
Co( T]; R") = (u: [0 - R t + u(t) is continuous and u())= = 0}.
We will assume that the underlying operator defines mapping + Co T}; IP) - Co, (): R9).
The approximation we discuss in this work are based on oscillatory systems starting from rest. These oscillators are foreed by the input signal u. For such systems the assumption that u(0) 0 is necessary, because the oscillator starting from rest takes a (arbitrarily small) time-interval to synchronize with the input signal (to "warm up"); If u(0) # 0, then the oscillator cannot accurately approximate the output during this warm-up phase. This intuitive fact is also implicit in our proofs. We will provide further comment this issue in Remark 3.2 below.
Operators of interest. We consider the approximation of an operator Co R): RP) +
Co((0, T}: IR9), mapping continuous input signal u(() to continuous output signal (t) We will restrict attention to the uniform approximation of over compact set of input functions K Co []; RP). We will assume that & satisfies the following properties:
5

---

+ is causal: For any t E [0. u, E Co R") are two ipput signals, such that
u u|lo.j] vlfo.e]. then (()) (t)) ie. the value of (t) time does depend on future values {u(r)] t}.
c is continuous operator
c (C. 7]; R"), [1=>) - (C.((O,T)):)) II,,),
with respect to the -orm on the input-/outpu--signal..
Note that the class of Continuous and Causal operators very general and natural in the contexts of mapping between sequence spaces or time-varying function spaces, see 7 6 and references therein.
3.2 Universal approximation Theorem
The universality of neural oscillators is summarized in the following theorem:
Theorem 3.1. [Universality of the multi-layer neural oscillator] Let + Co( R?) 7 Co(,O, R)) be a causal and continuous operator. Let K Co((O, R?) be compact. Then for any 0, there exist hyperparameters I., mm.....;mu. M,, weights w E R V IR me A E IRexmz and bias vectors b' E R= E R², for = I,...,L, such that the output 2: [0, T R2 of the multi-layer neural oscillator (.. satisfies
sup ((u)(t) 2(t) S e, YuEK. tE(o,T]
important observe that the sparse, independent multi-layer neural oscillator (2.2) suffices for universality in the considered class. Thus, there is no need to consider the wider class of neural oscillators 1..)..a.... at least in this respect. We remark in passing that Theorem 3.1 jimmediately implies another universality result for neural oscillators, showing that they can also be used to approximate arbitrary continuous functions IP Ra. This extension is explained in detail in SM
Remark 3.2.
We note that the theorem be readily extended to remove the requirement
(()) = and (0) 0. To this end, let C[( ; - C(O,,) : R9) be an operator
between spaces of continuous functions, - (fu) on [0. T]. Fix > O, and extend any input
function u [0,T] + Rp to a function Eu)) E Co to, T): (P),) by
(l++t) (()),
t E [- to 0),
((w)(I)= =
to
utt),
t [0,
Our proof of Theorem 3.1 can readily be used to show that the oscillator system with forcing {(u). and initialized time to can uniformly approximate (uu over the entire time interval [0, T without requiring that u(0) = 0, or (0) = O. In this case, the initial time interval [-to. provides the required "warm-up phase" for the neural oscillator.
Remark 3.3. The required compactness property of the set of input signals K Co T]: RP) in Theorem 3.1 is implied by suitable smoothness constraint; examples include uniform bounds on the Lipschitz norm, Hölder norms, uniform bounds on higher-order derivatives, under the assumption that the input signals are uniformly bounded and limited. The proposed compactness assumption subsumes all of these smoothness constraints, resulting in widely applicable universal approximation theorem.
Remark 3.4. In practice, neural ODEs such as (2.2 need to be discretized via suitable numerical schemes. As examples, CoRNN and UnICORNN were implemented in (27 and (8]. respectively, with implicit-explicit time discretizations. Nevertheless, universality also applies for such discretiza- tions as long as the time-step is small enough, as the underlying discretization is going to be sufficiently accurate approximation of (2.2) and Theorem 3.1 can be used for showing universality of the discretized version of the multi-layer neural oscillator (..2.

---

3.3 Outline of the Proof
In the following, we outline the proof of the universality Theorem while postponing the technical details to the SM. For given tolerance e, we will explicitly construct the weights and biases of the multi-layer neural oscillator (2.2) such that the underlying operator can approxi- mated within the given tolerance. This construction takes
Figure 1: Illustration of the universal 3- layer neural oscillator architecture con- structed in the proof of Theorem 3.1
imput
place in the following steps:
(Forced) Harmonic Oscillators compute a time- windowed sine transform. Recall that the forced har- monic oscillator (2.6) is the simplest example a neural oscillator 2. p. The following lemma, proved by direct calculation in SM shows that this forced harmonic oscillator actually computes a time-windowed variant of the sine transform at the corresponding frequency:
x((F)
eo forced linear ascillator
layer
eepor
layer
layer
Lemma 3.5. Assume that w * O. Then the solution of 2.6 is given by
y
1 / u(t - T) (iinww))er
oetgen
(())
(3.1)
layer
nowfineer
Given the last result, for function u, define its time- windowed sine transform as follows,
output
road out
Lu((w) = L. ult T) sin(wT) dr.
(3.2)
Lemma 3.5 shows that a forced harmonic oscillator computes
3.2 up to constant.
Approximation of causal operators from finite realizations of time-windowed sine transforms. The following novel result, termed the fundamental Lemma, shows that the time-windowed sine transform (3.2 composed with a suitable nonlinear function can approximate causal operators & to desired accuracy; as consequence, one can conclude that forced harmonic oscillators combined with nonlinear read-out defines a universal architecture in the sense of Theorem 3.1
Lemma 3.6 (Fundamental Lemma). Let K Co [0, 1): Co ([0, T"; J29 be a causal and continuous operator, with K c ((O, Т}: 12)) compact. Then for any E >> 0, there exists N E N, frequencies .,,....,WN and a continuous mapping v R x [0, T /4] - R", such that (t) - V (Cru(w)), .... ... Lru(wN); 8 f/4) S
for u E K and t e [).
The function Tr defined in Lemma 3.6 can be interpreted as fnite-dimensional encoding of the operator tb The main insight of the fundamental Lemma B.6 that the & can effectively be replaced by finite-dimensional function V, making the approximation by neural oscillators more tractable.
operator
The proof of the fundamental Lemma 3.6 detailed in SM B.2 is based on first showing that any continuous function can be reconstructed to desired accuracy, in terms of realizations of its time- windowed sine transform (3.2) finitely many frequencies Wi, WN (see SM Lemma B.I). Then, we leverage the continuity the underlying operator & to approximate it with a finite-dimensional function V, which takes the time-windowed sine transforms its arguments.
Given these two results, we can discern clear strategy to prove the universality Theorem 3.1 First, we will show that general nonlinear form of the neural oscillator (2.2) can also compute the time-windowed sine transform at arbitrary frequencies. Then, these outputs need be processed in order to apply the fundamental Lemma 3.6 and approximate the underlying operator &. To this end, we will also approximate the function V (mapping finie-dimeeniioal inputs finte--dimensional outputs) by oscillatory layers. The concrete steps in this strategy are outlined below.

---

Nonlinear Oscillators approximate the time-windowed sine transform. The building block of multi-layer neural oscillators (2.2) the nonlinear oscillator of the form,
( + + Vu b).
(3.3)
In the following Lemma (proved in SM B.3 we show that even for a nonlinear activation function such as tanh or sin, the nonlinear oscillator 3.3 can approximate the time- windowedd sine transform.
Lemma 3.7. Fix w + 0. Assume that (0) = 0, o'()) 1. For any > 0, there exist w, V, A e R,
such that the nonlinear oscillator (33). initialized at y(0) - =(0) = 0, has output
Ay(( L.n(w)) s,
Vu K, E [0,
with Lru(w) being the time-windowed sine transform
3.2
Coupled Nonlinear Oscillators approximate time-delays. The next step in the proof to show that coupled oscillators can approximate time-delays in the continuous input signal. This fact will be of crucial importance in subsequent arguments. We have the following Lemma (proved in SM SM B.4).
Lemma 3.8. Let K c Co( RR) be compact subset. For every e > 0, and At 0. there exist m E N. w E R V E gmxp, E R and A E Roxm, such that the oscillator 3.3 initialized at (0) (() 0, has output
sup lu(t At) Av(t)l VuEK, tE (0,77
where u(t) is extended to negative values & by zero.
Two-layer neural oscillators approximate neural networks pointwise. As in the strategy outlined above, the final ingredient in our proof of the universality theorem to show that neural oscillators can approximate continuous functions, such as the in the fundamental lemma 3.6] to desired accuracy. To this end, will first show that neural oscillators networks (perceptrons) and then use the universality of neural networks in the class of continuous functions prove the desired result. The main difficulty here that, due the underlying oscillatory dynamics, it is not clear whether the output of even shallow neural network can be reproduced by neural oscillators. The following lemma, visually represented by the bottom half of Figure guarantees this:
approximate general neural
Lemma 3.9. Let K Co((o, R") be compact. For matrices A and bias y, and any e > 0, there exists two-layer (I 2) oscillator 2.2). initialized (0) j (0) 0, l 1,2, such that
R²)
(Ay + c] (Au(())) + +7)l S Vu K. sup tE|0,T]
The proof, detailed in SM B.5 is constructive and the neural oscillator that we construct has two layers. The first layer just processes nonlinear input function through nonlinear oscillator and the second layer, approximates the second-derivativ, (in time) from time-delayed versions of the input signal that were constructed in Lemma 3.8
The affrre-contruuced ingredi- ents are combined SM Bo prove the universality theorem. this proof, we explicitly construct a three-layer neural oscillator (2.2 which approximates the underlying operator $. The first layer
Combining the ingredients to prove the universality theorem 3.1
follows the construction of Lemma 3.7 to approximate the time-windonn windowed sine transform (3.2) for as many frequencies as are required in the fundamental Lemma 3.6 The second- and third-layers imitate the construction of Lemma 3.9 to approximate a neural network (perception), which in turn by the universal approximation of neural networks, approximates the function V/ in Lemma 3.6 desired accuracy. Putting the network together leads to a three-layer oscillator that approximates the continuous and casual operator ②. This construction is depicted in Figure

---

4 Discussion
Machine learning architectures, based on networks of coupled oscillators, for instance sequence models such as CORNN 277 and UnICORNN 228, graph neural networks such as GraphCON 126 and increasingly, the so-called physical neural networks (PNNs) such as linear and nonlinear mechanical oscillators [35] and spintronic oscillators 124 32], are being increasingly used. A priori, it is unclear why ML systems based on oscillators can provide competitive performance on a variety of learning benchmarks, e.g. 22 28 26 35), rather than biasing their outputs towards oscillatory functions. In order to address these concerns about their expressivity, we have investigated the theoretical properties of machine learning systems based on oscillators. Our main aim was to answer fundamental question: "are coupled oscillator based machine learning architectures universal?". In other words, can these architectures, in principle, approximate a large class of input-output maps to desired accuracy.
To answer this fundamental question, we introduced an abstract framework of neural oscillators (2.1) and its particular instantiation, the multi-layer neural oscillators (2.2). This abstract class of second-order neural ODEs encompasses both sequence models such as CoRNN and UnICORNN, as well as very general and representative PNN, based on the so-called Trenkel-Kontrrow model. The main contribution of this paper was to prove the universality theorem [31 on the ability of multi-layer neural oscillators
(2.2) to approximate large class of operators, namely causal and continuous maps between spaces of continuous functions, to desired accuracy. Despite the fact that the considered neural oscillators possess very specific and constrained structure, not even encompassing general Hamiltonian systems, the approximated class of operators is nevertheless very general, including solution operators of general ordinary and even dima- differential equations.
The crucial theoretical ingredient in our proof was the fundamental Lemma 3.6 which implies that linear oscillator dynamics combined with pointwise nonlinear read-out suffices for universal operator approximation; our construction can correspondingly be thought of as a large number of linear processors, coupled with nonlinear readouts. This construction could have implications for other models such as structured state space models 9 the extension of our universality results to such models could be of great interest.
g 10 which follow a similar paradigm, and
Our universality result has many interesting implications. To start with, we rigorously prove that an ML architecture based on coupled oscillators can approximate a very large class of operators. This provides theoretical support to many widely used sequence models and PNNs based on oscillators. Moreover, given the generality of our result, we hope that such a universality result can spur the design of innovative architectures based on oscillators, particularly in the realm of analog devices as ML inference systems or ML accelerators (35]
It also instructive to lay out some of the limitations of the current article and point to avenues for future work. In this context, our setup currently only considers time-varying functions as inputs and outputs. Roughly speaking, these inputs and outputs have the structure of (infinite) sequences. However, a large class of learning tasks can be reconfigured to take sequential inputs and outputs. These include text (as evident from the tremendous success of large language models (2 ). DNA sequences, images [16), timeseries and (offline) reinforcement learning [15]. Nevertheless, a next step would be extend such universality results to inputs (and outputs) which have some spatial or relational structure, for instance by considering functions which have spatial dependence which are defined on graphs. On the other hand, the class of operators that we consider, i.e., casual and continuous, is not only natural in this setting but very general [7].).
Another limitation lies in the feed forward structure of the multi-layer neural oscillator 2.2 As mentioned before, most physical (and neurobiological) systems exhibit feedback loops between their constituents. However, this is not common ML systems. In fact, we had use a mass ordering in the Kontorrvv system of coupled pendula (2.8) in order to recast it in the form of the multi-layer neural oscillator (.2).. Such asymptotic ordering may not be possible for arbitrary physical neural networks. Exploring how such ordering mechanisms might arise in physical and biological systems in order to effectively give rise to feed forward system could be very interesting. One possible mechanism for coupled oscillators that can lead to hierarchical structure is that of synchronization [4 (1 and references therein. How such synchronization interacts with universality is a very interesting question and will serve an avenue for future work.

---

Finally, universality is arguably necessary but far from sufficient to analyze the performance of any ML architecture. Other aspects such as trainability and generalization are equally important, and we do not address these issues here. We do mention that trainability of oscillatory systems would profit from the fact that oscillatory dynamics is (gradient) stable and this formed the basis of the proofs of mitigation of the exploding and vanishing gradient problem for CoRNN in [27] and UnICORNN in
28 as well as GraphCON in [26]. Extending these results to the general second-order neural ODE 2.2 for instance through analysis of the associated adjoint system, is left for future work.
10

---

References
[1] A R. Barron. Universal approximation bounds for superpositions of a sigmoidal function.
IEEE Transcations on Information Theory, 39, 1993.
[2] O. M. Braun and Y.S. Kivshar. Nonlinear dynamics of the frenkel-kontoront model. Physics Reports, 306:1-108, 1998.
[3] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning:
Grids, groups, graphs, geodesics, and gauges. arXiv:2104. 13478, 2021.
[4] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, pages 6571-6583, 2018.
[5] G. Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303-314, Dec 1989.
[6] Lukas Gonon, Lyudmila Grigoryeva, and Juan-Pablo Ortega. Risk bounds for reservoir comput- ing. The Journal of Machine Learning Research, 21(1):9684-9744, 2020.
[7] Lyudmila Grigoryeva and Juan-Pablo Ortega. Echo state networks are universal. Neural Networks, 108:495 508, 2018.
[8] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Re. Hippo:recurrent memory with optimal polynomial projections. In Advances in Neural Information Processing Systems, pages 1474- 1487, 2020.
[9] A. Gu, K. Goel, and C. Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2021.
[10] Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Re. How to train your hippo: State space models with generalized orthogonal basis projections. In International Conference on Learning Representations, 2022.
[11] B-M. Gu, H. vanRijn, and W. K. Meck. Oscillatory multiplexing of neural population codes for interval timing and working memory. Neuroscience and Behaviorial reviews,, 48: 160-15 2015.
[12] Guckenheimer and P. Holmes. Nonlinear oscillations, dynamical systems, and bifurcations of vector fields. Springer Verlag, New York, 1990.
[13] Bing Han. Cheng Wang, and Kaushik Roy. Oscillatory fourier neural network: A compact and efficient architecture for sequential processing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36(6), pages 1838-6846 2022.
[14] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359 -366, 1989.
[15] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021.
[16] Andrej Karpathy and Li Fei-Fei. Deep sesual--emantic alignments for generating image descrip- tions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3128 3128-3137, 2015.
[17] T. Anderson Keller and Max Welling. Locally coupled oscillatory recurrent networks learn traveling waves and topographic organization. In Cosyne abstracts, 2023.
[18] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324,, 1998.
[19] Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski. and Pietro Liò. On second order behaviour in augmented neural odes. Advances in neural information processing systems, 33:5911-5922 2020.
[20] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Con- ference on Machine Learning, volume 28 of ICML. 13, page I11-1310-111-31111 JMLR.org, 2013.
11

---

[21] Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta numerica, 8(1):143-195, 1999.
[22] A. Radford, K. Narasimhan, T. Salimans, and I. Suktskever. Improving language understanding by generative pre-training. arXiv:, 2018.
[23] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference machine learning, pages 1530-1538, PMLR, 2015,
[24] M. Romera and et. al. Vowel recognition with four coupled spin-torque mano-oscillators. Nature, 588:230-234, 2018.
[25]
Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. arXiv:2303. 10993, 2023.
[26] T. Konstantin Rusch, Ben Chamberlain, James Rowbottom, Siddhartha Mishra, and Michael Bronstein. Graph-coupled oscillator networks. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18888-18909, PMLR, 2022.
[27] T. Konstantin Rusch and Siddhartha Mishra. Coupled oscillatory recurrent neural network (cornn): An accurate and (gradient) stable architecture for learning long time dependencies. In International Conference on Learning Representations, 2021.
[28] T. Konstantin Rusch and Siddhartha Mishra. Unicornn: A recurrent model for learning very long time dependencies. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9168-9178. PMLR, 2021.
[29] K. M. Stiefel and G. B. Ermentrout. Neurons as oscillators. Journal of Neurophysiology. 116:2950-2960, 2016.
[30] S. Strogatz. Nonlinear Dynamics and Chaos. Westview, Boulder CO, 2015.
[31] H. Strogatz. Exploring complex networks. Nature, 410:268-211 2766 2001.
[32] Torrejon and al. Neuromorphic computing with nanoscale spintronic oscillators. Nature, 547:428-431, 2017.
[33] A. Velichko, M. Belyaev, and Boriskov. A model of an oscillatory neural network with multilevel neurons for pattern recognition and computing. Electronics, 8, 2019.
[34] A. T. Winfree. Biological rhythms and the behavior of populations of coupled oscillators. Journal of Theoretical Biology, 16:15-42, 1967.
[35] Logan G Wright, Tatsuhiro Onodera, Martin M Stein, Tianyu Wang, Darren T Schachter, Zoey Hu, and Peter I. McMahon. Deep physical neural networks trained with backpropagation. Nature, 601(7894):549-555
2022.
[36] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnis mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 178.0777 2017.
12

---

Supplementary Material for: Neural Oscillators are Universal
A Another universality result for neural oscillators
The universal approximation Theorem 3.1 immediately implies another universal approximation results for neural oscillators, as explained next. We consider a continuous map F : R - RR; our goal is to show that F can be approximated to given accuracy c by suitably defined neural oscillators. Fix time interval [0, T for (an arbitrary choice) T = 2. Let Ko C IPP be a compact set. Given Co( T]; RP), , by setting
E E R we associate with it function (c(t) E
ug(t) == tς.
(A.1)
Clearly, the set == {ug & E Ko} is compact in Co((O, T]; RP). Furthermore, we can define an operator & Co: [R)) + Coo T): by
{ 0. (t 1)F(w(1)),
E (0, 1). E [1
(())(()
(A.2)
where F : R R2 is the given continuous function that we wish to approximate. One readily checks that 4 defines causal and continuous operator. Note, in particular, that
(T)) (T ( ())(((())))))))))))))))))))))))) F()),
is just the evaluation of F at E, for any Ko-
Since neural oscillators uniformly approximate the operator 4 for inputs ug K, then consequence Theorem 3.1 and (23).. follows that, for any > 0 there exists m E N, matrices W Rmxm V Rm* and A Roxmm and bias vectors b E R c E R2, such that for any E E Kg, the neural oscillator system,
(t) = o (W (t + + b), ve (0) = (0) 0,
(A.3)
(A.4)
<t(t) = Aye (t) + c,
(A.5)
satisfies
Ize(2) - F(E) = (T) - 2 (T)) < sup
(zt(t) (t)) Se
sE(o,]]
uniformly for all & E Kg. Hence, neural oscillators can be used to approximate an arbitrary continuous function F Rº R7, uniformly over compact sets. Thus, neural oscillators also provide universal function approximation.
B Proof of Theorem 3.1
B.1 Proof of Lemma 3.5
Proof. We rewrite y(t) 1 So u(T) sin(w(t 7)) dr. By direct differentiation. one readily verifies that ytt) so defined, satisfies
(()) - 1 u(T) cos(w(t - 7)) dr + [u(T) (r) ( ((sT)cos(((())) - 7)) dr,
account of the fact that sin(0) = Differentiating once more, we find that
v(t) = -w / u(T) sin(w(t - dr [u(T) cos(w(t 7))
-w + utt).
Thus y(t) solves the ODE 2.6). with initial condition y(0) j(0)) 0.
13

---

B.2 Proof of Fundamental Lemma 3.6
Reconstruction of a continuous signal from its sine transform. Let [] We recall that we define the windowed sine transform Leu(w) of a function u [0. T) + R2, by
T] C R be an interval.
Cr((w) = I ult - sin(wT) dr; WER.
In the following, we a compact set K C Co T): Note that for any a E K, we have w(O) 0. and hence K can be identified with subset of C((-oe, T))) consisting of functions with supp(u) C. [0, 27. We consider the reconstruction of continuous functions u E K. We will show that u can be approximately reconstructed from knowledge of (w.)). More preciscly, provide detailed proof the following result:
Lemma B.1. Let K C C((-oo, R)) be compact, such that supp(u) c [0, T) for all E K. For any e, At O, there exists E N, frequencies wi,..,,WN E RR {0), phase-shifts I....., ON E R and weights cn,,..,, ON E I, such that
sup [E(o,Ar)
u(1 ()
Σ ayLculus) sin(wsT - 02) Se,
for all u K and for all t e [0,
Proof. Step 0: (Equicontinuity) We recall the following fact from topology. If K C((-x,, T]; (RP) is compact, then it is equicontinuous; i.e. there exists continuous modulus of continuity á [0, 0o) (c) with d(r) - 0 as 0, such that
lu(t - r) u(t) S ((T), VT 2 0, te [0, T) VuEk.
(B.1)
Step 1: (Connection to Fourier transform) Fix to E [0, T] and u E K for the moment. Define f(r) = w(to- 7). Note that f E C([0, oo); and f has compact support supp(f) C [0, We also note that, by [.II. have
If + 7) - f(t) Φ(T), Vr te [0,7].
We now consider the following odd extension of to all of IR:
F(T) = [(()). (-((-));
for r 2 0,
for r S 0.
Since F' is odd, the Fourier transform of F' is given by
F(w) = f F(T)e HOT dr = =if F(T) sin(wT) dr 2i [' f(T) sin(wT) dr 2iL u(w).
Let 6 > ( be arbitrary. Our goal is to uniformly approximate F(T) on the interval [o, At.. The main complication here is that F lacks regularity (is discontinuous), and hence the inverse Fourier transform of F does not converge F uniformly over this interval: instead. more careful reconstruction based on mollification of F is needed. We provide the details below.
Step 2: (Mollification) We now fix smooth, non-negative and compactly supported function
p: RR-- + R, such that supp(p) C [0, 1), p 2 0, Jg p(t) di = 1, and we define a mollifier pe(t) == :((t/e) In the following, we will assume throughout that < T. We point out that supp(p.) C [0, c). and hence, the mollification (t) (F (t) satisfies, for 2 0:
F(t) - F.(t)) = ( (F(t) - F(t + (P) dr
1 (F(0)) - f(t (T) dr
sup If(t) f(t +1)) 1'' Pc(T) dr s o(c). rE(o.G]
14

---

In particular, this shows that
F(t)) - F.(0) S d(e), xE(o.T]
sup
can be made arbitrarily small, with an error that depends only on the modulus of continuity d.
Step 3: (Fourier inverse) Let (w(w) denote the Fourier transform of Fe. Since F, is smooth and compactly supported, it is well-known that we have the identity
F((T) = 2T
Felw)e-iwe iNT du, VIER,
where + [w( decays ===Fxx F* a convolution, we have Fe(w) = Pf (w), where [F(w) I 2 - is uniformly bounded, and (w)) decays quickly. In particular, this implies that there exists L L(c, T) independent of such that
very quickly (almost exponentially) |w| co. In fact, since
d F((T) 2x [
dul
/
((w) wo du
2TT11fll s
[e(w)] dw S [lflL
V
Ww/Z
(B.2)
Step 4: (Quadrature) Next, we observe that, since F and P. are compactly supported, their Fourier transform w + (( is smooth; in fact, for |T| s T, the Lipschitz constant of this mapping can be explicitly estimated by noting that
0 dw
[(w)) [
Sw supp(FF) (F (c)) di
[ !supptF))
i(t-T)(F * ( (t) dt.
We next take absolute values, and note that any in the support of F, obeys the bound |4| S + 27, while |r| T by assumption; it follows that
Lip w (w)) ) s (2T + F)| ll = 3T| Vr E [0,
thus follows from basic results on quadrature that for an equidistant choice frequencies W1 < wN. with spacing Aw = 2T/ 1), we have
1 1h 2T [" F(w)d(w)e iN de
Aw N 2x
CL2 Fl Z V [0,7],
Σ j=1
F(w;)d((w))ee 1UyT W0yT I
N
for an absolute constant C > 0. independent of F, T and N. By choosing N to be even, we can ensure that Wj # 0 for all j. In particular, recalling that L L(T, depends only on and T, and choosing N = N(T,e, e) sufficiently large, we can combine the above estimate with (B22) to ensure that
Aw (ww)) BOsT 2111fllree, V E
(T(T)
where we have taken into account that F| 1
Step 5: (Conclusion) To conclude the proof, recall that F(w) = 2iLr, u(w) be expressed in terms of the sine transform L/u of the function u which was fixed at the beginning of Step 1. Recall also that f(T) u(to - 7), so that || = ||l|iz> Hence, we can write the real part of Aw 24 Lap u(w; )e (e twyt, in the form aryLto (4i) (wi)))nn(( sin(w,r - .0,) for coefficients aj IR and 0, E RR which depend only Aw and p. (w;), but are independent of u. In
AW (e (wy)e -wwy twyz
2T
2n
15

---

particular, it follows that
N
Aw
N
sup >e re(0.Ar)
ulto - -r- Σ
ai u(ws)) sin(wyt 18))
= sup
F(T) - Re
Σ
(w))) vuss
2π
j=1
j=l
Aw
N
S sup
F(T) 2m
Σ
(ww))
inst
rElUAA]]
==1
sup rE(0,Ad]
[F(T) P((T)
Aw N ((wy)Pe(wss)e
+ sup rE[o,Ar]
Fe(T)
2m Σ
ResT
By Steps and 3, the first term on the right-hand side is bounded by S Φ(E), while the second one is bounded by < 2supuex ||u I S Ce, where C = C(K) oo depends only the compact set K C C([0, T[; RP). Hence, we have
N
ulto 7) rE(0.Ad
sup
Σ 3=1
az aaLio ((w;) sin(wyr - 0,) $ p(c) + +C..
In this estimate, the function u E K and to E [0, were arbitrary, and the modulus continuity ø as well the constant C' on the right-hand side depend only on the set K. it thus follows that for this choice of ag W and itj, we have
N ult -) Σ j=1
sup sup sup uek rE|o.T] (o(AA)))
aalls()))) sin(wyr - 0,) s <(() + Ce.
Since 0 was arbitrary, the right-hand side can be made arbitrarily small. The claim then readily follows.
The next step in the proof of the fundamental Lemma 3.6 needs the following preliminary result in functional analysis,
Lemma B.2. Let Y be Banach spaces, and let K C X' be a compact subset. Assume that Φ: X y is continous. Then for any e > O, there exists a ò > such that if I|u u "II1xx S 8 with E X, u E K, then [+(()) - ((u )ll Sc.
Proof. Suppose not. Then there exists Eo 0 and sequence uj. uz Gj N), such that ||uy u I1x I أー while [ı{(u)) - (u(k))ls 2 6g. By the compactness of K, we can extract a K subsequence ja - oo, such that uf + u converges to some u E K. By assumption uy this implies that
(k-re) O, |0 xK 11x & ||ug - u Ilx + Ilut "I
which, by the assumed continuity of 4. leads to the contradiction that 0 < < [[ ((u )ly + 0, as co.
Proof of Lemma 3.6 Now, we can prove the fundamental Lemma in the following,
Proof. Let > 0 be given. We can identify K C (R) with compact subset of C((-oc,T]::P") RP), by extending all u by zero for negative times, i.e. we set u(t) for t< O. Applying Lemma B.2 with X' [o, Ti: IRP) and y = Co((O, 7): R9), we can find a 5 > 0.
such that for any u Co T): RP) and n E K, we have
|| " مرا <
{1l")ll>~ Sc. [b Φ(u )1
(B.3)
16

---

By the inverse sine transform Lemma B.1 there exist N E N. frequencies WI..... WN # 0. phase- shifts i .,....,0 N and coefficients a1..... ON, such that for any a K and E [0;
N
sup rE(0.T]
ult - 7)
Σ
aj Lru(ws) sin(wit 02) 8.
Given L ([: by
Leu(wN), we can thus define a reconstruction mapping R. RN x [0, -
(NBI.....,B:I(((C Σ ag sin(w (t))
Then, for r E [0,4]. f). we have
|u(T) - xule S
We can now uniquely define : x 10,T /4| - Cu(lo, T): R²), by the identity
(rsu(wi),...,,u(((aa //44) = (R(Cru(w1), L(((wn):t).
Using the short-hand notation R₁u = R(C, u (w1),.... .... L Cuu(wN); we have supre(o.) lu(T)
Rau(T)] s 6, for all & E [0, T]. By (B33). this implies that
(t))(t) ve (C..(())).... Cul(ww); :/ //4]| = =I(()(t) -000---RRRA ((Ra))(t) sc.
B.3
Proof of Lemma 3.7
Proof. Let w o 0 be given. For (small) parameter 0, consider
Us
8 o o0-sw²y + su), (= = ÿ. (0) = 0.
Let Y be the solution of
- + u Y(0) (())) O. Then we have, account of (0) and o' o'(0) 1,
o-'o(-sw 2 su) 1- + u a 1--sw - su) o(0) -0(0)l-w2y + u
s
1 f á
lal-cw (w)] dc o d'(0)l-w + u|
s Jo ac
(L [o'(-SwPY (u) -0(0]] ds) l-wPy + u/
follows from Lemma 3.5 that for any input K, with supwek ||u||=> =; B < co, we have uniform bound YY| L00 BT/w, hence we can estimate
w + ul < B(WT + 1),
uniformly for all such u. In particular, it follows that
is a sY + su) [- + w]] < B(Tw+1]
sup (d(x))o'(00 [x(++B(Tw+1)
Clearly, for any O, we can choose (0, 1] sufficiently small, such that the right hand-side bounded by 6, i.e. with this choice of s,
|s-lo(-sw (t) + su(t)) - I- - u(t)] < 8, VE [0,7],
holds for any choice of E K. We will fix this choice of in the following, and write g(y, u) s- sw + su). We note that y is Lipschitz continuous y, for all ly/ S BT/W and lu| S B, with Lip,(g) S w² suP)e)s((T+)) (E))
17

---

To summarize, we have shown that Y solves
Y ,Y, u) S where I & 6. By definition, 1/8 solves
Y(0) Y(0) 0,
U. = 98.. u);
Va(0) - (. = 0.
It follows from this that
(1)(()) Y(01 < / L (0)(vs(0), (()) s(Y()), x(0)) 1[(0))) do dr
- / l'- (o),,, Y(0)+ + d0 dr
lo'(E)l /. (Va(t) (()) T26.
$ Tw2 sup BWT
Recalling that (t) = Lru(w), then by Gronwall's ' inequality, the last estimate implies that
sup 1E(0,77
(1) Lru(w)) sup ly. YI C6, 1Ej0.77
for a constant C = C(T, w, (; lo' > 0, depending only on w. B and o'. Since 5 0 was arbitrary, we can ensure that C8 S 6. Thus, we have shown that suitably rescaled nonlinear oscillator approximates the harmonic oscillator to any desired degree of accuracy, and uniformly for all K.
To finish the proof, we observe that solves
o=o(-w²p + su),
v(0) i(0)) 0,
if, and only if. ys s/s solves
Ü. s 1--oo(sss) swyyy Us su), (0) = (o) O.
Hence, with W 2 s, b and A s_ we have
sup [.TT]
AAy(l) - Cru(w)] = sup :E(0.7]
(1) - Cru(w)) Se.
This concludes the proof.
B.4 Proof of Lemma 3.8
Proof. Let At be given. By the sine transform reconstruction Lemma B.I there exists N N. frequencies Wi....,wy WNN weights aN and phase-shifts U1,..., UN. such that
sup rE(D,Ad]
u(t 7)
Σ
CyCiulyy) sin(wrr 0,) S At YueK,
(B.4)
j=1
where any u E is extended by zero to negative times. follows from Lemma 3.7 that there exists coupled oscillator network.
ÿ = (w 2 + Vu b).
v(0 = (()) = 0,
with dimension m = PN, and w E Th E mxp, and linear output layer y + Ay, À E mmxm such that [Ay(t)]s R C,u(w;)) for j = ,,..., N: more precisely, such that
sup Σ TE
- -(0l)))
VuEK. S 2
(B.5)
Composing with another linear layer B Imm RRXX R., which maps B = 13
3N]
BB = Σ 3 sin sin 0,) R"
18

---

we define A == BA, and observe that from
B.4 and (B.5p
sup tE[0,]]
lu(t At) Ay(t)) = sup LE(0,1]
u(t At) Σ ayCilluy) sinl - 03)
|ayl Crulw,)) [Ayls sin(w; (inl(t)/sinwws 0,)
sup *E[0.7]
S c.
B.5
Proof of Lemma 3.9
Proof. Fix Σ, A,7 as in the statement of the lemma. Our goal is to approximate u + Eo + 7). Step I: (nonlinear layer) We consider a first layer for hidden state 12)" E IRP+P, given by
51 (t) (t)
= (Au(t) + + J. v(0) 5(0) = 0.
(tt(t) = (n))
This layer evidently does not approximate (Au(tt) + 7); however, does encode this value in the second derivative of the hidden variable y1. The main objective of the following analysis is to approximately compute ği (t) through a suitably defined additional layer.
Step 2: (Second-derivative layer) To obtain an approximation of (AA((t) + 7)), we first note that the solution operator
S :: u(t) + (t), where ii()) = (Aa((t) + +7) - - 700) = (00) = 0,
(10 with (7 = n(() 7(0) 0. Note that 7 is very closely related to y1. The fact that fj = 0 is important to us, because it allows us to smoothly extend y) to negative times by setting ry() 0 for & < 0 (which would not be true for yı (t)). The resulting extension defines a compactly supported function 1) (-20.0) 0] + R". with 7) E (( Ti; RP). Furthermore, by continuity of the operator S, the image S(K) of the compact set K under S is compact in ((-oo, T); RP). From this, follows that for small At > 0, the second-order backward finite difference formula converges,
defines continuous mapping S Co((o, (I:
(1))) (t At) + n 2n1) i(() = -0)--0(1), An = -()), a EER K,
sup tE[0,T]
A
where the bound on the right-hand side is uniform in E K, due to equicontinuity of (i = S(u), u E K}. In particular, the second derivative of n can be approximated through linear combinations of time-delays of Tj. We can now choose At > 0 sufficiently small so that
ntt 2n(t At) ntt 2At)
sup tE|o,T]
n(tt) S 2|
Vy S(u), u E K,
At
where | denotes the operator norm of the matrix E. By Lemma 3.8
applied to the input set
K S(K) c Co((0, 7]; R"), there exists a coupled oscillator
E(t) a (w 2(t) Vn(() + +b), =(0) =(0) 0,
(B.6)
and a linear output layer + Az, such that
eAt² _|
sup +e(o,r]
[n(() 21)(1 At) n(( 2A() Āz(1)
-n= - S(w), uE K.
Indeed, Lemma 3.8 shows that time-delays of any given input signal can be approximated with any desired accuracy. and ry(t) 2n(t - A) - n7(t - 24) is simply linear combination of time-delays of the input signal 17 in (B.6).
To connect 1)( back to the (() = [n (1): (0) constructed in Step 1, we note that
î (Au(t)) + 6) o(b) Ø1 1/2
19

---

and hence, taking into account the initial values, we must have 31 y/2 by ODE uniqueness. In particular, upon defining matrix V such that Vy == Vyı V = Vnj. we can equivalently write (B.6) in the form,
(t) = (w @ 2(t) + Vy(t) + b), =(0) = (0) = 0.
(B.7)
Step 3: (Conclusion) Composing the layers from Step and 2, we obtain a coupled oscillator
v o(w @ of + V + +b'), (e 1,2),
initialized at rest, with y' y. y² 2, such that for : Σλ and En(r)) obtain
sup (() +c] - (o(((()) + ا - ||| sup <E(O,T]
A=(() - (o(A((() - 7) - (
tE[o,]]
|-|| sup tE[0,]]
A=(() i00
Az(I) ((t) - 2n(t At) + n1(t -24t)
|>||
sup [E[o,]]
A12
(1()) 2n(1 - At)
+ 17(t - 201) :j(t)
+ |||||
sup 1E[0,7]
At
E
$
+ E.
2
This concludes the proof.
B.6 Proof of Theorem 3.1
Proof. Step 1: By the Fundamental Lemma 3.6 there exist N, a continuous mapping V, and frequencies Wi,..., WN, such that
(t)))(t) - (LCu(u))) ... (uu(:)))),,,,,((w)):: 8//41/ sc
for u E K, and E [0,7]. Let M be a constant such that
2
(u((w)))...., (C((uuu)).,,,I(((xx))) _ S M,
MM
for all u E K and t E [0, 2J. By the universal approximation theorem for ordinary neural networks, there exist weight matrices 2, and bias 7, such that
( ...
.........N.......... B8: /² /4) Eo + se, ::= ........ Sx:t2/4/7/
holds for all E [0, 13,, 1........111====_1. S M.
S c, where Lip(a) = suPiess A||++v+++ (o'(E) denotes an upper bound on the Lipschitz constant of the activation func- tion over the relevant range of input values. follows from Lemma 3.7 that there exists an oscillator network,
Step 2: Fix 61 S sufficiently small, such that also
/ aw O y² + bl)+ y² (²) 0,
(B.8)
of depth 1, such that
sup (.Cu re(u.2]
- A (0) E1
for all u E K. Step 3: Finally, by Lemma 3.9 there exists an oscillator network,
jj² = o の 1 1² +
of depth 2. such that
sup re [0,7]
- -orA + 7) se.
20

---

holds for all y' belonging to the compact set K1 == S(K) C Co((o where S denotes the solution operator of (B.8).
Step 4: Thus, we have for any a E K, and with short-hand Li((w) = (Lealur),... Lou(wN)),
(tu)(tt) - - - (t)((t) - /4C)(w)):(
+ PONN //A
[=(A[C((()):P//)}}>-- Eo(AA' +7)]
(1) 1-)
By step 1, we can estimate
(t))((t) - V(Ccu(w): Sc. AtE [0,T], u K.
By the choice of E, 7 we have
[V(Lu(W);; /4) / + Sc, VIE (0, 77. wER K.
By construction of y² in Step 2, we have
[A[Cauu(( //4 + + - (t)) m
[ || | 8/41 -
14
|
for all E [0, T] and K. By construction of y² in Step 3, we have
(oAA'' + 7) 42y s,, [0, ue
Thus, we conclude that
A") 4e.
for all E [0. and u E K. Since > was arbitrary, we conclude that for any causal and continuous operator + Co, T); RP) Co((O, T]; R2), compact set K Co :RP) and e > 0, there exists a coupled oscillator of depth 3, which uniformly approximates to accuracy for all K. This completes the proof.
21


Reviews:
[{"review_id": "YEjfTCOs8D", "reviewer": "Reviewer_Ad2m", "summary": "This work presents a comprehensive review of diverse learning architectures based on ordinary differential equations (ODEs) and explores their relationship with coupled oscillator systems. Furthermore, the paper provides a proof showcasing the universality of the coupled oscillator architecture. The universality of ODEs is a discernible outcome, as discrete dynamics of neural networks can be effectively mapped to discrete solutions of ODEs utilizing various integration methods. Consequently, the novelty of this work might be slightly diminished by this intuitive understanding. However, the significance of this research lies in its ability to formally define the problem and rigorously prove its assertions. This is particularly noteworthy considering the limited attention given to analog-based learning systems in traditional machine learning and the growing relevance of analog systems in both classical and quantum domains. The majority of the proofs presented in the paper are well-written, rigorous, and appear valid, with the exception of the proof of the Fundamental Lemma. Addressing concerns regarding the Fundamental Lemma, which is utilized to prove the main theorem, would greatly strengthen the case for accepting this work. With this consideration, I am inclined to support the acceptance of this paper.\n", "questions": "How important is Lemma 3? The significance of Lemma 3 in the constructive proof of the mapping from neural networks to coupled oscillators warrants further examination. While its importance may not be immediately evident, providing a more detailed explanation of how Lemma 3 is utilized would greatly enhance readers' understanding of the underlying logic behind the proof. Clarifying the role and implications of Lemma 3 in the context of the proof would be beneficial for the readers' comprehension.", "limitations": "This work primarily concentrates on establishing a rigorous understanding of the universality of the coupled oscillator system. While this may appear somewhat self-evident to researchers engaged in developing and deploying similar systems to address real-world problems, it is important to note that the findings do not introduce any novel architectures stemming from this discovery. As a result, the impact of this work remains confined to theoretical considerations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Provides a comprehensive review of ODE-based learning architectures and their relationship to coupled oscillator systems. \nEmphasizes the importance of formalizing the problem and rigorously proving the assertions, given the lack of attention to analog-based learning systems in traditional machine learning and the increasing relevance of analog systems in classical and quantum domains.", "weaknesses": " The proof of the Fundamental Lemma claims the existence of a continuous function that transforms the oscillator output, which is a sine transform of an input function, into a target function. However, there seems to be a cyclic nature in the reasoning. The proof utilizes the target function, which is not known how to implement with oscillator networks composed of a reconstructing map R as the continuous function. This cyclic dependency raises questions about the validity and logic of the proof.\nIf one already has access to the target function, it becomes obvious that the target function can be approximated. See line 548 and the equation following it in the supplementary material as examples where this circular reasoning is apparent.\n\nAddressing these concerns is crucial to ensure the integrity and validity of the proof presented in the paper. Providing clarification and resolving the cyclic reasoning issue would greatly enhance the readers' understanding and confidence in the proof of the Fundamental Lemma.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for your appreciation of the merits of our paper and your welcome suggestions to improve it. Below, we address the concerns raised by the reviewer and thank the reviewer in advance for their patience in reading our detailed reply.\n\n\n1.Regarding the reviewer's concerns about the potentially cyclic nature of the Fundamental Lemma, we would like to clarify that our proof proceeds in two steps: In a first step (the fundamental lemma), we show that the underlying operator $\\Phi$ can be approximated by an operator of a very specific form (given by the composition $\\Psi$ with values of the sine transform). As the reviewer correctly points out, the function $\\Psi$ is closely related to the operator $\\Phi$; in fact, the function $\\Psi$ defined there could be interpreted as a finite-dimensional analogue, or a finite-dimensional encoding, of the operator $\\Phi$. The main point of the Fundamental Lemma is that we can effectively replace the operator $\\Phi$ with a function $\\Psi$, making the approximation by neural oscillators more tractable. This is completely analogous to classical proofs of the universal approximation of multilayer perceptrons (e.g. [Pinkus, Acta numerica, 8, 143-195]): The proof here proceeds in two steps. It is first shown that any continuous function $f(x)$ can be approximated by a specific polynomial $p(x)$, for example obtained by projecting the underlying function $f(x)$ onto the space of polynomials of a certain degree. Then, in a second step, it is shown that any such polynomial $p(x)$ can be approximated by an MLP. The first step is a reduction to a specific set of functions with additional structure. Thus, the $\\Psi$ in our Fundamental Lemma is the analogue of the afore-mentioned $p(x)$ in our proof of universality for neural oscillators. We thank the reviewer for raising this point and will include a more detailed explanation along these lines in the CRV, if accepted; in particular, we will also point out the above analogy with standard universal approximation results in the theory of multilayer perceptrons. \n\n2. Regarding the reviewer's question about Lemma 3, we presume that they refer to Lemma 3.8. This Lemma is indeed one of the most crucial ingredients in the proof of our universal approximation theorem, as it shows that the function $\\Psi$ of the Fundamental Lemma can be approximated by neural oscillators. To expand on this point, we note that it is a relatively simple observation that neural oscillators can approximately compute the sine-transform (Lemma 3.6), because even linear oscillators can compute it. It is also obvious that the function $\\Psi$ can be approximated by a single hidden-layer MLP, as it follows from classical universal approximation results. However, due to the underlying oscillatory dynamics, it is far from obvious that the output of a single hidden-layer neural network can be reproduced by neural oscillators. Lemma 3.8 is precisely the result that guarantees this. It is also visually represented by the bottom half of Figure 1. We thank the reviewer for this question, and will highlight the crucial role of Lemma 3.8 in the CRV, if accepted. \n\nWe sincerely hope to have addressed all concerns, particularly about the Fundamental Lemma, to your satisfaction, and hope that you will update your assessment accordingly. "}, {"review_id": "h4w0Ld2Nna", "reviewer": "Reviewer_y7S4", "summary": "The authors present a universal approximation theorem for a system of 2nd order ODEs they call \"neural oscillators.\" At the heart of the construction is the fact that a forced harmonic oscillator with input computes a windowed sine transform of the input, up to a constant (Lemma 3.4). The other major piece (Lemma 3.5) states that windowed sine transforms taken at a finite number frequencies can be input to a nonlinear function to approximate a causal, continuous operator to desired accuracy.", "questions": "See \"weaknesses.\" I remain fuzzy on exactly where nonlinear dynamics are needed in the construction.\n\nl144: Are the masses \\mu being ordered here? If so, then the quality of the upper diagonal approximation will depend on the differences between the masses and the gap bound 1/epsilon should be stated explicitly for clarity.\n\nMinor comments:\n\nEq 2.3: $w^L I$ -> $\\text{diag}(w^L)$\n\nl95: causally independent, not statistically independent due to common input.\n\nParagraph starting on l123: A sentence more of clarification here would be helpful.\n\nl131: this would be more readable if F were replaced with f to follow the lower-case vector/upper-case matrix convention.\n\nl140-141: It should be made more clear that we are supposing this particular structure for the symmetric matrix C.\n\nLemma 3.5: $t$ is not qualified. Should it be: $\\forall \\epsilon \\exists N$, frequencies $\\omega_1, \\dots, \\omega_N$ and a continuous mapping $\\Psi$ such that $\\forall t \\in [0,T^2/4]$ ?\n", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The recent applications of nonlinear oscillators in machine learning makes the work relevant and interesting.\n2. The work seems to be technically sound.\n3.The discussion brings up many interesting points including feedforward structure in physical systems, recent work in polynomial memory, expressibility versus learnability.\n4. The main paper and proofs in the appendix are all detailed, readable, and well-organized.\n", "weaknesses": "The paper would be helped by a more detailed discussion of where nonlinearity is needed to achieve the universal approximation result. For example, it is known that the time-delay result in Lemma 3.7 can be achieved with a linear system. See [1], for example. Then is it the case that nonlinear oscillators are only needed in the two-layer neural network construction (Lemma 3.8)? Another example would be the windowed sine transform, which can be computed by a linear system.\n\n[1] Gu et al. (2022) \"How to train your hippo: State space models with generalized orthogonal basis projections\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for your appreciation of the merits of our paper and your welcome suggestions to improve it. Below, we address the concerns raised by the reviewer and thank the reviewer in advance for reading our detailed reply.\n\n1. Regarding the reviewer's question about more detailed discussion of where nonlinearity is needed, we agree with their assessment that nonlinearity is only needed in Lemma 3.8. As rightly pointed out by the reviewer, linear oscillators are sufficient to compute the sine-transform and the time-delay operator (we also thank the reviewer for pointing out the relevant reference Gu et. al., which we will include in the CRV, if accepted).  Due to the underlying oscillatory dynamics, it is far from obvious that the output of a single hidden-layer perceptron can be reproduced by neural oscillators. The result of Lemma 3.8, which is visually represented by the bottom half of Figure 1, guarantees this. We thank the reviewer for their suggestions, and we plan to highlight the important role of Lemma 3.8 in the CRV, if accepted, and in particular, we will make it explicit that the other parts of our proof of universality do not require non-linearity.\n\n2. Regarding the question on the mass gap: Yes, indeed the intent is to introduce an ordering of the masses, such that $\\mu^\\ell / \\mu^{\\ell-1} \\sim \\epsilon$ is small. We will state this, and the explicit dependency of the quality of the upper diagonal approximation on this mass gap, in a CRV, if accepted. We thank the reviewer for suggesting this clarification.\n\n3. We thank the reviewer also for the minor comments, which we will readily incorporate in a CRV, if accepted. \n\nWe sincerely hope to have addressed your concerns, particularly about delineating the specific role of the nonlinearity in our universal approximation theorem, and would kindly request the reviewer to update your assessment accordingly."}, {"review_id": "v4ZIHo2oZE", "reviewer": "Reviewer_QtjX", "summary": "This paper introduces an abstract framework of neural oscillators and proves the universal approximation theorem of multi-layer neural oscillators, which shows that neural oscillators with a specific structure have the capability of approximating causal and continuous maps between spaces of continuous functions.\n", "questions": "Can it be stated whether it is possible to train parameters by approximating such a hierarchical separate approximation with existing algorithms? Or at the application aspect, can a new training method be designed to match the structure designed in this paper?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper provides a novel proof approach to the approximation theory of neural oscillators. They cleverly used the time-windowed sine transform and the approximation properties of traditional neural networks as a springboard to realize the proof of the entire approximation theory.\n2. They explicitly construct a three-layer neural oscillator structure which is ingenious and corresponds step-by-step to their proof process.\n3. The logic of the outline of the proof is clear and easy to follow.\n", "weaknesses": "The proof process is novel, and the structure is specially designed, but in terms of applicability, this design increases the limitation of the approximation theory. It is designed to approximate different items at different layers, not just based on the last loss function or the final operators to be approximated, so it does not match the gradient-based parameter training method well. For example, the first layer is supposed to approximate the time-windowed sine transform, but based on conventional gradient algorithms, this particular limitation cannot be guaranteed. In fact, it is unknown which training method can be used to achieve such a special structure with the approximation property.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for reading our paper and for your comments. Below, we address the concerns raised by the reviewer and thank the reviewer in advance for their patience in reading our detailed reply.\n\n1. The main contribution of this paper is fundamentally an approximation theoretic result, which shows that the neural oscillator architecture is sufficiently expressive to approximate a very general class of causal operators. While we uncover specific mechanisms that the architecture can exploit in principle, no claim can be made that a neural oscillator, trained in practice, will take on the specific form of the neural oscillator constructed in our proof. Nor would this necessarily be desirable, as the practical optimization may discover a more efficient, yet very data- and application-specific, approximation of a given operator of interest. This stands in contrast to the specific construction of our proof which is, in some sense, data agnostic, i.e, the same structure of the model is realized whatever the underlying task is. Thus, our theoretical contribution should be viewed as an analogue of well-known, classical universal approximation theorems for e.g. multi-layer perceptrons, convolutional neural networks, recurrent neural networks, transformers etc. Such results demonstrate that these architectures are sufficiently expressive to approximate a very wide class of functions, but it is generally not expected to retrieve the particular architecture, constructed in the proof of this universal approximation theorem, in practice. Moreover, the universal approximation theory also says that the global minimum of the optimization error can be made as small as possible assuming sufficiently large model size and sufficient data samples. \n\nHowever, as alluded to in our paper, our proof of universality does immediately imply that an architecture combining linear dynamics with a non-linear read-out is also universal. It would be interesting to further investigate this type of architecture, which directly reflects the basic mechanisms embedded in our proof. As already argued in lines 313-318 of this paper, this structure is reminiscent of the heavily used structured state space models (Refs [8,9] in our paper) for which no universal approximation theorems exist. So, a natural next step would be to investigate if our proof also applies to these architectures which use the structure explicitly in the model itself and will be a topic for future work. \n\nWe sincerely hope to have addressed your concerns, particularly the contributions of our paper in terms of universal approximation, and would kindly request the reviewer to update your assessment accordingly."}, {"review_id": "umRyBtlinx", "reviewer": "Reviewer_vPvT", "summary": "Various machine learning architectures inspired by oscillators perform well in a variety of tasks. The authors study the approximation ability of neural networks based on oscillators. The main contribution of this paper is the construction of a neural oscillator framework combining nonlinear dynamics and a linear read-out, and the proof of a Universality theorem for neural oscillators.", "questions": "As a further reflection on universal approximation theory, I'm curious about the effect of network width and depth on expressiveness. Specifically, is there a minimum width that can achieve a universal approximation in infinitely deep networks or in fixed-depth networks?\n\nThe title is confusing and needs to be revised: 'universal' for what? universal approximator?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The universal approximation property is an important theoretical support for neural network architectures. This is the novel theoretical result to get a Universality theorem for neural oscillators.\n* The proof strategy of the paper is relatively clear, and the illustration in Figure 1 makes it easier to understand.", "weaknesses": "It would be better if the authors could discuss how this universality result contributes to existing work in more detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for your appreciation of the merits of our paper and your welcome suggestions to improve it. Below, we address the concerns raised by the reviewer and thank the reviewer in advance for their patience in reading our detailed reply. \n\n1. Our main motivation for studying the universal approximation of neural oscillators is the empirical work on CoRNN and UniCoRNN, but also related systems discussed in the introduction. A priori, it is unclear what types of operators can be approximated, even in principle, with such oscillator-based architectures. Available empirical results suggest that this class includes several operators of interest, but the approach has remained without any solid theoretical foundation. Our result shows rigorously for the first time that neural oscillators are in fact very expressive, capable of approximating almost arbitrary causal operators.\n\n2. Regarding the reviewer's question about the effect of network width and depth on expressiveness, our proof shows that a neural oscillator of fixed depth 3 and a sufficiently large width can approximate any causal operator. Upon closer inspection of the Fundamental Lemma (Lemma 3.5), we can further relate the required width of the neural oscillator to two fundamental factors: \n\nThe first determining factor is the number frequencies of the sine-transform that is required to approximately reconstruct, or at least adequately distinguish, the input signals. The width of the first layer in our construction is directly proportional to this number of required frequencies. This number of required frequencies could for example be estimated in terms of the smoothness of the input signal (with smoother input signals requiring fewer frequencies). Such analysis leads to a width proportional to $width\\sim N \\sim \\epsilon^{-1/s}$, where $s$ is the degree of smoothness of input signals (e.g. $s=1$ for continuously differentiable inputs) and $N$ the number of frequencies $\\omega_1,\\dots, \\omega_N$.\n\nThe second determining factor is the width of a shallow neural network that is required to approximate the function $\\Psi$ of the Fundamental Lemma. The width of the second and third layers in our construction are directly related to the width of this neural network. Quantitative bounds for the required width can be derived in terms of e.g. the smoothness of the function $\\Psi$; in general, this approach leads to an estimate of the form $width \\sim \\epsilon^{-N/k}$, where $k$ is the smoothness of $\\Psi$ (e.g. $\\Psi \\in C^k$), and where we recall that $\\Psi$ depends on $N$ inputs. While this gives an upper bound, it should be kept in mind that typical operators $\\Phi$ (and as a consequence also the function $\\Psi$) often possess additional structure, which goes beyond smoothness. Such structure can be exploited by neural networks, making the afore-mentioned upper bounds in terms of smoothness often quite pessimistic.\n\nTo summarize the above discussion, the required width of the neural oscillator is related to (i) the complexity of the input functions, and (ii) the properties of the operator (reflected by the function $\\Psi$), such as its smoothness. We plan to address this issue in a CRV, if accepted, and thank the reviewer for pointing out this avenue for possible extension of our work\n\n3. We thank the reviewer for pointing out the possibly confusing title. We plan to address this in accordance with their suggestion in a CRV, if accepted. \n \nWe sincerely hope that we have addressed the concerns of the reviewer to your satisfaction and kindly request you to update your assessment accordingly. "}]

Author rebuttal:
At the outset, we would like to thank all four reviewers for their thorough and patient reading of our article. Their criticism and constructive suggestions will enable us to improve the quality of our article. All the changes outlined below will be incorporated in a camera ready version (CRV) of the article, if accepted. We proceed to answer the points raised by each of the reviewers individually, below. 

Yours sincerely,

Authors of "Neural Oscillators are Universal".

Meta-review: This paper shows that coupled neural oscillator architecture can universally approximate mappings from continuous input signal to continuous output signal. Although the topic of the paper is relatively speculative and the theory does not tell us if such functions can be learned, reviewers have found the paper rigorous and well written. More throrough discussions on how the smoothness of input / output signals translates to required number of neurons would be helpful. I recommend to accept this paper as a poster.
Decision: Accept (poster)