Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules
Authors: Kazuki Irie
Francesco Faccio
Jürgen Schmidhuber
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural controlled differential equations
Neural ODEs
continuous-time sequence processing
linear Transformers
fast weight programmers

Abstract: Neural ordinary differential equations (ODEs) have attracted much attention as continuous-time counterparts of deep residual neural networks (NNs), and numerous extensions for recurrent NNs have been proposed. Since the 1980s, ODEs have also been used to derive theoretical results for NN learning rules, e.g., the famous connection between Oja's rule and principal component analysis. Such rules are typically expressed as additive iterative update processes which have  straightforward ODE counterparts. Here we introduce a novel combination of learning rules and Neural ODEs to build continuous-time sequence processing nets that learn to manipulate short-term memory in rapidly changing synaptic connections of other nets. This yields continuous-time counterparts of Fast Weight Programmers and linear Transformers. Our novel models outperform the best existing Neural Controlled Differential Equation based models on various time series classification tasks, while also addressing their fundamental scalability limitations. Our code is public.

Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules
Kazuki Irie' Francesco Faccio Jürgen Schmidhuber The Swiss AI Lab, IDSIA, USI & SUPSI, Lugano, Switzerland "A Initiative, KAUST, Thuwal, Saudi Arabia (kazuki, francesco, juergen]@idsia.co ch
Abstract
Neural ordinary differential equations (ODEs) have attracted much attention as continuous-ting tie counterparts of deep residual neural networks (NNs), and numer- ous extensions for recurrent NNs have been proposed. Since the 1980s, ODEs have also been used to derive theoretical results for NN learning rules, e.g., the famous connection between Oja's 's rule and principal component analysis. Such rules are typically expressed as additive iterative update processes which have straightfor- ward ODE counterparts. Here we introduce novel combination of learning rules and Neural ODEs to build continuous-ting sequence processing nets that learn manipulate short-term memory in rapidly changing synaptic connections of other nets. This yields continuous-ti time counterparts of Fast Weight Programmers and lin- ear Transformers. Our novel models outperform the best existing Neural Controlled Differential Equation based models various time series classificatinn tasks, while also addressing their fundamental scalability limitations. Our code public [
Introduction
Neural ordinary differential equations (NODEs) [ have opened a new perspective on continuous- time computation with neural networks (NNs) practical framework for machine learning based on differential equations. While the original
as continuous-deply version of deep feed-forward residual NNs 3 covers autonomous ODEs entirely determined by the initial conditions, more recent extensions deal with sequential data (reviewed in Sec. 1. in way similar what is typically done with standard recurrent NNs (RNNs) in the discrete- timetete-mmmmeee scenario. This potential for continuous-timg (CT) sequence processing (CTSP) is particularly interesting, since there are many applications where datapoints are observed at irregularly spaced time steps, and CT sequence models might better deal with such data than their discrete-time counterparts. However, the development of NODEs for CTSP is still Neural Controlled Differential Equations 4 (NCDEs; also reviewed in Sec. Sec.2.T) has in practice only one architectural variant corresponding to the "vanilla" RNN (5] Discrete-time processing, however, exploits many different RNN architectures as well as Transformers [6].
early stage. For example, popular approach of
While it is not straightforward transform the standard Transformer into CT sequence processor, we'll show that the closely related Fast Weight Programmers (FWPs) 7 8 10) and linear Trans- formers [II (reviewed in Sec. 2.3 have direct CT counterparts. In FWPs, temporal processing of short-term memory (stored in fast weight matrices) uses learnable sequences of learning rules. Hence CT versions of FWPs will require differential equations model the learning rules. This relates to trend of the 1980s/90s. Among many old connections between NNs and dynamical systems described by ODEs (e.g., 12 13 14
15 16. 17)), the theoretical analysis of NN learning rules in the ODE
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(briefly reviewed in Sec. 2.2]; many results on its stability, convergence, and connection Principal Compo-
framework has been particularly fruitful. Consider the famous example of Oja's rule [8]
nent Analysis 19 20 were obtained using its ODE counterpart (e.g.. [8 21 22
23
24
25
26
27
Here we propose novel combination of Neural ODEs and learning rules, to obtain new class of sequence processing Neural ODEs which are continuous-tine counterparts of Fast Weight Program- mers and linear Transformers. The resulting models are paneral-purpose CT sequence-processing NNs, which can directly replace the standard Neural CDE models typically used for supervised CT sequence processing tasks. To the best of our knowledge, there is no previous work on Neural ODE- based Transformer families for CT sequence processing, despite their popularity in important types of discrete time computation such as Natural Language Processing and beyond. We also show how our approach solves the fundamental limitation of existing Neural CDEs in terms of model size scalability.
We conduct experiments on three standard time series classification tasks covering various scenarios (regularly sampled, irregularly sampled with missing values, and very long time series). We demon- strate that our novel models outperform existing Neural ODE-based sequence processors, in some cases by a large margin.
2 Background
We briefly review the main background concepts this work builds upon: NODEs for sequence processing (Sec. 2.I). NN learning rules and their connection to ODEs (Sec. (.2]. and Fast Weight
Programmers whose memory update based on learning rules controlled by an NN (Sec. 2.3
2.1 Neural ODEs (NODEs) and Their Extensions for Sequence Processing
Here we review the core idea of NODEs [ In what follows, let n, N, d, din denote positive integers, T' be a positive real number, and denote an arbitrary set of real numbers. We consider residual layer (say, the n-th layer with dimension d) in an N-layer deep NN which transforms an input hn E IRd an output h. E Rd with a parameterised function fo Rdd Rd as follows:
h hn-1 + fo (fn-1)
This coincides
[28
29
30
3
33
with the following equation for e =
htt..) (ta--1) + (h((t.)
where 0, T - Rd is function such that h((t,)) h, holds for all n 0 S tn E [0, such that fn - - = > if 2 1. This equation is forward Euler discretisation of the ordinary differential equation defined for all E (to, T' as
N and
h1(() = (o((() or ((t) = la(t)) + [" fa(h(s))ds
(3)
s=to
where h denotes the first order derivative. This establishes the connection between the ODE and the deep residual net with parameters shared across layers] given the initial condition (46)) ho. the solution to this equation evaluated at time T, i.e., h(T), corresponds to the output of this deep residual NN, which can be computed by an ODE solver. We denote
function ODESolve taking four variables: h(T") = ODESolve( fe. fo, to, T). During training, instead of backpropagating through the ODE solver's operations, the continuous adjoint sensitivity method [35) (which essentially solves another ODE but backward in time) can compute gradients with (dd) memory requirement, constant w.r.t.
natural next step is to extend this formulation for RNNs, i.e., the index n now denotes the time step, and we assume an external input Φn E ddd at each step n to update the hidden state h h,,
h fo (hn T.,)
(4)
Depending on the property of external inputs (ax,) (a) ..., 2x there are different ways of defining NODEs for sequence processing. We mainly distinguish three cases.
"rr we make 9 dependent of t such that parameters "depttwlayer-ddeeedddn"" in standard deep nets.

---

First, when there is possibility to construct a differentiable control signal æ t + x=t) E R for t E [to. 2] from the inputs (am))=mi: an attractive approach by Kidger et al. [ handles the corresponding dynamics in a neural controlled differential equation (NCDE):
(t) to(f + S Fo(h(s))dz(s) = -h(tt) + s=ta
[ Fo(h(s))z'(s)dd s==t
(5)
where F6 is a parameterised function (typically few-layer NN) which maps a vector h(s) E Rd to matrix F& (h(ss) E iRdxd (we already relate this component to Recurrent Fast Weight Programmers below) and thus, F& (o(()))) denotes a matrix-vector multiplication. There are several methods to construct the control x : (ko 7] - Rd based on the discrete data points (#n))=i, such that its differentiability is guaranteed. In this work, we follow Kidger et al. 4 and mainly use natural cubic splines over all data points (which, however, makes incompatible with auto-regressive processing); for better alternatives, we refer to Morrill et al. [36]. Since the final equation is again an NODE with a vector field of form ge. (5, h()))= = F& ((((s)))) ())) all methods described above are applicable: ODE solver for evaluation and continuous adjoint method for memory efficient training. A notable extension of Neural CDES is the use of log-signatures to sub-sample the input sequence [37]. The resulting NCDEs are called neural rough differential equations (NRDEs), which are relevant for processing long sequences. One fundamental limitation of the NCDEs above is the lack of scalability of Fg. R + Rdxda, For example, if naively parameterise F6 using a linear layer, the size of its weight matrix is d² * din which quadratically increases with the hidden state size d. Previous attempts 38 do not successfully resolve this issue without performance degradation. In Sec. we'll discuss how our models (Sec. (32) naturally circumvent this limitation while remaining powerful NCDEs.
On a side note, the NCDE is often referred to as the "continuous- time analogue" to RNNs 4 but this is a bit misleading: discrete-time RNN equations corresponding to the continuous- Eq. 5 do not reflect the standard RNN of Eq. 4 but:
hn hn-1+ Wn,, 11
(6)
Wn = (h,))
(7)
where one network (Eq. 6 learns to translate the variation of inputs (xn, - xn-1) into a change in the state space, using weight matrix Wn-1 which itself is generated by another network (Fa : Rd Rdxdu; Eq. 7 on the fly from the hidden state. This model is thus a kind of Recurrent
FWP 39 40 10].
Second, even if x is not differentiable, having access to (piece-wise continuous æ defined and bounded over an interval of interest [lo T) is enough to define sequence processing NODE, by making it part of the vector field:
(t) = h((to)+++)))) + /"' fo((ss), xss))ds
(8)
Je=to
(t, htt)) can effectively be evaluated at any time t E (lo. 7). We refer to this second approach as a direct NODE method. While Kidger et al. [ theoretically and empirically show that this approach is less expressive than the NCDEs above, we'll show how in our case of learning rules one can derive interesting models within this framework, which empirically perform on par with the CDE variants.
where the vector field fo fo(()).
Finally, when no control function with one of the above properties can be constructed, a mainstream approach dissociates the continuous- time hidden state update via ODE for the time between two observations (e.g., Eq. Beelowllw) from integration of the new data (Eq, beloow.). Notable examples of this category include ODE-RNNS 42 which transform the hidden states h to hn for each observation H, available at time tn as follows:
U ODESolve(fo, hn
(9)
h Φo, (In, u.,)
(10)
where Eq. autoonomously updates the hidden state between two observations using a function fo, parameterised by 81, while in Eq. 10 function Ø8, parameterised by 82 integrates the new input 2r, into the hidden state. In Latent ODE-RNN [4]] a popular extension of this approach to the variational setting, the initial recurrent state hg is sampled from prior (during training, an additional encoder trained to map sequences inputs to parameters of the prior). While this third case not
our focus, we"ll Ill also show how to use FWPs in this scenario in Sec. 3.3 for the sake of completeness.
3

---

2.2 Learning Rules and Their Connections to ODEs
Learning rules of artificial NNs describe the process which modifies their weights in response to some inputs. This includes the standard backpropagation rule (also known as the reverse mode of automatic differentiation) derived for the case of supervised learning, as well as rules inspired by Hebb's informal rule (43] in "unsupervised" settings. Here we focus on the latter. Let n, du d be positive integers. Given a linear layer with a weight matrix W, E IR Xd (the single output neuron case dout is the focus of the classic works) at time n which transforms input nn E Rd to output yn E R as
yn
W 1
(11)
the pure Hebb-style additive learning rule modifies the weights according to
W n W + nn @ 3n
(12)
where @ denotes outer product and ?n E R, is a learning rate time n.
Oja 18 proposed stability improvements to this rule through a decay term
Wr = Wn- + 9n (z,, W 1yn)
(13)
whose theoretical analysis has since the 1980s been a subject of many researchers covering stability, convergence, and relation to Principal Component Analysis [18 21 22 23 24 25 26 27 44) One
key approach for such theoretical analysis is to view the equation above as a discretisation of the following ODE:
('( (t))((t)
(=(t)= W(t () yt))
1)
(14)
On a related note, studies of RNNs (e.g.. 45
46
or learning dynamics (e.g.. have also profited
from ODEs.
2.3 Fast Weight Programmers & Linear Transformers
are general-purpose (auto-regressive) sequence processing NNs. In general, an FWP is a system of two NNs: a slow NN, the programmer, rapidly generates during runtime weight changes of another neural network, the fast NN. The (slow) weights of the slow net are typically trained by gradient descent. Variants of FWPs whose weight generation based on outer products between keys and values 7 have been shown g to be equivalent to Linear Transformers [I]] (using the mathematical equivalence known from perceptron/kernel machine duality 48 4 These FWPs use sequences of learning rules to update short-term memory in form of fast weight matrix. A practical example of such FWPs is the DeltaNet [9] which transforms an input Fr E the into an output yn, E Ru each time step n while updating its fast weight matrix W E Rdux dhey as follows:
Fast Weight Programmers (FWP: g
9 10)
Bn qn, kin Un
W Wn
(15)
Wn
= W Wn-1+o(Baa(U,, - Wn-10(ka)) @ Φ(kn)
(16)
Yn
W))(qn)
(17)
where the slow net (Eq. 15 with weights Watow slow (142oou +daa) xda) generates key/value vectors kn E Rh and Vn E Ro as well as a scalar 3n E R to obtain a dynamic learning rate by applying a sigmoid function a, and Φ is an element-wise activation function whose output elements are positive and sum up to one (typically softmax). These fast dynamic variables generated by a slow NN are used in a learning rule (Eq. [6) akin the classic delta rule (50] to update the fast weight matrix. The output is finally produced by the forward computation of the fast NN, i.e., by querying the fast weight matrix by the generated query vector q. day (Eq. 17 An intuitive interpretation of the fast weight matrix is a key-value associative memory with write and read operations defined by Eq. [6]and 17 respectively. This encourages intuitive thoughts about memory capacity (limited by the number of "keys" we can store without interference) 19. For instance, if we replace the learning rule (i.e., memory writing operation) of Eq. by pure additive Hebb-style rule (and a fixed learning rate of 1.0): W W + Un @ o( (nn), we obtain the Linear Transformer [I]] (we refer to prior work [9] for further explanations of the omission of attention normalisation). Such a purely additive learning rule often suffers from long term dependencies, unlike the delta rule (9. We'll confirm this trend also in the CT models (using the Figen dataset). For later convenience, we introduce a notation FWP which denotes generic FWP operations: yn, Wn FWP Wn-13 W Wllo).

---

3 Continuous-Tinne Time Fast Weight Programmers
We propose continuous-time counterparts of Fast Weight Programmers (Sec. 2.3 which naturally combine ODEs for learning rules (Sec. 2.2) and existing approaches for sequence processing with NODEs (Sec. 2.T). We present three types of these CT FWP models in line with the categorisation of Sec. 2 while the main focus of this work is on the two first cases.
3.1 Direct NODE-based FWPs
In the direct NODE approach (reviewed in Sec. 2.1), D. we assume (piece-wise) continuous control signal æ :t + xxt) bounded over an interval TT. We make it part of the vector field to define an ODE describing a continuous time learning rule for a fast weight matrix (t)):
((t) = ((t) + /"
(18)
where W : W(1) Rduixdry is a function defined on [Lo, T7. and F6 is an NN parameterised by which maps onto Rdux This is a neural differential equation for learning program neural net through continuous learning rules, that is, to train a fast weight matrix ((t) (t) of a fast NN
(Eq. 20 below) for each sequential control Like in the discrete-time FWPs (Sec. 2.3). the output
y (T) E Ri is obtained by querying this fast weight matrix (e.g.. at the last time step 7):
(T) = (T) y y(T) = W (T)
(19)
(20)
where W Rech ddo is slow weight matrix used to generate the query ((7) E Ruu (Eq.
19
Now we need to specify F6 in Eq. 18 to fully define the learning rule. We focus on three variants:
((s) & m(s) v(s) @ (k(() ((s v(s)
Hebb-style
(s(
Oja-style
(21)
(v(s) - (s))(s)) & k(s)
Delta-style
where [3(s), ((s).((s) W W with slow weight matrix Wslow E (lltth As in
the discrete-time FWP (Sec. (.3). the slow NN generates B(s) R (to which we apply the sigmoid function to obtain a learning rate), key ((s) E dh and value v(s) E Rdda vectors from input (s)). These variants are inspired by the respective classic learning rules of the same name, while they crucially different from the classic ones in the sense that all variables involved (key, value, learning rate) are continually generated by the slow NN. In the experimental section, we'll 'll comment how some of these design choices can result in tase-depnndent performance gaps. In practice, we use the multi-head version of the operations above (i.c.. by letting H be a positive integer denoting the number of heads, query/key/value vectors are split into H sub-vectors and Eqs. 20 are conducted independently for each head). The output is followed by the standard feed-forward block like in Transformers O Possible extensions for deeper models are discussed in Appendix
3.2 NCDE-based FWPs
We assume availability of differentiable control signal (tt), whose first order derivative is denoted by x' (t). Given the NCDE formulation of Eq. the most straight-forward approach obtain a CT Fast Weight Programmer is to extend the dimensionality of the recurrent hidden state, i.e., we introduce a parameterised function Fo which maps a matrix W (t) E IR X to a third-order tensor Fo(W(()) E Ride x thes Xoaa
Here we present models based on NCDEs (reviewed in Sec. 2.1
w(+) = ( + '" Fo (s) = W + L (s))aa) -=
(22)
However, this approach is obviously not scalable since the input and output dimensions (dout x dkey and d daey d'.,) of Fo can be too large in practice. more tractable CDE-based approach can
II practice we also apply element-wise activation functions to query/key/value vectors where appropriate, which we omit here for readability. We refer Appendix forr further details.
5

---

be obtained by providing and/or x' to the vector field:
( ((o) + [ (s), x (s)) (1)))((())) (s)ds
(23)
s=to
While this equation still remains CDE because of the multiplication from the right by dx = x' (s)dss the additional inputs to the vector field offer way of making use of various learning rules, as in the case of direct NODE approach above (Sec. 3.1) To be specific, either a and x' or only a'' is required in the vector field to obtain these tractable CDEs. Here we present the version which uses both x and x' The resulting vector fields for different cases are:
Whar(s) @ (s)
Hebb
Fo (w),, 2()) o' ())) -
(Wax(s)) ( W,,'(s)) @ W (Wrx(s) W(s) (s)) @ Ws
Oja
Delta
(24)
As can be seen above, the use of CDEs to describe continuous fast weight learning rule thus naturally results in a key/value memory where x' is used to generate either key or value vectors. Because of the multiplication from the right by x', the role of x' changes depending of learning rule: x' is used to generate the key in the Delta case but the value vector in the case of Oja. In the case of Hebb, the choice made in Eq. 24 of using x: for keys and x' for values is arbitrary since Eq. [24jis symmetric in terms of roles of keys and values (see an ablation study in Appendix [C.2 for the other case where we use x' to generate the key and for the value). The querying operation (analogous to Eqs. 20 for the direct NODE case) is also modified accordingly, depending on the choice of learning rule, such that the same input (æc or x') used generate both key and query:
the choice
(T) T (T) Hebb and Oja (T) W (T) Delta
(T)
(25)
Note that since the proposed vector field Fo (W (s))))sss), 2'))) (()) is more general than the one used in the original NCDE Fe (W))) a''))))))))) (s), any theoretical results on the CDE remain valid (which, however, does not tell us anything about the best choice for its exact parameterisation).
3.3 ODE-RFWP and Latent ODE-RFWP
The main focus of this work is the setting of Kidger et al. 4 where we assume the existence of some control signal æ (Sec. 3.1 and 3.2 above). However, here we also show way of using FWPs in the third/last case presented in Sec. 2.1 where no control (t) is available (or can be constructed), i.e., we only have access to discrete observations ( Here we cannot directly define the vector field involving continuous transformations using the inputs. We follow the existing approaches (ODE- RNN or Latent ODE; Sec. 2. which use two separate update functions: A discrete recurrent state update is executed every time new observation is available to the model, while continuous update using an autonomous ODE is conducted in between observations. Unlike with standard recurrent state vectors, however, it is not practical to autonomously evolve high-dimensional fast weight matrices We therefore opt for using a Recurrent FWP (RFWP) [10] and combine it with an ODE:
Up
ODESolve fo. -nn-1,1 (n-1,tn)
(26)
hn W. n
FWP(( u,,), Wn-1;02)
(27)
where we keep the fast weight learning rule itself discrete (Eq. 27 but evolve the recurrent state vector un using an ODE (Eq. (6) such that the information to be read/written to the fast weight matrix is controlled by variable which is continuously updated between observations. We refer to this model as ODE-RFWP and its variational variant as Latent ODE-RFWP.
Since we focus on continuous-time learning rules, the case above is not of central interest as the learning rule remains discrete here (Eq, 27). Nevertheless, in Appendix C.3 we also provide some experimental results for model-based reinforcement learning settings corresponding to this case.
4 "The equations for the version using only x' can be obtained by replacing æ by x' in Eq. 24
We provide an
ablation in Appendix C.2 As side note, also obtain the equation for the CDE using only x by replacing by æ' in Eq. [8 for the direct NODE case.
'Such an approach would require a computationally expensive matrix- to matrix transforming NN, whose scalability is limited.

---

Table 1: Aceuracy (%) on the Speech Commands classification task and AUC (x 102) on the PhysioNet Sepsis prediction task. PhysioNet has two cases: with (OI) without (no-OI) observa- tional intensity (see text for details). Numbers marked by * are taken from Kidger et al. Mcan and standard deviation (std) are computed over runs.
PhysioNet Sepsis
Type
Model
Speech Commands
O
no-O1
Direct NODE
GRU-ODE [4]
47.9 (2.9)
85.2 (1.0)
77.1 (2.4)
Hebb
82.8 (1.1)
90.4 (0.4)
82.9 (0.7)
Oja
85.4 (0.9)
88.9 (1.4)
82.9 (0.5)
Delta
81.5 (3.8)
89.8 (1.0)
84.5 (2.9)
CDE
NCDE 4
89.8 (2.5)
(0.6
77.6 (0.9)
Hebb
89.5 (0.3)
89.9
85.7 (0.3)
Oja
90.0 (0.7)
91.2 (0.4)
85.1 (2.5)
Delta
90.2 (0.2)
90.9 (0.2)
84.5 (0.7)
4 Experiments
We consider three datasets covering three types of time series which are regularly sampled (Speech
Commands [51]), irregularly sampled with partially missing features (PhysioNet Sepsis [52]) ), or very long (Eigen
(EigenWorms Worms [53)) D. We compare the proposed direct NODE and CDE based FWP models
(Sec. 3.1 & 3.2) to NODE baselines previously reported on the same datasets 4 [36. Appendix B provides further experimental details including hyper- saper--arameters..
is a single word speech recognition task. The datapoints are regularly sampled, and the sequence lengths are relatively short (< 160 frames), which makes this task a popular sanity check. Following prior work on NCDEs 4 we use 20 mel frequency cepstral coefficients as speech features and classify the resulting sequence to one out of ten keywords, The middle column of Table shows the results. The table is split into the direct NODE (top) and CDE (bottom) based approaches. We first observe that among the direct NODE approaches, all our FWPs largely outperform (2 80% accuracy) the baseline GRU-ODE performance of 47.9% (the best direct NODE baseline from Kidger et al. 4).. This demonstrates that with a good parameterisation of the vector field, the direct NODE approach hand, all CDE-based approaches yield similar performance. We also only see slight differences in terms of performance among different learning rules, without a clear winner for this task. This may indicate that the ordinary nature of this task (regularly sampled; short sequences) does not allow for differentiating among these CDE models, including the baseline.
Speech Commands. The Speech Commands [5]
achieve competitive performance. On the other
PhysioNet Sepsis. The PhysioNet Sepsis is a dataset of the sepsis prediction task from the PhysioNet challenge 2019 (52]. This is again dataset used by Kidger et al. 4 to evaluate NCDEs. The task is a binary prediction of sepsis from a time series consisting of measurements of 34 medical features (e.g., respiration rate) of patients' stays an ICU. Each sequence additionally labelled by five static features of the patient (e.g., age) which are fed to the model to generate the initial state of the ODE. Sequences many entries are missing, which makes this task challenging. without the so-called observation intensity information (denoted as "OI" and "no-Ol") which is one extra input feature indicating each observation's 's time stamp (providing the models with information on measurement frequencyy. This distinction is important since the prior work 4 has reported that existing ODE/CDE based approaches struggle with the no-OI case of this task. Following the previous work, we report the performance in terms of Area Under the ROC Curve (AUC). The right part of Table shows the results. We obtain large improvements in the no-10 case (from 77.6 to 85.7% for the CDEs and from 77.1 to 84.5% for the direct NODEs), while also obtaining small improvements in the Ol case (from 85.2 to 90.4% for direct NODEs, and from 88.0 to 91.2% for CDEs). The no-Ol performance of our models also comparable to the best overall performance reported by Kidger [38]: 85.0 % (1.3) achieved by GRU-D (54). This demonstrates the efficacy of
relatively short (S 72 frames) but datapoints
irregularly sampled and comes in two versions: with and
7

---

Table Classification Accuracy (%) on the
task. Numbers marked by * are taken from Morrill et al. [37]. Mean and standard deviation (std) are computed over 5 runs. "Sig- Depth" indicates the depth of the signature (with this number equal to 1, an RDE is reduced to a CDE). To
facilitate comparisons to prior work [37,, we also add the column "Step" indicating the sequence
down-sampling factor (even if we fix it to the best value [3 4).
Model
Sig-Depth
Step
Test Acc. [%]
NRDE 137
2
4
83.8
(3.0)
Hebb
2
4
45.6 (5.9)
Oja
46.7 (7.5)
Delta
87.7 (1.9)
NCDE [37
1
4
66.7 (11.8)
Hebb
1
4
41.0 (6.5)
Oja
49.7
(9.9)
Delta
91.8 (3.4)
CT FWP model variants for handling irregularly sampled data with partially missing features even in the case without frequency information. Differences between various learning rules are rather small again. In some cases, we observe performance to be very sensitive to hyper-paramettrs. For example, the best Oja-CDE configuration achieves 85. % (2.5) with a learning rate of 6e-5, while this goes down to 79.6% (4.7) when the learning rate is changed to 5e-5.
EigenWorms, The Eigen Worms dataset (which is part of the UEA benchmark [53) is a 5-way classification of roundworm types based on time series tracking their movements. To be more specific, motions of worm are represented by six features corresponding to their projections to six template movement shapes, called "eigenworms." While this dataset contains only 259 examples, is notable for its very long sequences (raw sequence lengths exceed 17 K) and long-span temporal dependencies 337 55 (66. We use the same train/validatiinn/est split ratio as the prior work [ which reports Neural RDEs (NRDEs) as achieving the best NODE model performance on this dataset. The equations of our CT FWPs for the RDE case can be straightforwardly obtained by replacing the input æ in Eqs. 18 19 of the direct NODE formulation
of the NCDEs) (o and x'' in Eqs. 23 25
by the corresponding log-signatures. Table 2 sampling rate which is fixed to 4 for which the prior work (37 reports the best NRDE and NCDE performance. "Sig-Depth" denotes the depth of the log-signature (the deeper, the more log-signature terms we take into account, thus ending up with larger input feature vector; we refer to the original paper (37] for further details). We consider two values for this parameter: 1 and 2. When set to 1, the input feature contains only the first derivative x' (s) and thus the NRDE is reduced to an NCDE (with controls constructed via linear interpolation). We take the best NCDE performance from Morrill et al. 37 as the 1epth-1 1 baseline. Morrill et al. 37 report the best overall performance for the depth-2 NRDE (depth-2 baseline in our table). In both cases, we first note a large performance gap between models with different learning rules. While the naive Hebb and Oja based models struggle with this very long sequence processing (sequence length still exceeds 4 K with a down-sampling step size of 4), the Delta rule performs very well. This confirms the prior result in the discrete-time domain [9] which motivated the Delta rule design by its potential for handling long sequences (we refer to prior work for further explanations). Since its performance on other tasks comparable the one of Hebb and Oja variants, the Delta rule is a natural default choice for parameterising the CT FWPs
shows the results, where "Step" denotes the time sub-
In both the 1 1 and 2 2 cases, we obtain large improvements compared to the respective baselines. It is counter-intuitive to find certain depth-2 models underperforming their depth-1 counter- parts, but this trend has also been observed in the original NRDEs (36.. Our best overall performance is obtained in the depth-1 case: 91.8 % (3.4) exceeds the previous best NRDE based model per-
'Conceptually these two approaches are equivalent: the direct NODE and NCDE coincide here. In practice, there can be a subtle difference due to an implementation detail. The direct NODE approach can apply layer normalisation to the input fed to both key and value projections (as they are both inside the vector field). In this case (as in implementation), the corresponding NCDE formulation we obtain is based on the normalised input.
8

---

formance of 83.8 % (3.0) [36. This almost matches the state-of-the-rd accuracy of 92.8 % (1.8) reported by Rusch al. [56] (using an ODE-inspired discrete- time model). Our model's performance variance is high (best single seed performance is 97.4% while the standard deviation is 3.4). The wall clock time is similar for our best model (last row Table 2 and the NRDE baseline (36s/epoch on a GeForce RTX 2080 Ti) and their sizes are comparable (87 K vs. 65 K parameters respectively).
5 Discussions
Scalability Advantage Compared to Standard NCDEs. In addition to the good empirical results shown above, our FWP approach also addresses important limitation of existing NCDEs 4 their scalability in terms of model size. The vector field in standard NCDEs (Eq. 5 requires an NN F6 which takes a vector (s) E Ed an input to produce matrix of size dx This can be very challenging when din or/and d is large. Actually, the same bottleneck is present in the weight generation of FWPs [7]. The use of outer products can remediate this issue in discrete FWPs as well as in CT FWPs: the computations in our FWP-based NODE/NCDEs only involve "first- order" dimensions (i.e., no multiplication between different dimensions, such as du,) for NN outputs. This can scale well with increased model size, making feasible larger scale tasks infeasible for existing NCDEs. On the other hand, Kidger et 4 report that using outer products (in their "Sec. 6. 1 on limitations")) in standard NCDEs does not perform well. Why do outer products work well in our models but not in the original NCDEs? The answer may be simple. In the original NCDEs (Eq. 5). multiplications occur at each (infinitesimal) time step between the generated rank-one weight matrix (h(s)) and x' (s) before the sum. All these transformations are thus of rank one while we expect expressive transformations to be necessary to translate x' (s) into changes in the state space. In contrast, in our CT FWPs, the ODE only parameterises the weight generation process of another net, and thus the rank-one matrices never used in isolation: they summed up over time (Eq. 18 or 23) to form an expressive weight matrix which is only then used for matrix multiplication (Eq. 20
or 25 The proposed EWP-NODE/NCDES thus offer scalable alternatives to existing NCDEs, also yielding good empirical performance (Sec.
Importance of Memory Efficient Backpropagation for FWPs. Memory efficiency of contin- uous adjoint backpropagation may be not so important for standard NCDEs of state size (d)), but is crucial for FWPs of state size (dd)) which can quickly become prohibitive for long sequences, as naive backpropagation stores all states used in the forward pass. Prior works on discrete FWPs [9, 57 [I0 solve this problem by custom memory- memory-effcient implementation. Here, the continuous adjoint method naturally addresses this problem.
Limitations. Our ablation studies and configuration/rcchitcttuu From the NODE perspective, other parameters may further improve performance or alleviate performance variability/stabiiity issues observed in some cases (Sec. [4) For example, we use the numerical solver configurations of the baselines (see Appendix B) without tuning them. Similarly, we use natural cubic splines to construct the differentiable control signals for NCDEs in the Speech Commands and PhysioNet Sepsis tasks, following the original NCDE paper [4] Morrill et al. (36] report performance enhancements by improving the corresponding interpolation methods (c.g.. 93.7% on Speech Commands). Such further optimisation is not conducted here.
tuning focus on optimising the model
Generally speaking, real benefits of using continuous-tim time sequence processing models are yet to be proved. While we achieve improvements over the best existing NODE/NCDE models on multiple datasets, discrete-time models tailored to the corresponding problem still perform as well or even better than our improved CT models (e.g., Rusch et al. (56) for Eigen Worms and Che et al. [54] for PhysioNet Sepsis; see Sec.
Related Work on Parameter/Weight ODEs. Other works use ODEs to parameterise the time- evolving weights of some model. However, they are limited to autonomous ODEs (i.e., no external control æ is involved). Zhang et al. [58] and Choromanski et al. [59] study coupled ODEs where one ODE is used for temporal evolution parameters of the main Neural ODE The scope of these two works is limited to autonomous ODEs corresponding to continuous-depth residual NNs with different parameters per depth. Deleu et al. (60) consider an ODE version of a gradient descent learning process for adaptation, but also formulated autonomous ODE, In contrast, our focus is really on

---

sequence processing where the model continuously receives external controls æ and translates them into weight changes another network.
Conclusion
We introduced novel continuous- sequence processing neural networks that learn to use sequences of ODE-based continuous learning rules as elementary programming instructions to manipulate short- term memory rapidly changing synaptic connections of another network. The proposed models are continuous-time counterparts of Fast Weight Programmers and linear Transformers, Our new models experimentally outperform by large margin existing Neural ODE based sequence processors on very long or irregularly sampled time series. Our Neural ODE/CDE based FWPs also address the fundamental scalability problem the original Neural CDEs, which is highly promising for future applications of ODE based sequence processors to large scale problems.
7
Acknowledgements
We would like thank Kidger et al. 4 Morrill al. 37 and Du et al. [61] for their public code. This research was partially funded by ERC Advanced grant no: 742870, project AlgoRNN, and by Swiss National Science Foundation grant no: 200021 1e2_1_1996,,,_1 project NEUSYM. We are thankful for hardware donations from NVIDIA and IBM. The resources used for this work were partially provided by Swiss National Supercomputing Centre (CSCS) project s1 145 and s1154.
References
[1] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 6572-6583, Montréal, Canada, December 2018.
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 770- 778, Las Vegas, NV, USA, June 2016.
[3] Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. the Deep Learning workshop at Int. Conf. on Machine Learning (ICML), Lille, France, July 2015.
[4] Patrick Kidger, James Morrill, James Foster, and Terry J. Lyons. Neural controlled differential equations for irregular time series. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual Only, December 2020.
[5] Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):17-2211 1990.
[6] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 5998-6008, Long Beach, CA, USA, December 2017.
[7] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Technical Report FKI-147-91, Institut für Informatik, Technische Universität München, March 1991.
[8] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139, 1992.
[9] Imanol Schlag, Kazuki trie, and Jürgen Schmidhuber. Linear Transformers are secretly fast weight programmers. In Proc. Int. Conf. on Machine Learning (ICML),, Virtual only, July 2021.
[10] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber: Going beyond linear transformers with recurrent fast weight programmers. In Proc. Advances Neural Information Processing Systems (NeurIPS), Virtual only, December 2021.
10

---

[11] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proc. Int. Conf. on Machine Learning (ICML), Virtual only, July 2020.
[12] Ken-ichi Funahashi and Yuichi Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neurul networks, 6(6):801-806, 1993.
[13] Alan Lapedes and Robert Farber. Nonlinear signal processing using neural networks: Prediction and system modelling. Technical Report No. LA-UR-87-2662, 1987.
[14] Fernando J Pineda. Generalization of back-propagation recurrent neural networks. Physical review letters, 59(19)2222, 1987.
[15] Barak A Pearlmutter. Learning state space trajectories in recurrent neural networks. Neural Computation, 1(2):263-200 269, 1989.
[16] Masa-aki Sato and Yoshihiko Murakami. Learning nonlinear dynamics by recurrent neural. In Some Problems on the Theory of Dynamical Systems Applied Sciences- Proceedings of the Symposium, volume 10, page 49, 1991.
[17] Ramiro Rico-Martinez, K Krischer, Ioannis Kevrekidis, MC Kube, and JL Hudson. Discrete- vs. continuous-time nonlinear signal processing of Cu electrodissolution data. Chemical Engineering Communications, 118(():25-48, 1992.
[18] Erkki Oja. Simplified neuron model principal component analyzer. Journal of mathematical biology, 15(3):267-27,, 1982.
[19] Karl Pearson. LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559-572, 1901.
[20] Harold Hotelling. Analysis of complex of statistical variables into principal components. Journal of educational psychology, 24(6):417, 1933.
[21] Erkki Oja and Juha Karhunen. On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. Journal of mathematical analysis and applications, 106 (1):69-44, 1985.
[22] Erkki Oja. Neural networks, principal components, and subspaces. International journal of neural systems, 1(01):61-68, 1989.
[23] Mark D Plumbley. Lyapunov functions for convergence of principal component algorithms. Neural Networks, 8(1):11-22) 1995.
[24] Kurt Hornik and C-M Kuan. Convergence analysis of local feature extraction algorithms. Neural Networks, 5(2): 240, 1992.
[25) Terence D Sanger. Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural networks, 2(6):459 473, 1989.
[26] John L.. Wyatt Jr. and Ibrahim M. Elfadel. Time-domain solutions of Oja's equations. Neural Computation, 7(5):915-922, 1995.
[27] Jean-Claude Fort and Gilles Pages, Convergence of stochastic algorithms: From the Kushner- Clark theorem the Lyapounov functional method. Advances in applied probability, 28(4): 1072-1094, 1996.
[28] Weinan E. A proposal on machine learning via dynamical systems. Mathematics and Statistics, 1(5):1-11, March 2017.
Communications in
[29] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, December 2017.
[30] Eldad Haber, Lars Ruthotto, Elliot Holtham, and Seong-Hwan Jun. Learning across scales multiscale methods for convolution neural networks. Proc. AAAI Conf. on Artificial Intelligence, pages 3142-3 3 148, New Orleans, LA, USA, February 2018.
11

---

[31] Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Reversible architectures for arbitrarily deep residual neural networks. Proc. AAAI Conf. on Artificial Intelligence, pages 281 1-2818, New Orleans, LA,, USA, February 2018
[32] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In Proc. Int. Conf. on Machine Learning (ICML), pages 3282-3291, Stockholm, Sweden, July 2018.
[33] Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert. Multi-level residual networks from dynamical systems view. In Int. Conf. on Learning Representations (ICLR), Vancouver, Canada, April 2018.
[34] Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, and Faustino J. Gomez. NAIS-Net: Stable deep networks from non-autonomous differential equations. In Proc. Ad- vances in Neural Information Processing Systems (NeurIPS), pages 3029-3039, Montréal, Canada, December 2018.
[35] Liev S. Pontryagin, VG Boltyanskii. RV Gamkrelidze, and EF Mishchenko. LS Pontryagin Selected Works: The Mathematical Theory of Optimal Processes. 1962.
[36] James Morrill, Patrick Kidger, Lingyi Yang, and Terry Lyons. Neural controlled differential equations for online prediction tasks. Preprint arXiv:2 106. 1028, 2021.
[37] James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In Proc. Int. Conf. on Machine Leurning (ICML), pages 7829- 7838, Virtual only, July 2021.
[38] Patrick Kidger. On Neural Differential Equations. PhD thesis, Mathematical Institute, University of Oxford, 2021.
[39] Jürgen Schmidhuber. Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets. In International Conference on Artificial Neural Networks (ICANN), pages 460-463, Amsterdam, Netherlands, September 1993.
[40] David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. In Int. Conf. on Learning Representa- tions (ICLR), Toulon, France, April 2017.
[41] Yulia Rubanova, Tian Qi Chen, and David Duvenaud. Latent ordinary differential equations for irregularly-amppled time series. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 5321-5331 Vancouver, Canada, December 2019.
[42] Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE- Continu- ous modeling of sporadically-ooserved time series. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 7377-7388, Vancouver, Canada, December 2019.
[43] Donald Olding Hebb. The organization of behavior, a neuropsycholocigal theory. A Wiley Book in Clinical Psychology, 62:78, 1949.
[44] Chi-Ning Chou and Mien Brabeeba Wang. ODE-inspired analysis for the biological version of Oja's rule in solving streaming PCA. In Proc. Conf. on Learning Theory (COLT), pages 1339- 1343, Virtual only, July 2020.
[45] Huaguang Zhang, Zhanshan Wang, and Derong Liu. A comprehensive review of stability analysis of continuous taaiiuuuus-tmeettte recurrent neural networks. IEEE Transactions on Neural Networks and Learning Systems, 25(7): 2014.
[46] Adeline Fermanian, Pierre Marion, Phanllpilippe Vert, and Gérard Biau. Framing RNN as a kernel method: A neural ODE approach. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2021.
[47] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 6626-6637, Long Beach,
CA, USA, December 2017.
12

---

[48] Mark A. Aizerman, Emmanuil M. Braverman, and Lev I. Rozonoer. Theoretical foundations of potential function method in pattern recognition. Automation and Remote Control, 25(6):
917-936, 1964.
[49] Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. The dual form of neural networks revisited: Connecting test time predictions training patterns via spotlights of attention. In Proc. Int. Conf. on Machine Learning (ICML), Baltimore, MD, USA, July 2022.
[50] Bernard Widrow and Marcian E. Hoff. Adaptive switching circuits. In Proc. IRE WESCON Convention Record, pages 96-104, Los Angeles, CA, USA, August 1960.
[51] Pete Warden. Speech commands: A dataset for limited- speech recognition. Preprint arXiv:8004. 03209, 2018.
[52] Matthew A. Reyna, Christopher Josef, Salman Seyedi, Russell Jeter, Supreeth Shashikumar, M. Brandon Westover, Ashish Sharma, Shamim Nemati, and Gari D. Clifford. Early prediction of sepsis from clinical data: the PhysioNNercoopputigg in cardiology challenge 2019. In Proc. Computing in Cardiology (CinC), pages 1-4, Singapore, September 2019.
[53] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018. Preprint 1xXix:811.00077. 2018.
[54] Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
[55] T Konstantin Rusch and Siddhartha Mishra. UnICORNN: A recurrent model for learning very long time dependencies. In Proc. Int. Conf. on Machine Learning (ICML), pages Virtual only, July 2021.
[56] T Konstantin Rusch, Siddhartha Mishra, N Benjamin Erichson, and Michael W Mahoney. Long expressive memory for sequence modeling. In Int. Conf. on Learning Representations (ICLR), Virtual only, April 2022.
[57] Kazuki Irie and Jürgen Schmidhuber. Training and generating neural networks in compressed weight space. In ICLR Neural Compression Workshop, Virtual only, May 2021.
[58] Tianjun Zhang, Zhewei Yao, Amir Gholami, Joseph E. Gonzalez, Kurt Keutzer, Michael W. Mahoney, and George Biros. ANODEV2: A coupled neural ODE framework. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 5152-5162, Vancouver, Canada, December 2019.
[59] Krzysztof Marcin Choromanski, Jared Quiney Davis, Valerii Likhosherstov, Xingyou Song, Jaan-Jacques E. Slotine, Jacob Varley, Honglak Lee, Adrian Weller, and Vikas Sindhwani. Ode to ODE. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2020.
[60] Tristan Deleu, David Kanaa, Leo Feng, Giancarlo Kerg, Yoshua Bengio, Guillaume Lajoie, and Pierre-Luc Bacon. Continuous-time meta-learning with forward mode differentiation. In Int. Conf. on Learning Representations (ICLR), Virtual only, April 2022.
[61] Jianzhun Du, Joseph Futoma, and Finale Doshi-Velez. Model-based reinforcement learning for semi-Markov decision processes with neural ODEs. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, 2020.
[62] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. Preprint 1607::660.00455 2016
[63] Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. Language modeling with deep Transformers. In Proc. Interspeech, pages 3905 3909, Graz, Austria, September 2019.
[64] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In JEFF/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), pages 5026-5033, Vilamoura, Portugal, October 2012. IEEE.
13

---

[65] Sahil Sharma, Aravind S. Lakshminarayanan, and Balaraman Ravindran. Learning to repeat: Fine grained action repetition for deep reinforcement learning. In Int. Conf. on Learning Representations (ICI.R) Toulon, France., April 2017.
[66] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. 1994.
[67] Arthur George Richards. Robust constrained model predictive control. Phl) thesis, Mas- sachusetts Institute of Technology, 2005.
[68] Eduardo F Camacho and Carlos Bordons. Model predictive control. 2013.
[69] David Q Mayne, James B Rawlings, Christopher V Rao, and Pierre OM Scokaert. Constrained model predictive control: Stability and optimality. Automatica, 36(6):789 814,988114100 2000.
[70] Timothy P. Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Int. Conf. on Learning Representations (ICLR), San Juan, Puerto Rico, May 2016.
[71] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proc. Int. Conf. on Machine Learning (ICML), Beijing, China, June 2014.
[72] Richard S Sutton. Temporul Credit Assignment in Reinforcement Learning. PhD thesis, University of Massachusetts Amherst, 1984.
[73] Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 1008-1014, Denver, CO, USA, November 1999.
[74] Jan Peters, Sethu Vijayakumar, and Stefan Schaal. Natural actor-critic. In Proc. European Conference Machine Learning (ECML), pages 280-291, Porto, Portugal, October 2005.
[75] Çagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. Continuous-time model-based reinforcement learning. In Proc. Int. Conf. on Machine Learning (ICML), Virtual only, July 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We comment on relatively
high variance in the final performance in two sub-paragraphs "PhysioNet Sepsis" and "Eigen Worms" under the experimental section.
(c) Did you discuss any potential negative societal impacts of your work? [No] because we do not see any such impacts for this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The code is included in the supplemental material and it will be public if the paper is accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We report means and standard deviations obtained for five seeds.
14

---

(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Some essential numbers are reported in the main text. There is more in the appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See details in the appendix.
(b) Did you mention the license of the assets? [Yes] Licence files included in the code/supplemental material.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Yes, our own code.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] We use standard public benchmark datasets.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "__VurSc46CQ", "reviewer": "Reviewer_B6yz", "summary": "The authors propose several parameterisation of the vector field in neural ODEs/CDEs/ODE-RNNs based on FWP and learning rules.", "questions": "I have direct questions for the authors. I have made some suggestions above; to highlight them:\n\n1. I think the experiments could be improved by moving beyond just the CDE/RDE case.\n2. The choice of numerical method and software should definitely be discussed.", "limitations": "There are no ethical concerns with this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength: topic**\n\nThere has been relatively little convincing work on good architectures for RNNs, NCDEs, etc. This work addresses a meaningful problem.\n\n**Strength: originality**\n\nTo the best of my knowledge the ideas presented in this paper (combining NCDEs with FWP, etc.) are new.\n\nThe scalability, as compared to the original formulation of neural CDEs, is an important contribution.\n\n**Strength: literature review**\n\nThis work ties together many individual lines of work -- neural CDEs, neural RDEs, long sequence modelling, learning rules, FWP, etc. -- and so far as I can determine does an excellent job providing complete references for all of them.\n\n**Strength: experiments**\n\nThe experiments look to have been carefully conducted. They go into a great deal of detail. Moreover the results obtained demonstrate meaningful improvements on existing benchmarks.\n\n**Strength: presentation**\n\nThe prose is carefully worded and precise. I enjoyed reading this paper.\n\n**Weakness: experiments**\n\nI believe the experiments are overly-focused on neural CDE and neural RDE comparisons. Whilst these are the natural choice given the framing of the paper, I think it would improve the paper to include more quantitative comparisons against other works, e.g. [0], [Ref. 34, 46, 47 from the paper]. Indeed there have been quite a few papers proposing novel architectures for RNNs/etc.\n\n**Weakness: presentation**\n\nAt least for me -- I am largely unfamiliarS with the work on learning rules, FWP etc. -- I had to wait until the experimental section to determine that what is being proposed is essentially a new architecture, for the purposes of supervised learning etc. I would suggest explicitly stating this kind of thing up-front.\n\n**Weakness: \"the adjoint sensitivity method\" (line 66)**\n\nThe term \"adjoint\" has become heavily overloaded. In the current literature is is often used to mean specifically optimise-then-discretise, but in older literature it was often used to mean specifically discretise-then-optimise. Following [1, Remark 5.5] I recommend avoiding it altogether.\n\nIn passing, since backpropagation is performed via OtD, then the authors may find speedups by utilising the technique of [2].\n\n**Weakness: numerical/software**\n\nThe choice of computational framework, and choice of numerical solvers, seems to go entirely undiscussed. (I did not see it mentioned in the appendix either.)\n\nThis is a meaningful limitation: the choice of numerical solver, its choice of tolerance, etc., can greatly affect results.\n\n**Minor issues**\n\nLine 33: I wouldn't use the word \"ancient\". Some of us were around back then!\n\nLine 78: I would argue that natural cubic splines should not be mentioned here. As [Ref. 31 of this paper] shows, there are now much better alternatives, so I think it would be better to avoid discussing natural cubic splines altogether. (e.g. they are not even implemented in [3])\n\n**Commentary**\n\nLine 85: I'm not sure I completely agree with this analysis. As per the next heading (\"Second\") then the absolute value of $x$ may be treated as an input as well, by recording and replaying it. In practice it's not been demonstrated that this actually occurs in practice, so I think it possible that this paper's side-note may be true in practice, if not in theory.\n\nI quite like the idea of directly providing $x$ or $x'$ to the vector field, in addition to leaving $x'$ outside, as in equation (23). Most works have usually framed the inside/outside placement of $x$ or $x'$ as a dichotomy.\n\n**References**\n\n[0] Coupled Oscillatory Recurrent Neural  Network (coRNN): An accurate and (gradient) stable architecture for  learning long time dependencies, Rusch and Mishra, ICLR 2021\n\n[1] On Neural Differential Equations, Kidger, Doctoral Thesis, University of Oxford 2021\n\n[2] \"Hey, that's not an ODE\": Faster ODE Adjoints via Seminorms, Kidger et al, ICML 2021\n\n[3] Diffrax, https://github.com/patrick-kidger/diffrax", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3C4nr_B30M", "reviewer": "Reviewer_95a7", "summary": "This work introduces variants of Fast Weight Programmers, where the Fast Weight matrix W evolves via neural DEs, with vector field structures inspired by learning rules (namely Hebb, Oja, and Delta). These can be applied to continuous learning tasks. \n\nThe structure of the neural DEs here fall under a subclass considered by Kidger et al in the paper introducing neural CDEs (that is, the neural CDEs subsume these methods). However, this paper demonstrates empirically that the specific restriction to these models seem to perform well compared with optimising over the broader class of neural CDEs, as well as providing scalability benefits.", "questions": "In Section 3.3, it is assumed that the continuous control is not available and only discrete observations are accessible. In practice, this is almost always the case, in the paper by Morrill et al. [31], there is an extensive discussion on the construction of a continuous control path from the discrete observations. Perhaps the authors can highlight and clarify, when might one choose to take the discrete observations and apply methods in Section 3.3 rather than apply interpolation and use Neural CDEs?", "limitations": "The authors have discussed that some of their results have rather large variances, in particular Delta achieved the highest single seed performance for sepsis no-OI, but on average Hebb performed best in this case.\n\nAs discussed in weaknesses, the authors do not discuss comparisons with other baselines from non-ODE based neural networks. A discussion on where this work sits in the broader literature and any limitation of application of these new networks in broader problems would be beneficial.\n\nNegative societal impact not applicable for this paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a very nicely written paper, which presents the work done in a clear and concise manner. The background and literature is explained well. There is full transparency, the code is available for validation and reproduction of the results.\n\nAn observation is that this is pooling together a few ideas from different areas, so the improvement is more incremental, however this is not necessarily a weakness, and the empirical results and benefits seem to be good.\n\nIn terms of weaknesses, although it is not realistic to be comparing with all models out there, this paper does not discuss that in Kidger et al, the model that performed best for sepsis no-OI is in fact GRU-D, with a performance of 85% plus/minus 1.3, which is superior to the best performance of the three models introduced in this paper.\n\nFurthermore, most of the comparison here is with Kidger et al. (2020), which has been superseded by Morrill et al. (2021) in Neural CDEs for online prediction tasks. It would be good for come comments and comparison with the results in the latter.\n\nSome minor issues:\n\"ancient\" for the 80s/90s can be slightly offensive!\nEqn (27) should be RFWP?\nTable 1 should be AUC rather than AUR.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-kZuL0AUMz3", "reviewer": "Reviewer_mFC6", "summary": "The paper presents a continuous time version for Fast Weight Programmers family of sequence classification models. It presents continuous time analogues to Hebbian learning rules. It introduces continuous time versions of multiple FWPs variants. The continuous time versions are empirically tested three standard sequence classification tasks, and compared to other sequence classifiers. In the appendix, the method is also applied to two basic RL problems.\n", "questions": "### Experiments:\n\nIn the experiments, why were these particular baselines chosen? Why were only ODE based methods chosen as comparison? Could a non-ode method be included in these problems?\n\nTables 1 & 2:  Are the models comparable? What are the parameter counts of each row? What are the runtimes of training and inference? What are the architecture hyperparameters? \n\nTable 2: Are the results for “Direct NODE” or “NCDE”?\n\nLine 264: Did the authors perform the experiment comparing NODE and NCDE equations for this problem? It is not clear from the text or results.\n\n### Usage of “memory efficient” adjoint \n\nAs mentioned, it is not clear that the checkpoint-free adjoint method can be used to these equations. Are the equations of the continuous FWP models reversible, and guaranteed to remain so?\n\nIs the adjoint method only applied on the fast weight learning rule, or the entire network?\n\nDid you compare using the checkpoint-free adjoint method vs. regular backpropagation in your examples? \n\nWere the baseline methods also able to use the “memory saving” adjoint method for training?\n\n### Clarifications\n\nLine 211: Why is it not practical? Is it because of the number of parameters, or computational cost, something else?\n\nLine 217: Is the Latent ODE RFWP used anywhere in this work?\nLine 218: This last sentence is confusing. “While this case” Which is “this case”? Does this paragraph mean to say that all of section 3.3 is not of central interest? \n\nLine 263: What is an RDE?\n\n### Minor\n\nThis is a very superficial comment, but the paper title makes it difficult to understand what the paper is about. The phrase “Learning to Program” suggests a different field of program synthesis or general optimization algorithms, which does not fit. The title could be simplified.\n", "limitations": "The authors focus on how their method mitigates limitations of the original family of methods. The authors have some discussions about the cost limitations of the family methods, and hypotheses on the mechanism by which their method performs well with low-rank methods when previous variants do not.\n\nThere are no potential negative societal impacts that are particular to their work.\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is overall well presented with a thorough explanation. The continuous analogous of FWPs is an original contribution to the literature of neural ODEs. A particular novelty is the consideration of Hebbian learning methods.\n\nA wide variety of experiments are performed, which is a strength of the paper. However, the main weakness of this paper is that each experiment only has one baseline method as comparison, and does not have enough details, which makes it unclear if the proposed method is significantly better than existing methods. \n\nThis is a minor aspect of the work: the “memory saving” adjoint method is mentioned as a “pro” of their method. A questionable claim is “Memory Efficient Backpropagation for FWPs”. The “memory-saving” adjoint method only works if the equation is reversible (not all ODEs can be integrated in both directions c.f. heat equation). Otherwise, checkpointing of the forward pass of the ODE is required.   (See, e.g. Zhuang et al. “Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE'' 2020). Additionally, the baseline methods could also use it (with the same error) so it is not a pro unique to their method. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept