Learning PDE Solution Operator for Continuous Modeling of Time-Series
Authors: Yesom Park
Jaemoo Choi
Changyeon Yoon
Chang hoon Song
Myungjoo Kang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural ODEs
Partial differential equations
Neural operators
Time-series

Abstract: Learning underlying dynamics from data is important and challenging in many real-world scenarios. Incorporating differential equations (DEs) to design continuous networks has drawn much attention recently, the most prominent of which is Neural ODE. Most prior works make specific assumptions on the type of DEs or restrict them to first or second-order DEs, making the model specialized for certain problems. Furthermore, due to the use of numerical integration, they suffer from computational expensiveness and numerical instability. Building upon recent Fourier neural operator (FNO), this work proposes a partial differential equation (PDE) based framework which improves the dynamics modeling capability and circumvents the need for costly numerical integration. FNO is hard to be directly applied to real applications because it is mainly confined to physical PDE problems. To fill this void, we propose a continuous-in-time FNO to deal with irregularly-sampled time series and provide a theoretical result demonstrating its universality. Moreover, we reveal an intrinsic property of PDEs that increases the stability of the model. Several numerical evidence shows that our method represents a broader range of problems, including synthetic, image classification, and irregular time-series. Our framework opens up a new way for a continuous representation of neural networks that can be readily adopted for real-world applications.

Under review as a conference paper at ICLR 2023
LEARNING PDE SOLUTION OPERATOR FOR CONTINUOUS MODELING OF TIME-SERIN
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning underlying dynamics from data is important and challenging in many real- world scenarios. Incorporating differential equations (DEs) to design continuous networks has drawn much attention recently, the most prominent of which is Neural ODE. Most prior works make specific assumptions on the type of DEs or restrict them to first or second-order DEs, making the model specialized for certain problems. Furthermore, due to the use of numerical integration, they suffer from computational expensiveness and numerical instability. Building upon recent Fourier neural operator (FNO), this work proposes partial differential equation (PDE) based framework which improves the dynamics modeling capability and circumvents the need for costly numerical integration. FNO is hard to be directly applied to real applications because it is mainly confined to physical PDE problems. To fill this void, we propose a continuous-in-time FNO to deal with irregularly- sampled time series and provide a theoretical result demonstrating its universality. Moreover, we reveal an intrinsic property of PDEs that increases the stability of the model. Several numerical evidence shows that our method represents a broader range of problems, including synthetic, image classification, and irregular time- series. Our framework opens up a new way for continuous representation of neural networks that can be readily adopted for real-world applications.
INTRODUCTION
The modeling of time-series data plays an important role in various applications in our everyday lives including climate forecasting (Schneider) [2001] Mudelsee] (2019). medical sciences (Stoffer
Ombao 2012 Jensen et al., 2014), and finance (Chatigny et al.) 2020 Andersen et al. 2005).
Numerous deep learning architectures (Connor et al. 1994 Hochreiter & Schmidhuber 1997 Cho
jt al., 2014) have been developed to learn sequential patterns from diverse time-series datasets. In
recent years, leveraging differential equations (DEs) to design continuous networks has attracted increasing attention, first sparked by neural ordinary differential equations (ODEs) (Chen et al.] 2018).
Differential equations that characterize the rates of change and interaction of continuously varying quantities have become indispensable mathematical language to describe tioe-vvolving real-world phenomena (Cannon & Dostrovsky 2012; 2 Sundén & Fu 2016 Black & Scholes 2019). By virtue of their ability to represent and predict the world around us, incorporating differential equations into neural networks has reinvigorated research in continuous deep learning, offering new theoretical perspectives on neural networks. Moreover, they provide memory efficiency, invertibility, and the ability to handle irregular time-series (Rubanova et al. 2019 Chen al. 2019 Dong al. 2020).
Despite their eminent success, Neural ODEs have yet to be successfully applied to complex and large-scale tasks due to the limitation of expressiveness of ODEs. To respond to this limitation, there are several works that enhance the expressiveness of Neural ODEs (Gholami et al. 2019 Gu et al., (021). Another line of works attempts to introduce more diverse differential equations. such as controlled differential equations (Kidger et al.) (020), delay differential equations (Zhu cet al. 2020 Anumasa PK 2021), and integro-differental equations (Zappala et al.) (022). In real applications, however, we usually know little about the underlying dynamics of the time evolution system. In general, we are hard to knowledge about how the temporal states evolve, which kind of differential equation it follows, how variables depend on each other, and how high derivatives it contains. Therefore, it is necessary to develop a model that can learn an extended class of differential equations that is able cover more diverse applications, in data-driven manner (Holt et al. 2022).

---

Under review as a conference paper at ICLR 2023
In this work, we propose partial differential equation (PDE) based novel framework that can learn broad range of dynamics without prior knowledge of governing equations. PDEs that enjoy relations between the various partial derivatives of multivariable states represent much general dynamics, including ODEs as a special case. There have been several attempts to design neural networks
through the lens of PDEs (Eliasof al., 202T; Ruthotto & Haber 2020 Yair et al. 202T
Sun et al., 2020) Kim et al. 2020b). Most of the prior works have been designed under specific assumptions about the type or structure of the PDEs. As the underlying dynamics are unknown in real-world data, however, it should be oblivious to the knowledge of the underlying PDE structure and needs be learned from data. Moreover, because the appropriate properties of PDE differ for each given problem (Eliasof et al. 2021]. it is necessary to be able to represent wide range of PDEs as possible. To this end, we adopt Fourier neural operator (FNO) (Li et al., (021). an emerging model that directly parametrizes the PDE solution operator without prior information on the governing PDE, By learning the solution operator, the model automatically learns diverse equations in a completely data-driven way, as well as attains the solution in a single call without using numerical integration that invokes computational expensiveness and numerical instability.
Because FNO has been scrutinized on mathematically well well-posed PDE problems, adapting FNO to practical applications confronts several drawbacks. First, due to its notion of discrete time, FNO is difficult to directly transfer to irregularly-sampler time-series commonly arising in real-world problems. To render it more suitable for continuous time-series, we propose a continuous-in-time FNO. termed CTFNO, that can be evaluated at arbitrary time points. By learning continuous-timi solution operator, CTFNO can flexibly capture diverse time-series data. Moreover, we demonstrate the representational power of CTFNO via a rigorous theoretical proof of the universal approximation theorem. Secondly, we develop a network architecture that guarantees stability and leads to well- well-posed learning problems, By ensuring the stability of the model, can defend against noisy observations and alleviate overfitting. We also verify that it enhances the adversarial robustness for image classification. The results of various experiments show that our model provides superior performance on wide array of real-world data with applications in time-series and image classification.
BACKGROUND
Fourier Neural Operator Let Ω C R" be bounded domain, and A= = (Q) and U u (9:1Rdu be spaces of functions from 12 to gd. and Egd respectively. Fourier neural operator (FNO) (Li et al. 2021 learns a map 9 : A + Ul between infinite-dimensiinnal function spaces, with special focus on PDEs. An input function a E AA is any of source or initial functions, and u E u is the corresponding solution. The solution to fairly general PDEs is represented as convolution + Rdd xd. called by a Green's function as follows:
operator with kernel G : R="
S G (x y) a (1) dy, Va E Ω.
(x) - (
This solution formula elucidates an elegant way to design FNO. The overall computational flow of FNO for approximating the convolution operator is given as a Pa Bi بi 11 L Liz
UL -, u, for given depth L. To recover the sorts of universality, the input function concatenated with position æ, denoted by [a(z); x]] E R(d.-n), is lifted to higher dimensional representation to (I)) de by (a))(a) == P [a(x); a:] with a matrix P E xdax(da+t))
Q is projection operator of the form (zv)(x)= = Qu(x) for Q E Xds, Fourier layers Cs are defined as follows:
Definition 2.1. (Fourier layers Li al. 2021b) For convolution operator Ke L (2; Rd.))
u (R.).. with kernel K₃ : R -> 1 dd., a linear transform W, : IR.. -> Rdd., ; Rg = F (m²)
R" + Cd.xd. Fourier transform F, and an activation function a IR + R which is component-
wisely applied, the E-th Fourier layer La is defined as follows: Ve E l (S); Rd.). , x Ω
Le (w) (x) = (Wev (x) + KCe (u) (x)) = (Wed (x) + +F (Re (x)) RR (E) (Fv) (E) f
where Rs Re.FF
E TR..
Both Fourier transform JF and inverse Fourier transform J- are implemented by the fast Fourier transform (Nussbaumer 1981) with truncated frequencies at maximum modes max [&] S N.
Treatment of time-varying problems When applied to time-dependent PDEs, the original FNO can only learn an operator that maps the initial function to a solution for single fixed time. To
2

---

Under review as a conference paper at ICLR 2023
Spiral
Square
Sawtooth
1.0
1.0
2.0
0.0
1.0
0.5
0.0
0.0
-1.0
-1.0
-1.0
0.0
1,0
0.0
0.5
1,0
0.0
0.5
1,0
ANODE
Neural Flow
Neural Laplace
CTENO (Ours)
True
Figure 1: Visualization of learned solutions to synthetic datasets.
deal with tame-varying problems, two methods suggested in (Li et al. 2021 (i) It poses the time-dependent problem as sequence-to-sequenee task. (ii) Treats it as (n + 1)- problem by adding one more dimension and apply FNO layers convolve in space-time domain. In this case, FNO can only predict the solution at times on fixed equispaced temporal mesh.
METHOD
3.1 MOTIVATION
Why do we consider PDEs for time-series modeling? Starting with Neural ODEs, leveraging DEs has been found to be effective in modeling time-series data (Kidger et al. 2020: Tia & Benson 2019). They have shown promising results, however, their model architectures and inference schemes specialized to the DEs on which they are based. These bespoke model structures rule out their generalization ability to other classes of DEs, which is further exacerbated in real-world applications. Therefore, need model that can learn a broad range of dynamics cope with wide variety of applications. In this paper, we propose a novel framework for modeling time-series data through the lens of PDEs, richer family of differential equations. Unlike ODEs which deal with functions of single variable, time-dependen PDEs describe the evolution of a physical quantity u (t, r), not only with time t, but also according to spatial variable (I1, ... .n) Q having the form:
du (t, Lu((,x)) in [0, (00000)x2 xΩ, (0, (1:) on
0t
where 0/0t denotes a partial derivative w.r.t time t, L some differential operator consists of
spatial derivatives, and [0, oo) x Ω - Rd. satisfying is called by the solution with initial condition (x:) E A at = 0. For example, 2 with L p/rr + ..... /// /0x1. called by heat equation, describes the diffusion process of temperature
surface. We refer the interested reader to [Evans] 2010) for more details PDEs. Due to their heavy expressivity, PDEs are widely used to describe complex continuous processes including the laws of physics (Temam 2001), principles of (.00)...) Expe Experimental results in Figure[[onndutee conducted on three synthetic examples proposed by (Bilos al. 2021) confirm the ability of our FDE-based model to represent more general temporal dynamics. It can be seen that our model, which based on PDEs, which represent more diverse classes of dynamics than DEs on which baselines are built, consistently achieves superior performance in all examples. Please consult Section 4.1 for additional results and experimental details. In this paper, we harness FNO for modeling a variety of time-series applications through PDEs.
engineering (Kulov & Gordeev 2014), and medical science [Joshi (002).
Figure
3.2 CONTINUOUS- -TIME FNO
In wide variety of real-world problems, the system of interest evolves continuously over time. FNO has recently shown the promise ability to learn complex PDEs, however, the notion of discrete time of FNO hinders wider applicability to varying systems. To ameliorate this limitation, we introduce continuous-in-fin
Fourier neural operator (CTFNO). Consider time-dependent PDE 2 The Green's 's function formula says that there exists a Green's function G [0, 20) x R" Rd.xa such that the solution for 2 is represented by follows:
u (t. / G (t, v) (y) dy, & Vt, E o0) x Ω.
This shows that the weights an ENO layer should be conditioned on time, to learn an operator (1, (z:) > (1) for arbitrary time and initial condition a. For example, the Fourier transform
3

---

Under review as a conference paper at ICLR 2023
sharing not ((())
R1
=((.x)
Mg(x)
V1 (t.,x)
W,
(c.x.)
W;
Py((.x)
shuring 2))
Figure 2: The visualization of CTFNO architecture. Two time embedding networks specify the time I into hidden representations (t)) and (t). Then sp(() and v)(() incorporate the temporal information into each Fourier layer through time modulation operators A and Bg- See Appendix C.2 for details.
of the Green's s function of the heat equation is given by R (1., time-aware Fourier layer that updates the solution based on 33) as follows:
e- To this end, we propose a
Definition 3.1. (Continuous-time Fourier layers) Under the same notations with Definition 2.1
and for E [0, oo), convolution operator K2 (t) U (9;; R + u (2; R2.) with kernel Kt (t)
R TR xd. linear transform We (t) Rdd - Rdd, Re (t) F (NE (1)) IR" - Cd.Xds,
oe (t) R.n - C, and the (t) E TR.x Xdv the E-th continuous-time Fourier layer is defined as follows:
Le (b) = a (We (x)) (1) (v)
=a (Wc (Re (t) (FF))
(4)
(We))()) (pe (f) Ra (Fo)) (z)) Vr E E
Time Modulating To equip the FNO network with the ability to capture information on the time of observations, time-dependent layer (4) constructed as follows. Two sharing networks. parameterized by two-layer fully connected networks together with sinusoidal embedding (Vaswani al. 2017), first convert the input time t into multi-dimeesional representations yp (t). v (8) E T for a hidden dimension For notational simplicity, we denote yps (t) = yp (f) (5) and similar for R(t, c). For each Fourier layer Ls and frequency E, learnable A R" + C and Bg E Rd produce time information (t,
(t)" A (E) E C and tे (t) = diag (B.b (t)) E Dd (the diagonal matrix with the elements of vector Brú (t) on the main diagonal). For each Fourier mode Ç. Rs (E) is parameterized by a (dw, x ccomplx matrix, and the dimendependent Rs (t) is given by multiplying complex number yoe (t, i.e., Re (t. = si (t,g) Re (E). In addition, We (t) is parameterized by multiplication of two real (d, x d, -maariitssess Wt and (t). This is somewhat reminiscent of weight modulation in StyleGAN2 (Karras et al. 2020). but applied per-frequency in the Fourier domain and per-channel in the spatial domain.
Ablation study for time-aware network We provide insights into our model by investigating how useful the structure of the CTFNO designed based the theoretical time-dependent Green's formula B) is for learning time-dependent PDEs. We study three alternative ways to equip the temporal infor- mation into the FNO network, rather than weight modulation:
Table I: Relative L (x10-2)
errors of ablation studies.
Model
Heat
Burgers
Baseline 1:
concatenating time & with input function uo (I)). concatenating encoded time (t)(t) with a lifted input function (z()))
Baseline
11.60
19.63
Baseline 2 2.84
18.68
Baseline 2:
Baseline 3 2.68
18.92
FNOseq
75.61
73.10
Baseline 3:
concatenating encoded times (o(()) s (t),...,,PI() with intermediate features uo(x)), vn (x)...., (.)))...))) spectively.
FNO2d
0.58
10.32
(z),,,
CTENO 0.22
5.68
See Appendix D for more details. we test the ability of these models to learn canonical linear and nonlinear PDEs: -inedimenioonl heat and Burgers' equations. Table show that, in all cases, the three ablation study models, which are structures that can continuously handle arbitrary time f, are similar to worse than FNO2d (proposed in the original FNO paper) that can only evaluate the solution value at times on a specific grid. However, CTFNO significantly outperforms other models. The rationale of the way of imposing temporal information of time-dependent PDEs. The results validate that the time modulating structure of CTFNO designed based on the time-dependent Green's formula is much more expressive in actually learning the PDE.
CTFNO is based on Green's 's formula
4

---

Under review as a conference paper at ICLR 2023
3.3 UNIVERSAL APPROXIMATION
The universality of FNO has been proven by Kovachki et al. (2021). Based on this, we prove the universality of the proposed CTFNO, which condensed the following informal statement. A formal statement and details of proof of Theorem 3.1 are provided in Appendix A
Theorem 3.1. (Informal) CTFNO can approximate any time-dependent continuous operator, of arbitrary accuracy.
3.4 STABILITY
When we learn model for the time evolution of dynamical systems, the learned dynamics should be guaranteed to be well-posed. This is important because the learned system can be unstable when using generic neural network (Szegedy al., 2013 Moosavi-Dezfooli et al. 2017). In this case, the resultant output of the learned dynamics may not be continuous with respect to its input. Also, such unstable networks are vulnerable to adversarial attacks, overfitting, and unauthorized exploitation, which may render the network useless in practice. Therefore, stability is necessary condition in real-world applications. The stability of model is measured by the sensitivity of the prediction with respect to small perturbations of the inputs. A formal definition is given as follows.
Definition 3.2. (Stability (Hadamard, 1902)) PDE 2p is said to be stable if for any solution (f, with initial condition uo (z) and e 0, there exists 8 >> O such that for all new initial function üg (z) satisfying [uo uoll < 8, the corresponding solution ū xr) satisfies || - uI| 6 for all O.
The stability is a hard constraint imposed upon the model. While some recent studies have begun to address the stability of neural network architectures, it has typically been used as a soft constraint by adding an extra regularization loss term (Moosavi-Deztooli al. 2019), or required computation of the eigenvalues of the Jacobian matrix (Ross & Doshi Velez, 2018 Hoffman et al.) 2019). Besides. stability of Neural ODEs are elicited by stable discretization techniques of the ODEs (Haber & Ruthotto 2017 Yan et al. 2020). Similarly, the stability of CTFNO is connected to the well- posedness the learned PDE, whose solution operator the operator learned by CTFNO. The kernel formulation 3 sheds light way ensure stability. A proof is deferred to appendix
Proposition 3.2. (Stability of CTFNO) IF || Elll, and ||W (t)ll, are bounded for every t > 0, then the corresponding CTFNO with Lipschitz continuous activation function is stable.
Gershgorin dises normalization As given in proposition 3.2 the global stability of CTFNO guaranteed if the Fourier kernel and weight have bounded norm. Because of its expensive computational cost, however, suggest a practical method for enforcing stability conditions. Gershgorin's circle theorem (Vargal 2010) allows us to make fast deductions on the bound of eigenvalues. It states that every eigenvalue of square matrix A lies within the union of discs centered at diagonal elements with a radius of the sum of the magnitudes of the off-diagonal entries from the same Therefore, normalize the 7,- norm of each row, can impose the requisite stability.
4 EXPERIMENTS
In this section, we empirically validate the applicability of the proposed model to study variety of synthetic, and real applications. In all experiments, we build models with comparable network sizes to make a fair comparison. Precise implementation details can be found in Appendix C
4.1 SYNTHETIC DATA
In this section, we use several illustrative examples to demonstrate the outstanding capacity of the proposed method in learning diverse class of dynamics. In all experiments, every model was trained with mean squared error (MSE) loss. We evaluate the performance of CTENO in comparison with several related lines of work on a broad range of differential equations:
Datasets We compare the performance of CTFNO on broad range of synthetic data. Mathematical formulation of these dynamics can be found in Appendix C.1
5

---

Under review as a conference paper at ICLR 2023
Table 2: RMSE (x10 results on synthetic data.
Model
Stiff
Spiral
Sawtooth
Square
Reaction
Heat
Burgers
NODE
44.23
3.26
28.09
97.50
5.109
0.351
5.348
ANODE
37.18
3.26
9.92
80.95
4.241
0.351
4.214
Neural Flow
26.39
3.26
7.93
20.40
10.300
3.038
12,602
Neural Laplace
20.83
4.25
4.84
17.06
2.804
2.205
5.291
DON
3.055
0,473
6.022
PDN
3.316
0.3234
5.796
FNO2d
0.472
0.0330
3.136
CTFNO
16.53
1.87
3.90
11.69
0.239
0.0269
1.952
Stiff ODE (Holt et al. 2022 is a typical example that Neural ODEs fail to learn.
Spiral ODE is two-dimensional system of nonlinear ODEs, commonly arising in biological systems. The dynamics describe spiral shaped trajectories.
Sawtooth and Square (Bilos et al. 2021 considered to evaluate the capability
modeling waveform signals. These are piecewise differentiable having cusps.
Reaction ODE commonly used to model chemical reactions and of the form 0u/0l 6u (1 u). Unlike to aforementioned ODEs, solution to Reaction ODE is regarded function and in implementation it can be regarded as a high-dimensioon
ODE.
Heat (Baron Fourier 1878) and Burgers' equations (Bateman) 1915) are canonical one- dimensional linear and nonlinear PDEs. They take the form
du du dr
8uu Oz
Ōt + au
(), e (0,7),
(5)
with the corner cases: heat 0 and Burgers' equation 1. Given initial function Un, we train models to learn (t, uo (I)) + u (t,z) for f E (0, T)
Baselines We evaluate the performance of our model in comparison with several related lines of works: the standard Neural ODE (NODE) (Chen et al. 2018). ANODE (Dupont et al. 2019). and Neural Flow (Biloš al. 2021), which directly parametrizes the solution operator of an ODE, are adopted for ODE-b models. We also consider with Neural Laplace (Holt et al.) 2022) which can represent diverse classes of equations by modeling them the Laplace domain. Furthermore, we compare with representative PDE-based models; DeepONet (DON) (Lu et al. 2019a), alternative operator learning method, POD-DeepONet (PDN) (Lu al. 2022). a functional PCA-based model, Fourier neural operator with spatio-temporal inputs.
and FNO2d (Li et al. 2021
Results Discussion The overall results of root mean squared error (RMSE) are reported in Table 2 The results show that our CTFNO substantially achieves conspicuous success all baseline models by a large margin on all datasets. We can see that the performance difference of baselines on different datasets varies Jot. An ODE model that performs well on certain data fails to learn the dynamics of another data. But, Neural Laplace, which can express wider dynamics than ODE-based models, performs better than the three ODE-based models (NODE, ANODE, and Neural Flow) throughout all datasets. Further, our model, which can describe broad family of DEs, consistently approximate all synthetic dynamics much better. These results demonstrate how important the range of expressible dynamics of the model is. Figure Tigure[provide. pictorial results of the learned solutions to Spiral, Square, and Sawtooth data, and further quatlitative comparisons are included in Appendix [D.] Table also contain the comparison to the original FNO2d. Comparison results of CTFNO with the original FNO demonstrate that the use of the proposed time-dependent structure significantly improves the capacity of the model to handle time. Moreover, our model is superior to existing benchmark PDE models. The overall results confirm that CTENO is amenable to learn a wide array of dynamics.
4.2 REAL TIME-SERIES APPLICATIONS
In this section, we investigate the performance of our method on interpolation and prediction tasks using three real-world time-series datasets, including partially observed, multi-variate sequences. All models were run three times with different random seeds and we report the averaged value. Results show that the CTFNO performs outperforms range of baselines.

---

Under review as a conference paper at ICLR 2023
Table 3: MSE (x10-3) on the MuJoCo dataset.
Model
Interpolation (% Observed Points)
Prediction (% Observed Points)
10%
20%
30%
50%
10%
20%
30%
50%
RNN-VAL
65.14
64.08
63.05
61.00
23.78
21.35
20.21
17.82
ODE-RNY FEE-RNN
16.47
12.09
9.86
6.65
135.08
319.5
154.65
264.63
LODE (RNN enc)
24.77
5.78
27.68
4.47
16.63
16.53
14.85
13.77
LODE (ODE enc)
3.60
2.95
3.00
2.85
14,41
14.00
11.75
12.58
Neural Flow
7.15
5.58
4.96
4.60
17.99
16.10
15,48
15,29
CTENO
1.53
1.18
1.12
1.15
9.26
8.93
8.42
8.70
Datasets
We evaluate our model on MuloCo hopper, PhyisioNet 2012, and Human Activity datasets,
as in Rubanova et al. (2019). For more details, see Appendix C.I
MuJoCo Tassa et al. 2018) We consider the hopper physics environments from the Deepmind Control Suite. Data records 14-dimensional attributes, including state and action, with 100 timestamps. To deal with partial observations, we conduct interpolation and prediction tasks, in which we reveal either 10%, 20%, 30%, or 50% of the ground truth.
PhysioNet 2012 (Silva al. 2012) To investigate sparsely observed time series, evaluate our model on a real-world clinical dataset. PhysioNet Challenge 2012 dataset is irregularly sampled and sparsely observed. The goal interpolate and predict 41 biomedical features, such as heart rate and glucose, of intensive care unit (ICU) patients.
Human Activity Human Activity dataset consists of sporadically observed sensor data collected from five individuals performing several activities (i.e. walking, standing, laying, etc). We use the Activity dataset together with all pre- pre-processing steps as they were provided by (Rubanova al. 2019), resulting in 6554 sequences of 211 time points. We train our model to classify the type of human activities from sequential data.
Baselines Five models are chosen for the comparisons: (1) RNN-VAE, variational autoencoder (VAE) (Kingma & Welling. 2014; Rezende et al., (2014) model whose encoder and decoder are recurrent neural networks (RNNs). (2) ODE-RNN (Rubanova al.) 2019), RNN model which uses Neural ODEs to model hidden state dynamics. (3) Two Latent ODE (LODE) models with RNN (Chen 2018) and ODE-RNN (Rubanova al., 2019) encoders. (4) Coupling Flow in Neural 2021> a neural operator which directly models the solution curves of an ODE.
Flow (Bilos et al.
Results Discussion We follow the encoder-decoder framework of Latent ODEs that leverages VAE architecture to represent incomplete time-series data as continuous-time model. For CTFNO, we employ a simple RNN encoder, and focus on the model representations of inherent dynamics in the latent space. Results MuJoCo reported in Table show that CTFNO consistently outperforms the baseline models to a large extent on all tasks both interpolation and prediction across all kinds of observed time points. This validates that our model can be used relevant model for real applications on time-series with missing time steps. Table 4 summarizes interpolative and predictive performance on PhysioNet, and per-time-poin classification accuracy on Activity. CTFNO superior to all of the benchmark models on both PhysioNet and Activity, which indicate that our model provides useful utilization of sparsely observed time-series data and meaningful representation for classification. The overall results demonstrate that our proposed model, which is based on PDEs, consistently improves the performance of ODE-based models in real applications and is novel approach that can handle wide range of real-world problems.
Table 4: MSE (x10-3) (210-3)a00 on PhysioNet and per-time- point classification accuracies (%) on Activity.
2.5
Model
PhysioNet
Activity
Interpolation Prediction Accuracy
MSF 110
Neural ODEs CTFNO(Train)
RNN- VAE
5.93
3.05
34.3
CTENO(
ODE-RNN
2.36
82.9
1.0
LODE (RNN
3.16
5.78
83.5
LODE (ODB enc)
2.23
2.95
84.6
Neural Flow
2.94
65.7
1.2
CTENO
1.80
2.10
85.0
M
Figure 3: Effect of stabilization on the Plane Vibration dataset.
7

---

Under review as a conference paper at ICLR 2023
4.3 STUDIES ON STABILITY
We This section is devoted to validating the effect of the proposed stabilization scheme in Section 3.4 include experimental results on real-time series data and adversarial attacks on image classification.
Datasets
We the stability on a real-world time-series data and two image datasets.
Plane Vibration (Nol & Schoukens 2017) is considered to elucidate how stabilization scheme works practical time-series data. a multi-variate data, consisting of five fea- tures; force, voltage, and accelerations measured in three spots. We comply the experimental setup as provided HBNODEs (Xia al. 2021). The task of plane vibration is interested in forecasting the next eight time steps from the previous 64 consecutive time observations.
image classification, which is of the bench- mark applications of Neural ODEs, on two benchmark datasets: MNIST (LeCun et al. 1998) and CIFARIO (Krizhessyy et al. 2009). Although this is not time-dependent problem, include classification to validate the effect of the proposed stabilization scheme. Images could be considered functions a [0, 1]² - R of light defined continuous region of pixel locations, instead of 32 32 pixel vectors of RGB values. Classification requires us to learn dynamics that simufraneously drives each input (x) to the corresponding final feature (x), allocated depending on its label. Therefore, posing the problem as learning an operator that maps the image a (x) to (x), we can apply FNO to image classification.
Image datasets We perform experiments
Baselines We compare the performance plane vibration with the results reported in (Xia et al. 2021 which contains second-order Neural ODE models including ANODE, SONODE (Norcliffe al. 2020), HBNODE, and GHBNODE (Xia al. 202T). For image classification, take CNN and ANODE as baselines. All models are fit to share the similar number of parameters.
Improvement in Model Generalization MSE (x10-3) of Neural ODEs reported in (Xia et al. 202T) is 2.5, and is annotated by green dashed line in Figure 3 Our CTFNO achieves much lower MSE of 1.96. Figure B presents positive effect of stabilization; a tendency of CTFNO to be less prone to overfitting. The ability to generalize well outside the training dataset is essential for models to be practically useful. We observe that increasing the Gershgorin's discs stabilization parameter M leads to overfitting, and too small M produces degenerated system, dropping the performance. Besides, the model with congenial stabilization does not suffer from overfitting or degradation of the accuracy, which validates the effect of stabilization. Morcover, it is notable that Neural ODEs require tremendous computational burdens due to their use of numerical ODE solvers. By obviating the need for the ODE solver, the computational time of CTFNO is merely about 1 10% of Neural ODEs. Similar investigation in image classification is included in Appendix D.2
Table 5: Classification accuracy (%) and robustness of different models on MNIST.
Clean
Gaussian noise
Adversarial attack
Model
50
100
FGSM-30/255
FGSM-50/255
PGD-30/255
PGD-50/22
CNN
99.0
29.7
10.4
27.3
15.3
10.8
4.6
ANODE
97.9
91.8
56.4
26,9
1.5
62.5
9.7
FNO
98.6
90.2
45.6
36.9
25.8
8.1
4.1
FNO (WD)
98.9
90.0
43.6
57.6
35.6
22.1
11.7
FNO (Stab)
98.8
97.8
88.2
36.2
3,43
62.6
47.1
Table 6: Classification accuracy (%) and robustness of different models CIFARIO.
Clean
Gaussian noise
Adversarial attack
Model
15
20
FGSM-2/255
FGSM-5/255
PGD-2/255
PGD-5/255
CNN
75.5
60.0
$1.1
23.0
4.7
16.7
0.43
ANODE
59.4
40.9
34.9
0.1
0.0
0.0
0.0
FNO
64.0
57.9
52.2
3,4
1.4
1.5
0.0
FNO (WD)
66.9
54.9
46.2
2.0
FNO (Stab)
65.5
64.3
63.3
38.6
11.1
38.1
9.1
8

---

Under review as a conference paper at ICLR 2023
Robustness on Adversarial Attack By ensuring the stability of the learned PDE in Section 3.4 we may expect that the resulting solution for an input with a small perturbation converges to the same label as the unperturbed one. To investigate this stabilization effect on FNO, we test the trained model in defending against Gaussian noise, fast gradient signed method (FGSM) (Goodfellow et al. 2014), and projected gradient descent (PGD) (Madry et al. 2017) attack. Weight decay (WD) is a standard practical technique to regularize the L norm of weights of models. Because WD has similar intention our stabilization scheme, these two are also compared. For fair comparison, CNN and ANODE are trained with WD. and show that for both datasets, our stabilization significantly enhances the robustness of the model. Our model achieves better results in both accuracy and robustness than ANODE. Although CNN produces around 10% higher accuracy in CIFAR 10, our model outperforms CNN for perturbed images. Moreover, the results indicate that FNO trained with WD is prone to mislead by adversarial attacks. This demonstrates that it is much more accurate to apply a model-suitable stabilization based on the PDE theory, rather than naively regularizing the L norm of the weights.
5 RELATED WORK
ODE based Networks The interpretation of residual network (He et al. (016)) as a discretization of an ODE (Chen et al. 2018 Haberr& & Ruthotto 2017; Lu et al. 2018) provided an interface between deep learning and ODEs. Subsequently, extensive work has been conducted on parametrizing the continuous dynamics of hidden states using an ODE specified by neural networks (Greydanus et al.,
2019; Lu et al. 2019b Liu al. 2021). These approaches have found wide applicability across diverse applications including vision tasks (Scao 2020, Park & Kim 2021 Pinckaers & Litjens 2019 Xu et al.
2021), mean-field games (Ruthotto et al. al.] 2020)) speech synthesis (Kim al. 2020aa.
and density estimation (Chen et al., 2019 Yildiz et al. 2019). Owing to the continuous representation of neural networks, Neural ODEs are particularly attractive for cergulaaly-samplee time series data
(Rubanova et al., 2019; De Brouwer et al. 2019) Chang et al. 2019: Kidger et al.) 2020)] Chen et al.
2020). A recent work(Bilos al. 2021) circumvents the usage of an expensive numerical integration by directly parametrize the solution trajectory of an ODE. Furthermore, stable (Kolter & Manek 2019; Zhang et al. 2019 Kang et al. 2021] Tuor et al. 2020) and reversible (Chang et al., 2018
Grathwohl et al. 2018) architectures has been developed based on the intrinsic properties of ODEs.
PDE based Networks Some studies have integrated PDEs and neural networks (Eliasof et al.
2021 Ruthotto & Haber 2020; Ben- Yair et al. 2021 Sun al.) 2020). These works have establ established
based on a link between convolution filters CNNs and differential operators (Long et al. 2018).
or finite difference discretization (Strikwerda) 2004 of derivatives. Recently, new approach (Kim et al. 20205) that uses a physics-informed neural network (PINN) (Raissi et al. 2019) has been proposed. All of these works assign author-defined specific PDEs to neural networks. Because the nature of each problem to which the model be applied differ considerably, the choice of dynamics has been shown be quite important. Therefore, such architectures designed with fixed PDEs cannot be widely applied. Despite their PDE-based motivation, those approaches were only applied to image classification tasks and did not explore the application on continuous inputs and time-series data.
CONCLUSION & LIMITATIONS
In this paper, we presented novel approach for modeling time-series in terms of PDEs. As time is intrinsically continuous, we proposed a neural operator CTFNO that learns the underlying PDE by equipping FNO with the ability to represent time in a continuous manner. We also provided theoretical guarantees for the universal approximation and the stability of CTFNO. Our comprehensive experiments demonstrated that CTFNO outperforms existing differential equation based models on synthetic and real-world datasets, and the proposed stabilization method effectively improves robustness and prevents overfitting.
We note that FNO is hard to embody discontinuous features because Fourier transform only captures global information. We expect that extending our approach to model that can extract local features, such as (Gupta et al.) 2021). provides interesting avenues for future work. Also, the stabilization parameter M is a detrned hyperparameter. The optimal M for a given data is unknown and this one of our limitation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Torben G Andersen, Tim Bollerslev, Peter Christoffersen, and Francis X Diebold. Volatility forecast- ing, 2005.
Srinivas Anumasa and Srijith PK. Delay differential neural networks. In 2021 6th International Conference on Machine Learning Technologies, pp. 117-121, 2021.
Jean Baptiste Joseph Baron Fourier. The analytical theory of heat. The University Press, 1878.
Harry Bateman. Some recent researches on the motion of fluids. Monthly Weather Review, 43(4):
163-170, 1915.
Ido Yair, Gil Ben Shalom, Moshe Eliasof, and Fran Treister. Quantized convolutional neural networks through the lens of partial differential equations. arXiv preprint arXiv:2109.00083 2021.
Marin Biloš, Johanna Sommer, Syama Sundar Rangapuram, Tim Januschowski, and Stephan Gün- nemann. Neural flows: Efficient alternative to neural odes. Advances in Neural Information Processing Systems, 34, 2021.
Fischer Black and Myron Scholes. The pricing options and corporate liabilities. World Scientific Reference Contingent Claims Analysis in Corporate Finance: Volume I: Foundations of CCA and Equity Valuation, pp. 3-21. World Scientific, 2019.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint wwXiv:1509.005/00 2015.
John T Cannon and Sigalia Dostrovsky. The Evolution of Dynamics: Vibration Theory from 1687 to 1742: Vibration Theory from 1687 to 1742, volume 6. Springer Science & Business Media, 2012.
Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Reversible architectures for arbitrarily deep residual neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi. Antisymmetricrnn: A dynamical system view on recurrent neural networks. The International Conference on Learning Representations, 2019.
Philippe Chatigny. Jean- Mare Patenaude, and Shengrui Wang. Financial time series representation learning. arXiv preprint arXiv:2003.12194 2020.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn Mennikk Jacobsen. Residual flows for invertible generative modeling. Advances Neural Information Processing Systems, 32, 2019.
Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural networks. In International Conference on Learning Represen- tations, 2020.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409) 1259, 2014.
Jerome Connor, R Douglas Martin, and Les E Atlas. Recurrent neural networks and robust time series prediction. IEEE transactions on neural networks, 5(2):240-254, 1994.
George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-obseree time series. Advances neural information processing systems, 32, 2019.
Chengyu Dong, Liyuan Liu, Zichao Li, and Jingbo Shang. Towards adaptive residual network training: neural-ode perspective. In International conference on machine learning, pp. 2616-2622 PMLR, 2020.
10

---

Under review as a conference paper at ICLR 2023
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in Neural Information Processing Systems, 32, 2019.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Newworks, 107:3-11, 2018
Moshe Eliasof, Eldad Haber, and Eran Treister. Pde-gen: Novel architectures for graph neural networks motivated by partial differential equations. Advances in Neural Information Processing Systems, 34, 2021.
Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
Amir Gholami, Kurt Keutzer, and George Biros. Anode: Unconditionally accurate memory-efficient gradients for neural odes. arXiv preprint arXiv:1902.10298 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: wxXv::442.66572, 2014.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367, 2018.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, arXiv preprint arXiv:2111.00330 2021.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catan- zaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13581 2021.
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet- operator learning for differential equations. Advances in Neural Information Processing Systems, 34, 2021.
Eldad Haher and Lars Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34 (1):014004, 2017.
Jacques Hadamard. Sur les problèmes aux dérivées partielles et leur signification physique. Princeton university bulletin, pp. 49-52, 1902.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 770-778, 2016.
Dan Hendryeks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
Judy Hoffman, Daniel Roberts, and Sho Yaida. Robust learning with jacobian regularization. arXiv preprint arXiv: 2019.
Samuel Holt, Zhaozhi Qian, and Mihacla der Schaar. Neural laplace: Learning diverse classes of differential equations the laplace domain. pp. 8811-8832, 2022.
Anders Boeck Jensen, Pope L Moseley, Tudor Oprea, Sabrina Gade Ellesøe, Robert Eriksson, Henriette Schmock, Peter Bjødstrup Jensen, Lars Juhl Jensen, and Søren Brunak. Temporal disease trajectories condensed from population-wide registry data covering 6.2 million patients. Nature communications, 1-1) 2014.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.

---

Under review as a conference paper at ICLR 2023
Hem Raj Joshi. Optimal control of an hiv immunology model. Optimal control applications and methods, 23(44)199-211) 213, 2002.
Qiyu Kang, Yang Song, Qinxu Ding, and Wee Peng Tay. Stable neural ode with lyapunov-stable equilibrium points for defending against adversarial attacks. Advances in Neural Information Processing Systems, 34, 2021.
Tero Karras, Samuli Laine, Miika A Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality stylegan. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 81 8110-8119 2020.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
Hyeongju Kim, Hyeonseung Lee, Woo Hyun Kang, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Wavenode: A continuous normalizing flow for speech synthesis. arXiv preprint arXiv:2006.04550 04598, 2020a.
Jungeun Kim, Seunghyun Hwang, Jihyun Hwang, Kookjin Lee, Dongeun Lee, and Noseong Park. Pde-regularized neural networks for image classification. 2020b.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. International conference machine learning, 2014.
J Zico Kolter and Gaurav Manek. Learning stable deep dynamics models. Advances neural information processing systems, 32, 2019.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art-No, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
NN Kulov and LS Gordeev. Mathematical modeling in chemical engineering and biotechnology. Theoretical Foundations of Chemical Engineering, 48(3):225-229, 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2 222244,,, 1998.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. The International Conference on Learning Representations, 2021.
Guan-Horng Liu, Tianrong Chen, and Evangelos Theodorou. Second-order neural ode optimizer. Advances in Neural Information Processing Systems, 34, 2021.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International Conference Machine Learning, pp. 3208-3216. PMIR, 2018.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv: /9Xiv:99/0.... 10t.::990003/90900 193, 2019a.
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393: 14778, 2022.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pp. 3276-3285. PMLR, 2018.
12

---

Under review as a conference paper at ICLR 2023
Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong. Tao Qin, Liwei Wang, and Tie- Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint wwXh:11906.02762, 2019b.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706.06083 2017.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1765-1773, 2017.
Seyed-Mohsen Moosavi- -ezzool... Alhussein Fawzi. Jonathan Uesato, and Pascal Frossard. Robust- ness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Manfred Mudelsee. Trend analysis of climate time series: review of methods. Earth-science reviews, 190:310-322, 2019.
Jean-Phillipe Noël and Maarten Schoukens. F-16 aircraft benchmark based on ground vibration test data. In 2017 Workshop on Nonlinear System Identification Benchmarks, pp. 19-23, 2017.
Alexander Noreliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Liò. On second order behaviour in augmented neural odes. Advances in Neural Information Processing Systems, 33:5911-5921, 2020.
Henri J Nussbaumer. The fast fourier transform. In Fast Fourier Transform and Convolution Algorithms, pp. 80-111. Springer, 1981.
Seobin Park and Tae Hyun Kim. Progressive image super-resolution via neural differential equation. arXiv e-prints, pp. 2101, 2021.
Hans Pinckaers and Geert Litjens. Neural ordinary differential equations for semantic segmentation of individual colon glands. arXiv preprint arXiv: 190.11000, 2019.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics- innfore- neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1286-1286. PMLR, 2014.
Andrew Ross and Finale Doshi- Velez. Improving the adversarial robustness and interpretabilitty of deep neural networks by regularizing their input gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for ireeularly-samplee time series. Advances in neural information processing systems, 32, 2019.
Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, 62(3):352-364 2020.
Lars Ruthotto, Stanley J Osher, Wuchen Li, Levon Nurbekyan, and Samy Wu Fung. A machine learning framework for solving high- high-dimensional mean field game and mean field control problems. 2020.
Proceedings of the National Academy of Sciences. 117(17):9183995
Teven Le Scao. Neural differential equations for single image super-resolution. arXiv preprint 00866 2020.
Tapio Schneider. Analysis of incomplete climate data: Estimation mean values and covariance matrices and imputation of missing values. Journal of climate, 14(5):853-871, 2001.
13

---

Under review as a conference paper at ICLR 2023
Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. Predicting in in-hospital mortality of icu patients: The physioner/computnng in cardiology challenge 2012. In 2012 Computing in Cardiology, pp. 245- 248. IEEE, 2012.
David S Stoffer and Hernando Ombao. Special issue on time series analysis in the biological sciences, 2012.
John C Strikwerda. Finite difference schemes and partial differential equations. SIAM, 2004.
Yifan Sun, Linan Zhang, and Hayden Schaeffer. Neupde: Neural network based ordinary and partial differential equations for modeling time-dependent data. Mathematical and Scientific Machine Learning, pp. 352-372. PMLR, 2020.
Bengt Sundén and Juan Fu. Heat Transfer in Aerospace Applications. Academic Press, 2016.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.000 00690, 2018.
Roger Temam. Navier-Stokes equations: theory and numerical analysis, volume 343. American Mathematical Soc., 2001.
Aaron Tuor, Jan Drgona, and Draguna Vrabie. Constrained neural ordinary differential equations with stability guarantees. arXiv preprint arXiv:2004. 10883, 2020.
Balth Van der Pol and Jan Van Der Mark. Frequency demultiplication. Nature, 120(3019):363-364, 1927.
Richard S Varga. Gersgorin and his circles, volume 36. Springer Science & Business Media, 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Hedi Xia, Vai Suliafu, Hangjie Ji, Tan Nguyen, Andrea Bertozzi, Stanley Osher, and Bao Wang. Heavy ball neural ordinary differential equations. Advances in Neural Information Processing Systems, 34, 2021.
Junshen Xu, Eric 7. Chen, Xiao Chen, Terrence Chen, and Shanhui Sun. Multi-scale neural odes for 3d medical image registration. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 213-223. Springer, 2021.
Hanshu Yan, Jiawei Du, Vincent YF Tan, and Jiashi Feng. On robustness of neural ordinary differential equations. The International Conference on Learning Representations, 2020.
Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. Ode2vae: Deep generative second order odes with bayesian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Emanuele Zappala, Antonio Henrique de Oliveira Fonseca, Andrew Henry Moberly, Michael James Higley, Chadi Abdallah, Jessica Cardin, and David van Dijk. Neural integro-differential equations. arXiv preprint arXiv:2206. 14282, 2022.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. Advances in Neural Information Processing Systems, 32, 2019.
Qunxi Zhu, Yao Guo, and Wei Lin. Neural delay differential equations. The International Conference on Learning Representations, 2020.
14

---

Under review as a conference paper at ICLR 2023
A
UNIVERSAL APPROXIMATION
A.1 MATHEMATICAL NOTATION
We introduce the symbols and mathematical notations that are frequently used in this paper.
Symbol
Description
activation function
d
spatial dimension of domain
T da
periodic torus, identified with [0, 2T]
dimension of the input a (x)
du
dimension of the output solution (a:)
du
dimension of the augmented representation (r)
T
terminal time
L
the number of layers spatial domain point in the the spatial domain frequency variable in the Fourier domain input function space
Ω c. IRd
E
A(D; Rdd)
U(D; Rdd)
output function space
Fx, F
Fourier transform and inverse Fourier transform with respect X
FN. F
discrete Fourier transform and inverse
EN
Fourier wavenumbers KCx = {e 2d 15 S N}
P
lifting operator
Le
neural operator layer
projection operator space of square-integrable functions
T
I2
I C L trigonometric polynomials of degree S N
H
Sobolev space of smoothness s, with norm || -1
PN
2 -rtohoogan Fourier projection PN L L
C(O0,T] x X)
space of continuous functions u [0, + [ ostsT
satisfying
max ||u (I)]l oc:
Dn
space of x n diagonal matrices n identity matrix
In
O
component-wise matrix multiplication
A.2
PROOF OF UNIVERSAL APPROXIMATION
neural operator
N A(;; + C ((O,T) x x (2; Rd))
(6)
N(a)
defined as the following form
N(a) Q L1 o P(a))
(7)
with lifting operator
P: A R2) + C (10,T) x U (9;
((a)((,x)) Pa(x),
P PE Xd
for all a E A R9) ), E Ω, and projection operator
Q: C (10, 7] x 1 (2; Rd) - C' (1, x (9))
Q(v)(t,2) = 2))
Q E ddu
Moreover, introduced in Section 3.2 the time-dependent neural operator layer is defined by
15

---

Under review as a conference paper at ICLR 2023
(t())(t,a)
= a b (Re (t, Fx (w) (5))
(((() (K( (t,x))
(8)
= a (We(t)
(Wedu, (b.b) Fx () (x, (t,
Before establish proof of the universality of CTENO, let us start by recalling the following
Lemma provided in (Kovachki et al. 2021).
Lemma A.1 (Kovvahhi al. 20211). For FNO introduced in Section 2 the following holds.
(a) Letg: : H² RT. + H R be continuous operator with 8, s' 0 and C H be compact subset. Then for any there exists N N, such that for all a K
(I{(a)
Gx(a)ll2 <C
where 9N : H (TM) - 1 (T*) is defined by 9N (a) = PNG (Pxa).
(b) Let and > be given. Then there exists a FNONNr such that for all T?2 with
|o|| La S B
FN o PNu - -NFrUllo <.
(c) Let B > 0 and >> 0 be given. Then there eixsts FNO NIFT such that for all {ō(E) :
KN with ||0| (m(x)) $ B
FFN' ((6(E))) - NIFT ((6(E)))) <.
(d) Let B and > 0 be given. Then there exists a FNO N such that for any {6(E) e CKw
with [=]))=((*x) S B such thar
IN -N (((()))
In what follows, we prove our main theorem showing that the proposed CTFNO universal approximator of continuous operators.
Theorem A.2 (Universal approximation of CTENO), Let s, s' 2 0. Suppose G H (T,
C. (10, [10,T) H (T4; a continuous operator and K H" (Td; (T4,; Rd,) is compact subset.
Then for any 6 > 0, there exists a CTFNO N: H (Td:RR + C (0. H' (T (T<; of the form
continuous as an operator H (T C (10, x Hd (T Rd.) such thar sup IG(a) M(all()(aa(( (14 (T(R))) <e
aek
The proof is essentially split into two parts. The first part states that the Fourier projection can approximate continuous operators. To simplify notation, we use shorthand H H² (T4:
H = H (T4. and (x) = a(t, x) in the remainder of the paper.
Lemma A.3. Let G C ((0, T) H*) C (10, T] H²') be continuous operator and K C
([0, T x H²) compact subset. Then for any there exists N N, such that for all K and E [0.T]
[[g(a): GN (a)ellm <c
Proof. Since G is uniformly continuous on a compact domain G(K), for any > there exists 0 > 0 such that
[I)() g(a)ealln <6, Va E K,
16

---

Under review as a conference paper at ICLR 2023
provided It1 - tgl < 8. 0 to < t <... < <tas = T be partition of [0, T] with 8,41 - t, < and 9; H* -> H' be defined by (()),, for each i. Then, (a) in Lemma A.1 furnishes that there exists N, E N such that
[[((a): - GN. (a)),allle <c, Ya E K. Combining all, for > max (Na:i = 0, O,..., M). we have
[g((); - (a)l
- <I][(a): G(a),,lmm 119(a),, GN(a)allne + 1[Nx(a),, GN(a)elln < 3E,
for any a E K and I E [0, Tj. This proves the lemma.
The second part is to prove that any time-dependent continuous function can be approximated by three-layered feed forward network combined with the proposed time-module.
Lemma A.4. Let f [0, T| x Rd. + Rdd, be a continuous function and C Rd a compact subset. Then for any > there exists three-layered feed forward network with time
N Q L2 L1. Ce(t, = (t, a (t)Wee + vo,e(t)be). ((t,x) Qu,
satisfying
x)(())) - [(1, <e v(l,)) [0, K,
where is activation function, thw,e: [0 Dd, and the.e [0, - Rd continuous, Q projection matrix, and We, by are weight matrix and bias vector for each f, respectively.
Proof. Without loss of generality, we may assume dy 1. Construct W1 R(d++1)xd, and b E Rd++1
Ldz b = (0))
0sh
W.
with 1 (1) 0dg+1, (() ILa,+1 1. Then, LI (t, x)) (4, (2, (r,)))) E [0, Rd++1, By the universal approximation thcorem of Cybenko (1889), for any > 0 there exists W2 E (dsxxd by E R.d.. and Q E Rlxd. such that
Qo We
x
+ b2 f(t,x) <e, V(t,z) E [0,7] x K.
By setting 2',22 Id,, 4/6,2 1d,, have the desired N.
With these lemmas out of the way, are ready to provide proof of Theorem 3.1
For given continuous operator 9 C ((0. T x H²) + C (Io, T] x L²), compact subset K C
C' ([0, T] x H*) and small > 0, Lemma A.3 gives an integer N such that
sup [[[(a), (n(a)ell <, ask
for each time E [0, T]. Since K is compact, there exists B > 0 satisfying || S B for all E K and t [0, T.. For given small 5 E (0, 1), let NET be CTFNO such that for all function with || < B
[ e Pxw NFTWwl <.
In fact, 8 is chosen according to the Lipschitz constant of A, which will be constructed. As claimed by (c) in Lemma A.1 there exists CTFNO NiFT such that for all (w(E) & E KN) with [i((E) < suPrejo.r supaek FI lx +1 B' C(N) suprejo,ry suPaek ||aell +
| ((W(E)) - Nirr ((+(E))) <e
17

---

Under review as a conference paper at ICLR 2023
The continuity of FN and NIFT leads to the existence of B" > 0 that satisfies
| ((x(E))) BB, WiFF (( - B", for all (20(E): 5 E K with (x(E) S B". By (d) in Lemma A.1 and Lemma A.4 a CTFNO N such that for any {E)()) E CKw with || I B".
we can construct
GN (1, (m((9)) - N (1, (w(())) re k
sup
<E
Note that the construction of N is independent on ô and NET. Now we set d >
small that
Lip
(V) sup
|F (of(a))-rr((a)l.) <c. rE[O,T]
for all a E K, which implies that
N o o - N NFT (at) <e.
sup aek
12
Finally, for eacht t E [0, T] we can deduce the following inequality:
sup Æe
|[((a), - Naaeellzz
< E sup
[[N[(), - -(d)elli
ack
Fr IN o FN o PN (at) NIFT N Nrr (at)
= sup aEK
12
Fr EN 0 Fv o Pn (at) - Fx' N NET (at)
< 6 + sup aEK
1.a
+ sup aeK
JF''
(a.) - NIFT o N NET (as)
GN o FN PN (a)) N NET (w))
< sup oEK
+ sup
| ((=(E))) NIFT (1
(O(K))ECKN [(((E) SB"
GN FN PN (a:) No FN PN (a))l
+ sup aeK
+ sup oek
|
FN o PN (a:) N - NFT (a:)
12
@N ((w(E)) N ((((E)))))
< 2c +
sup (CCK)) (E)
|V NoFN' Pn (ax) - N (a:)
+ sup aeK
12
[ F o PN (ot) - N o NrT (at)
< 3c + sup
aeK
< 4c.
Since & E [0,T]
2"] is arbitrary, the above inequalities bring us to the desired result
sup ||9 (a) - -N -()alll((oo))))) <
B STABILITY
Proof of stability Suppose o is Lipschitz continuous activation function with Lipschitz constant C, and both sup, || (1, and |[ (1)l2² are bounded by M for every t. Let be a perturbed
18

---

Under review as a conference paper at ICLR 2023
initial condition with we have
| -oll
Then by Plancherel theorem and Fourier convolution theorem,
- = -0000000000(((000000000000000000 (x)))
= =c / (0 (x) (x)
SC / (w +0 (t,E) -
SC f [ (1)) (x) de +C fitt+c/IRR (I..
o) (E) de
s CM / (66 10) de + CM (F(0- (5)
2CM 1|0 110=
20ME.
C EXPERIMENTAL DETAILS
In this section, we present our experimental settings in detail. All experiments were conducted on single NVIDIA RTX 3090 GPU. We also detailed the algorithm of the proposed CTFNO in the one-dimensional case in Algorithms [ 2 and 3 Algorithms can be extended to the higher
dimensional case dimension-by. -yymsionn---diiiiioonn manner.
Algorithm 1 CTFNO (I-dim)
Input: input æ E Rehxn at initial = 0 with grid resolution n, output timestep / E IR, number of blocks L, width w, activation
I: grid + [0, 1/n,..., (n 1)/n] E 1x
2: I + concat(æ,, grid) E (cc++))xx
x + convix1 E Rwxn
4: for i = 1, ......., L do
5:
I +- o (SpecraFFee@tmmeCoon(((aa + (imeConvld((z 1)) e Rexx
6: end for
7: x <- convixi (x) E Rehxn
8: return x
Algorithm 2 SpectralFreg/TimeCon//
Input: input R=x" time R, shared sinusoidal embedding R R width Fourier modes N, number of heads h, learnable parameters A E (2ax(e/h)x2N, b RIx2N E C N Gershgorin stabilization constant M
1: & + FFT(x) [...,: N] RExN (filter N Fourier modes after FFT)
2: ((t) + reshape(ø(1)) E (Rx((e/h) ("reshape' denotes natural map Ie - (lx(e/)))
3: ((t)(n.)) + ottm,,
R2N for 1) E {I....,h} h}
4: @(t) t repeet((((), w/h) +b 12X22N CWNN
5: y 2 E CexN
6: - Ry E " for & (1....,NN NN
7: norm((,E) + E, (0(1)).))) E TR for g E (1,...,N)
8: norm + max(norm/MM 1, 1,0) +1
9: + y=pad(y/nnore ('pad' denotes zero- pero-ig for frequencies > N)
10: return InverseFFT(y)
19

---

Under review as a conference paper at ICLR 2023
Algorithm 3 TimeConvId
Input: input x E IRuxn, time / I2, shared sinusoidal embedding v IR + IR, width we learnable
parameters B E Rwxw, Gershgorin stabilization constant M
(t) (C)(((),, 2w) R (where (t) R+)
2: W - =b() [ ....; w] R", b + (tt) [..,, w R
3: W - Bdiag (W) Roxw
4: < Wz E R wxx
5: norm + łj Wh,, E R
6: norm < norm, / /M- 1,0) +1
7: y t- y/norm
8: return
C.I DATASETS
Spiral Data generation of Spiral, Stiff, Sawtooth, and Square follows the implementation Holt
al. (2022) unless stated. We generate the Spiral dataset by following governing equation:
ú(t) = tanh u(t),
(10)
where (-1/8
-18)-
-1/8) The other settings such the number of training and test samples,
initial value u(0), and the number of timesteps follow the implementation Holt al. (2022)
Stiff Van der Pol Oscillator We generate data by following the instructions (Van der Pol & Van
Der Mark 1927). which exhibits regions of high stiffness. The governing equation
i == y
(11)
i 1 21))))²)
(12)
where f = 1000. We sample initial conditions from sx(0) E [0.1,2] v(0) = 0.
Sawtooth & Square We also explore the benchmarks on a periodic discontinuous funetion u(t). We sample two datasets, namely sawtooth and square, and these are sampled by the equation u(t)
23 - - and u(t) 2 (1 12 ( 1/ l) I), respectively. We sample initial values
(to. u(to)) by sampling to uniformly random on the interval [0, 2T). Each trajectories are generated from intervals of [to, to + 20).
Reaction equation
The semi-linear ODE presented Section 4.1 has an analytic solution f (x: e
(t, fx) eht fx)
(13)
for an initial condition ff (I). The reaction coefficient is chosen to be 6 and the domain is unit interval. Initial conditions randomly generated as (2) sin (2m1) + 2n sin (2ækœæ)) + 23e where ~ N (0, 1) and k; is uniformly sampled integer. Analytic solutions are attained up to t 1. The solution data generated by solving (13. The spatial grid discretized with resolution of 100 along time step size of At 0.02.
Heat equation
The one-iimensional heat equation used in Section 4.1 takes the form
du -== 22 0t u (0, uo
(0, L (0,).5)
(14)
uo) (x), x 10,
with periodic boundary conditions. It describes how a quantity such as heat diffuses through a given region over time. The initial function un(x) is generated from the Gaussian random field N (0, 20 (-4 + 3.5 -25). where A refers to the Laplacian. The diffuusiityy constant is set to be 0.001 and the solution data is generated by exactly solving (44p in Fourier space on a uniform spatial grid with resolution 1024 along time step size At 0.05.
20

---

Under review as a conference paper at ICLR 2023
The one-dimnnli Burgers' equation with diffusive regularization is canonical non-linear PDE, taking the form
Burgers' equation
Ou et + du = v d'a de
x E (0, 1), te (0,1] x x (0,
0g2
(15)
u (0, un (x),
with periodic boundary condition. It is dissipative nonlinear system with shock formation and has various applications including the flow of viscous fluid dynamics. The viscosity is set to = 0.001.
Starting from initial functions sampled from N 70, + 49) 49)-22)). numerical ground truth
is generated separately where the linear diffusion component is exactly solved and the remaining nonlinear part is solved in Fourier space using forward Euler method as in (Li et al. (021). The spatial domain is discretized with resolution 1024. The training set consists of 400 periodic trajectories and each trajectory has egularly-sampled time points with At 0.005 and we test the trained models for 100 trajectories.
MuJoCo We use trajectory of physical simulation for the hopper with three joints and four body parts. Each time series is 14-dimensional, consisting of five-dimensional position, six-dimensional velocity, and three-dimensione action. [Rbbanvaa et al. (2019) generated 10,000 sequences of 200 timesteps and used 80% of the data for training and the rest 20% for evaluation. For the interpolation task, they randomly sampled 100 consecutive observation timesteps. For the prediction task, they use all 200 observation timesteps and divide them into two parts: the first 1/3 as an input for the model and the latter 2/3 as output to be forecasted. For all the tasks, they randomly sampled portion of the input timesteps at a specified rate and masked the rest of them. Note that the values used for the subsampling ratio are 10%, 20%, 30%, and 50%.
PhysioNet This a publicly available dataset of 8000 time series describing the stay of patients within an ICU over 48 hours. For each patient, 41 biomedical features are irregularly observed and converted to one minutely resolution. In (Rubanova et al., 2019), they used 80% of the data for training, and the rest 20% for evaluation, as in MuloCo. For the interpolation task, they did not subsample the data because the measurements were already sparse. For the prediction task, they halved the data so that the first 1/2 timesteps were used for inputs, and the latter 1/2 timesteps were used for outputs to be forecasted. Note that Rubanova et al. (2019) also performed per-sequence classification experiments on PhysioNet. Since Latent ODEs with RNN or ODE-RNN encoders compress the whole input data to single latent vector 20 and an MLP classifier receives only 20 as its input, such a per-sequence classification does not depend on all the other latent vectors 21,..., 2T- This is inconsistent with the goal of CTFNO, which focuses learning representations of the inherent dynamics for all timesteps. Hence we excluded per-sequence classification experiments in this paper.
Activity This dataset consists of time series from five individuals performing serveral activities (i.e. walking, standing, laying, etc). Each time series includes 12 features indicating tags attached to their belt, chest and ankles. We used the same pre-processing as (Rubanova et al. 2211). resulting in 6554 sequences of 211 time points. The task is classify each time point into one of seven types of activities and we performed per-time-point classification experiments as in (Rubanova el al. 2019).
Plane Vibration The dataset consisted of time 73627. with five attributes recorded per timestamp. We randomly take out 10% of data to make the time series irregularly sampled. Following the implementation of (Xia et al. 2021). we use the first 50% of data as our train set, the next 25% as a validation set, and the rest as test set. We divide each set into partitions of consecutive 64 timestamps of the seguually-sampled time series, and our goal to forecast consecutive timestamps starting from the last timestamp of the segment.
C.2 NETWORK STRUCTURE
Time Embedding order to obtain ptt) and @(t), we first adopt a positional embedding of transformer Vaswani al. 2017). A time encoding function converts an one-dimensional time into
multi-dimeensoonll input by passing the time through trigonometric functions of varying frequencies:
t - (sin (wif), , cos (wxtt) We 10-
21

---

Under review as a conference paper at ICLR 2023
RC0
(1))
RC2)
Sharing netl
RC3)
Embedding
MLP
A
()
RC4)
(1
(t)
R
Figure 4: The architecture of multi-head CTENO.
where i runs a range of integers {O..... m 1) with encoding dimension Then, we pass MLP layers with the number of hidden dimension c. The contextual visualization of sharing net (p()) can be found in Figure 4
Fourier kernel Rg (E), that is,
Time modulation multiples a single value şo (4,5) E I to a dy x d, Fourier kernel
Re (t, pe (t, Rs (E).
However, often there multiple different aspects channel elements attend to, and scalar multipli- cation may not be a good option for it. To attempt to lift this restriction, we employ parallel heads. Specifically, we divide the Fourier kernel R2 into h kemne blocks 12(2) (E) E Rdx xd= for i = 1... h, where h is the number heads and dy = d./h.. Afterward, we concatenate the heads and combine them with a final kernel matrix. (See Figure 4 This operation can be expressed as:
Re (t. {) = Concat (R() ..... R(
R (t. = -0 (t. R (E), i= I....h
The multi-head kernel allows the model to share different representations different channels.
Model architecture for 2D data While the Fourier kernel R plays a key role in FNO models for learning the solution operator PDEs, requires quite many parameters (de for each frequency). This parameter redundancy worsens for two-dimensional data; images in Section 4.33 To overcome parameter inefficiency, we modify the Fourier kernel as follows. One modification is the use of block- diagonal structure of R, as proposed in (Guibas et al. (02T). This reduces the number of parameters, however, the diagonal structure does not fully mix information between channels, which may lessen the expressive power of the model. To compensate for this, we additionally introduce local 1 x convolution layer V in Fourier space. The overall architecture can written as
Le (v) (z) (Wev (z) F (Re (E) F (v) (5) (zF (E) (z)).
(16)
We empirically show that this improves the training ability of FNO with significantly less parameters.
Lifting Layers for Real Application Fourier transform may be hard embody discontinuous functions because Fourier filters are global sinusoidal functions. As a consequence, FNO has difficulty in learning sharp features (Lu et al. (022). many applications, data discontinuous, such as images. Hence, the performance of FNO models is plagued by the discontinuous nature of such input data. Contrary to regular PDE problems where the pointwise lifting is adequate, we hereby propose an alternative lifting layer to provide a smoother form of the data. A low pass filter is the standard way of smoothing out the data; tends to retain the low frequency modes while truncating out the high frequency information. To cutoff high spatial frequency. we additionally introduce low pass filter after the original lifting layer. For parameter efficiency, the low pass filter is implemented by an 1 x 1 convolution layer in Fourier space, introduced in the preceding paragraph. In this way, input data is smoothed by decreasing the disparity between neighboring values without loss of information.
22

---

Under review as a conference paper at ICLR 2023
Table 7: The number of parameters of each model in synthetic experiments. Low indicates the experiments on Spiral, Stiff, Sawtooth, and Square. High denotes the high dimensional experiments, including Heat and Burgers' equations. LODE, NF, NL. DON, and PDN stand for Latent ODE, Neural Flow Neural Laplace, DeepONet, and POD-DeepONet, respectively.
Data
NODE
ANODE
LODE
NF
NL
DON
PDN
CTFNO
Low
17025
17282
18565
18307
17194
17858
High
5.24M
5,45M
16,57M
4.51M
1.58M
1.78M
2.38M
C.3 HYPERPARAMETERS AND OTHER EXPERIMENTAL DETAILS
We choose the hyperparameters and other experimental details that delivered the best performance for the baseline models. Note that each model has different optimal optimizer settings, so we carefully choose the learning rate and its decay rate that show the best performance.
C.3.1 SYNTHETIC
Low dimensional data Low dimensional data includes Spiral, Stiff, Sawtooth, and Square. All the experiments on these datasets are trained for 1000 epochs with batch size of 120. We do not normalize data since all data are properly bounded. Implementation of NODE, ANODE, Latent ODE, Neural Flow, and Neural Laplace follow (Holt et al., 2022), other than the data normalization. For CTENO, a single data point is linearly transformed into a 16-dimensional vector. Then, we regard the temporal dimension of 100 as a channel size and convert learning rate of le-3 with no decay. We do not use the stabilization constant and use Gaussian error Linear Units (GeLU) (Hendrycks Gimpel 2016) activation in all FNO-based experiments. In the time-embedene emeeemiedd layer, we employ Sigmoid Linear Units (SiLU) (Elfwing et al., 2018) activation function. Note that all experiments for CTFNO use the same activation function.
16 by lifting layer. We use the
Niih--dimensional data In Reaction ODE, Heat, and Burgers' experiments, which are high di- mensional datasets, we set comparison models to have a large number of parameters, at least 1M. In NODE and ANODE experiments, we set ODE function ft, u(t) five layer Multi-Layer Perceptron (MLP), of 1024 units. For ANODE, we set the augmented dimension of zeros to 100. We employ the learning of 1e-5, and the number of epochs of 2000. Other hyperparameters follow the NODÉ implementation of Holt et al. (2022). Moreover, in the implementation of Neural Laplace, we set the latent dimension and the number of hidden units to 64. Other hyperparameters are set the same as -ow-dimensional data experiments. In the implementation of Neural Flow, we use a 32-layered coupling flow. Each coupling flow has two hidden layers with 128 hidden dimensions. Timesteps are embedded into 32 sinusoidal features. We set the learning rate of 1e-3 with the decay rate of 0.8 for every 200 epochs. Other hyperparameters for Neural Flow follow the synthetic experiments et al. 2021). In DON and PDN experiments, we use the number of channels of 256 with six residual layers for each branch and trunk net. We use SiLU activation, the training epochs of 20000, the batch size of 50, and the learning rate of 1c-3. The hyperparameters of ENO-based models, including CTFNO, are summarized in Table g We train ENO-based models with a batch size of 20, the training epochs of 2000, and the learning rate of 1e-3 with decay rate of 0.8 for every 100 epochs.
(Biloš)
C.3.2 REAL TIME-SERIES
Our implementation builds on open-source codes (MIT License). For all real time-series datasets, we refer to the results of RNN-VAL ODE-RNN and Latent ODEs reported in (Rubanova et al. 2019). For CTFNO, we commonly use the number of heads of one and two padding dimensions. Note that all layers are equipped with GeLU activation. except for only time-embedding layers with SiLU activation. Other model hyperparameters for CTFNO are shown in Table 9 Moreover, use the batch size of 50, and the Adamax optimizer (Kingma & Ba 2014) with the learning rate le-2.
httass//4ttbbb.coo////lll///////
23

---

Under review as a conference paper at ICLR 2023
Table 8: Hyperparameter settings of CTFNO, FNO2d, and FNOseq on synthetic data. Low indicates the experiments on Spiral, Stiff, Sawtooth, and Square data. Layers, Modes, and Channels are the number of Fourier layers, Fourier modes, and channels, respectively. The first value in Time channels is the dimension used for time embedding and the second value is the dimension of sinusoidal embedding of time. Params denotes the size model parameters.
Model
Data
Layers
Modes
Channels
Time channels
Params
CTENO
Low
3
4
(32, 16)
17858
Reaction
32
64
(512, 128)
1.82M
PDEs
2
64
64
(512, 128)
2.38M
FNO2d
Reaction
(32, 16)
32
7.00M
PDEs
3
(64, 16)
32
12.00M
FNOseq
PDEs
64
64
2.02M
Table 9: Hyperparameter settings of CTFNO on real-world time-series data. Layers, Modes, and Channels are the number of Fourier layers, Fourier modes, and channels, respectively. The first value Time channels is the dimension used for time embedding and the second value is the dimension of sinusoidal embedding of time. M stands for the stabilization constant. Params denotes the number of model parameters.
Data
Layers
Modes
Channels
Time channels
M
Params
MuJoCo PhysioNet
10
32
(64, 32)
10
150K
10
32
(64, 8)
68K
Activity
10
32
(64, 32)
10
124K
Plane Vibration
8
(16,8) 8)
1.2
6K
Latent ODE framework We use the same Latent ODE framework proposed in (Rubanova et 2019). First, a given input time series is passed through an RNN encoder. Here, latent vector 2o is sampled using the last feature vector as the mean μ and standard deviation a, which the same as VAE (Kingma & Welling, 2014 Szegedy et al.) (013). Now, assuming implicit dynamics in the latent space with Eg as the initial point, each model (such as Neural ODEs and CTFNO) returns an output vector at the desired timesteps. Finally, after passing through decoder with fully connected layer, the difference between the output and the target becomes the loss function for training.
MuJoCo Overall experimental settings follow (Rubanova et al. 2019), unless stated. We the latent dimension of 20, the hidden state dimension of TO0 for the RNN encoder, and the training epochs of 300. Since Biloš et al.) (02II) II did not contain the results of interpolation and prediction tasks with different subsampling ratios, we carefully trained Neural Flow, following the official implementation Following (Rubanova et 2019). we employ the importance weighted likelihood loss (Burda et al. 2015).
PhysioNet Overall experimental settings follow (Rubanova et al. 2019), unless stated. We use the latent dimension of 20 and the hidden state dimension of 40 for the RNN encoder. The training epochs are 200 and 50 for the interpolation task and the prediction task, respectively. As in (Rubanova et al.)
2019), employ the importance weighted likelihood loss (Burda et al. 2015) without leveraging classification loss.
Activity Overall experimental settings follow (Rubanova al. 2019). unless stated. We use the latent dimension of 20, the hidden state dimension of 100 for the RNN encoder, and the training epochs of 200. For classification, the cross entropy loss was used, as (Rubanova al. 2019.
http:://ggttttbb.cmm//////////// w1wpaal----oo-e-eremmeme experimennt
24

---

Under review as a conference paper at ICLR 2023
Table 10: The number of parameters for each models for image classification.
Data
CNN
ANODE
FNO
MNIST
78K
83K 168K
66K
CIFARIO
154K
153K
C.3.3 STUDIES ON STABILITY
Plane Vibration this experiment, we follow the settings of Xia et (2021) For CTFNO. five-dimensional attributes are concatenated with input time and then embedded into 20-dimensional space by a linear transformation. We regard temporal dimension 64 as channel size and convert to eight by a lifting layer. We train the model for 500 epochs and use the learning rate of 1e-2 with decay rate 0.5 per 100 epochs. We trained and evaluated with MSE loss.
Image classification All of the experiments use gradient clipping of 10 and cross-entropy loss. We use a learning rate of 1e-3 and weight decay of le-4 for ANODE and CNN. Other implementation details of ANODE follow Dupont et al. (2019). We use five convolutional layers in CNN experiments. For MNIST, we apply a kernel size of seven for the first two convolutional layers and 32 number of channels. For CIFARIO 10, number of channels. Other convolutional layers have a kernel size of three. We train CNN for 30 epochs. Note that accuracy of both ANODE and CNN tend to decrease after the designated number of epochs.
kernel size of seven for the first convolutional layers and 64
In FNO experiments, FNO with weight decay (WD) and FNO with normalization (Stab), we train models for 100 epochs with the learning rate of 5e-3. We used three Fourier layers with six Fourier modes. For MNIST, we employ the number of channel of 32, and three-dimensional zero padding. For CIFAR 10, we employ the number of channel of 64, and four-dimensional zero padding. For the experiments with decay, we choose the decay rates 1e-6 and le-4 for MNIST and CIFARIO 10, respectively. For the experiments with normalization, we employ the spectral norm of 1.2 for the normalization bounds. Note that GeLU activation function was used for both CNN and FNO experiments.
To estimate the robustness of the trained model, we consider three commonly-used perturbation schemes, namely random Gaussian perturbations, FGSM, and PGD attacks. For random Gaussian attack, we use zero mean Gaussian noise with standard deviation of 50/255, 100/255 for MNIST and 15/255, 20/255 for CIFAR10. 10. For adversarial attacks, we use 30/255, 50/255 attack for MNIST and 2/255, 5/255 for CIFAR10. The number of steps of PGD attack is 10.
D FURTHER RESULTS
D. COMPARISON TO VARIOUS TIME-INJECTED ENO-BASED MODELS
In Section 3
we construct various basclines injecting time t into the network for comparative analysis of CTFNO. This section includes implementation details. Since CTFNO puts into weights of the Fourier layers, the other way inject t is join it in the input function uz( intermediate features un (x) 0i (z), Cr (2). To cover variants of such cases, we constructed three other base- lines adding L into the input or feature space discussed in Section 3.2 The architecture of these
three scenarios is pictorized in Figure [
Comparison results of these three baselines together with FNO2d (proposed in the original FNO paper) and CTFNO in ODE/PDE problems are reported in Table [T] The results show that our model significantly outperforms other models. The results further show that three scenarios, which are structures that can continuously handle arbitrary time f, are similar to or worse than FNO2d that can only evaluate the solution value at times on a specific grid. Even though all three baselines and CTFNO model continuous time operators, it was found through the experiments which only the structure of CTFNO is effective. The difference from the baselines is that CTFNO inserts temporal information into weights. The rationale of the way of imposing temporal information on CTFNO is based on Green's 's formula of denendependnnt PDEs. The results validate that it is much more effective construct model structure based on the theoretical Green's 's formula approximate the
25

---

Under review as a conference paper at ICLR 2023
1
(a) Scenario
P
x(txx)
(1)(1,x)
(o()x)
2y (,,
P2((,
Py (b,
sharing net
(())
I
(b) Scenario
L2
Q
x(txx)
Ma (x)
Pp(C,))
(z(t.x)
(t,,x)
sharing
((()
Au
I
(c) Scenario 3
L2
Q
x(txx)
Mp
Po((,))
(2(t,)
v2(t.))
Vt(t.x)
Figure 5: The visualization of architecture of three baselines: (a) Concatenates the input time f to an input function. (b) Concatenates the encoded time (tt) to the lifted input function. (c) Concatenates the encoded time to the lifted input function and intermediate features. Here, all denotes learnable affine transform.
solution operator of PDE. In summary, validate through additional experiments that the structure of CTFNO designed based on the solution operator of the PDE (what we want to learn) very effective in actually learning the PDE.
Table 11: Results of time-aware FNO models in Synthetic Data
Relative 1?2 (x10-2)
Reaction ODE
Heat
Burgers
Scenario
0.19
11.60
19.6
Scenario
0.13
2.84
18.6
Scenario
0.14
2.68
18.9
FNO2d
0.16
0.58
10.32
CTFNO
0.04
0.22
5.68
D.2 STABILITY
As we discussed in Section 3.4. the upper bound M on the R (1, 1l2 and || (1,-)]l2 emanates from the necessity of stabilizing the proposed model. In Section 4.3 conducted experiments to evaluate the robustness of our proposed stabilization scheme, and compare it with the vanilla FNO and weight decay. From Tables and 6 FNO trained with stabilization outperforms vanilla FNO on all types of perturbations.
M is user-defined hyperparameter. Here, we analyze the performance of FNO illustrate how increasing stabilization parameter M leads to more robust model. Experiments are carried out on CIFARIO and we perturb images by PGD-5/255 attack. The left panel of Figure shows the accuracy and robustness of the learned models for several values of M. We can see that imposing stability does come with trade-offs between test accuracy and robustness on the attack. The accuracy higher low M and becomes decreasing as MM increases, which makes the model more robust various attacks. Moreover, the models with weak stabilization show brittle training procedures. Also, the model degrades the overall performance stabilization effects become dominant. Results certify that small corruptions or extra noise the input are not likely to change the output of the network with proper stabilization.
Morcover, the stability of the model is related to how well the model generalizes on the data on which has trained, which in turn is related overfitting. This can be confirmed in Figure (right).
26

---

Under review as a conference paper at ICLR 2023
It presents the evolution of the loss function of models with and without stabilization. As see, after some iterations, the test accuracy of the model without stabilization has started to decrease. This means that the longer train the model, the training data. This is evidence of overfitting. On the other hand, the model with stabilization keeps increasing its test accuracy. This confirms that the stabilization prevents overfitiigg and helps the model to work better on unseen data.
specialized the weights will become to the
15
Clean Ace (%)
Attack Ace (%)
Test Ace (%)
50
10
0.5
40
Clean
0.4
M-1.2
30
Noisy
None
0.3
0.5
1.2
1.6
4
10
None
20
40
60
80
100
M
Epoch
Figure 6: The Trade-off between the accuracy and robustness on PGD-5/255 attack (left) and observation for overfitting (right). Tests are conducted CIFARIO (4.3).
The relation between the stabilization and overfitting was also discussed in Section 4.3 on the Plane Vibration dataset. By increasing M, the training MSE loss gradually decreases, while the test MSE eventually goes up (see Figure 3). Figure 7 plots the distribution of the L norm of weights of trained CTFNO with M 1.2 and 10 on the plane vibration data. The weights of the model trained with .M 10 have larger L norms. As analyzed in Theorem 3.2 the weights that have large spectrum make the network unstable. More precisely, the spectrum of the weights grows in size to handle the specifics of the training data. As the weights become specialized the training data, overfitting occurs. On the other hand, the trained model with M 1.2 learned weights with a spectrum smaller than 1. It forces the network to have small changes in output for small changes in the inputs, which gives more ability to generalize better.
w/ normalize
w/o normalize
30
30
Singular Value
Singular Value
20
20
2
2
10
10
1
0
2
3
4
5
6
3
4
5
6
7
Timestep
Timestep
Figure 7: Plot of the singular values of weights of CTFNO with (left) and without (right) stabilization on the plane vibration dataset ($4.3)
D.3 GENERALIZATION RESOLUTION
Table 12: Resolution invariance of FNO.
Classical neural networks that map between finite- dimensional Euclidean spaces are grid-dependent, and thus cannot generalize over image resolution. By viewing images as functions, FNO is not tied to specific dis- cretization and can be applied to arbitrary discretization
Dataset
Original
Resolution
x2
x4
x8
MNIST
98.8
98.6
98.6
98.6
CIFAR10
65.5
64.8
64.6
64.7
27

---

Under review as a conference paper at ICLR 2023
of a (x). To validate the resolution invariance of FNO, classify high-resolution images using FNO trained on the original now-resolution images. Table [2 validates that FNO is able to learn from coarse images and generalize to higher resolutions.
D.4 SPEED IMPROVEMENTS
Table 13: Measured training and inference time
Model
Training (second/epoch) Inference (second/epoch)
Neural ODE
23.6
2.20
Neural Flow
4.38
0.62
CTENO
7.39
0.89
We carefully measured execution time on the MuloCo experiment ($4.2) and the results are reported in Table 13 The main reason for this difference is that numerical integration used in neural ODEs is computationally expensive, and Neural Flow and CTFNO are neural operators that directly represent the solution trajectory of ODE and PDE, respectively. In other words, the neural operator's replacement of the role of ODE solvers has contributed significantly to overcoming the increased complexity of PDEs compared to ODEs and reducing time costs.
D.5 ADDITIONAL SYNTIETIIC RESULTS
We investigate the generalization ability of our method to time points beyond those that were used for training on the wwo-dimensional spiral data in Section 4.1 Predicted and extrapolated trajectories are depicted in Figure together with the ground truth. Each model is trained to predict spiral trajectories at 100 time points (blue lines Figure (). And we run trained models to forecast 500 future time steps (red lines in Figure 8 We can observe that our CTFNO is better generalizing for extrapolation compared to baseline models. Our CTFNO correctly extrapolates the spiral trajectory for beyond the training time interval, converging to the equilibrium of the spiral. From Table we have that CTFNO best approximates the spiral comparing against the benchmarks, achieving the lowest RMSE, The results depicted in Figure @anreeeconiimn that CTFNO accurately captures the dynamics of the spiral during training and predicts well even for non-training times, based on memories analyzed in the past.
We also include additional heatmaps for reaction, heat, and Burgers' equations and
4.
See Figure 9
10
E
BROADER IMPACT
We introduced a new framework for modeling temporal dynamics of observed data. has wide range of potential applications, some of which we investigated in this paper. We explored a healthcare dataset and here we hope to bring affirmative impact medical applications. We also examined the modeling the vibration of an airplane and we expect it to contribute to the advancement of airplane designs and civil engineering. Furthermore, as many problems have arisen in sciences and engineering tied with complex PDE systems, we expect that our work has potential applicability in the enormous area such as climate forecasting, epidemics, molecular simulations, miero-mechanics and modeling turbulent flows. PDEs can also be applied in the development of military equipment. As with all numerical methods, however, is not a work of developing a technique to go to warfare, and we hope and encourage of our model to concenter on the positive impact of this work.
28

---

Under review as a conference paper at ICLR 2023
ANODE
ANODE (Zoomed)
LODE
LODE (Zoomed)
Neural Laplace
Neural Laplace (Zoomed)
Neural Flow
Neural Flow (Zoomed)
CTENO
CTFNO (Zoomed)
Groud Truth
Prediction
Extrapolation
Figure 8: (Left) Prediction and extrapolation for beyond the training time points of the two- dimensional spiral ($4.J), and (Right) the zoomed extrapolation trajectories with a comparison to ground truth. CTFNO performs best not only for prediction but also for extrapolation.
29

---

Under review as a conference paper at ICLR 2023
True
1.0
2
0.5
0
0.0
0.25
0.50
1.00
CTFNO
Difference between true solution and CTFNO
0.08
1.0
0.5
0.04
0.02
0.0
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Neural Flow
Difference between true solution and Neural Flow
0.08
2
2
0.06
0.5
0.04
0.02
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Neural Laplace
Difference between true solution and Neural Laplace
0.08
1.0
2
2
0.06
0.5
0.04
1
0.02
0
0.0
0
0.00
0.25
0,50
1.00
0.25
0.50
1.00
FNO2d
Difference between true solution and FNO2d
0.08
1.0
0.5
0.04
0.02
0.0
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure 9: Heatmap of exact and predicted solution for reaction ODE ($ model can exactly represent the dynamics the reaction ODE.
4.1
This shows that our
30

---

Under review as a conference paper at ICLR 2023
True
0.2
0.1
0.1
0
0.2
0.00
0.25
0.50
0.75
1.00
CTТNO
Difference between true solution and CTFNO
0.2
0.1
0.06
0.0
0.04
-0.1
0.02
0.00
0.00
0.25
0.50
0.75
1,00
0.00
0.25
0.50
0.75
1,00
Neural Flow
Difference between true solution and Neural Flow
0.2
0.08
2
0.1
2
0.06
0.0
0.04
0
0.2
0
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Neural Laplace
Difference between solution and Neural Laplace
0.2
0.08
0.1
0.06
0.0
0.04
0.1
0.02
0.2
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
FNO2d
Difference between true solution and FNO2d
0.2
2
0.1
2
0.06
0.0
0.04
0.1
0.02
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure 10: Heatmap of exact and predicted solution for heat equation
($4.1
31

---

Under review as a conference paper at ICLR 2023
True
0.2
0.1
0.0
0.1
0
0.2
0.00
0.25
0.50
0.75
1.00
CTТNO
Difference between true solution and CTFNO
0.2
0.1
0.06
0.0
0.04
-0.1
0.02
0.00
0.00
0.25
0.50
0.75
1,00
0.00
0.25
0.50
0.75
1,00
Neural Flow
Difference between true solution and Neural Flow
0.2
0.08
2
0.1
2
0.06
0.0
0.04
0
0.2
0
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Neural Laplace
Difference between solution and Neural Laplace
0.2
0.08
0.1
0.06
0.0
0.04
0.1
0.02
0.2
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
FNO2d
Difference between true solution and FNO2d
0.2
2
0.1
2
0.06
0.0
0.04
0.1
0.02
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure 11: Heatmap exact and predicted solution for Burgers' equation (4.1
32


Reviews:
[{"review_id": "TbOxpdEGo-8", "reviewer": "Reviewer_dm3F", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an extension to the Fourier Neural Operator, called CTFNO that can continuously condition on time, including a universal approximation theorem for the new architecture, and a stability guarantee that should defend against noisy observations. The conditioning on the time is achieved via the adoption of a positional embedding as used in Transformers (Vaswani et al., 2017), and added block wise via shared networks both to the spatial and to the frequency domain.", "summary_of_the_review": "The ideas are novel and interesting. The current way of how the experiments are presented and compared are weak and hard to judge. I recommend the authors to improve upon the experiments section!", "strength_and_weaknesses": "Strength:\nThe paper addresses an important topic, namely that many neural PDE surrogates are trained on fixed time grids. Whereas DeepONets like operator learning methods have time dependencies naturally encoded via the trunk net, Fourier Neural Operators do not.\n\nThe Gershgorin disc normalization is an interesting way of enforcing stability. \n\n\nWeaknesses:\nUnder the hood, the newly proposed architecture conditions on the time variable. Conditioning is a super important property in various computer vision tasks and natural language translation tasks. As such a lot of different techniques have been developed:  for example, the encoding of the input into a vector representation by using sinusoidal Fourier embeddings as is common in Transformers, or AdaGN (Nichol et al., 2021), which is based on affine transformations of normalization layers via projections of the embeddings. I am not fully convinced with the three presented baseline approaches. They read rather week, but ok, it might just be really hard to condition for FNOs. A different why of encoding parameters in the Fourier space can be found e.g. in Poli et al. Potentially also relevant is https://openreview.net/forum?id=Uk40pC45YJG\n\nExperiment-wise I am missing how the new time modulation improves stability. I would have liked to see long rollouts, and the conditioning of different time windows.  In that sense I have also waited for studies which address the effect of the Gershgorin disc normalization. That is missing in the current version and makes me wonder what the whole theoretical reasoning is about.\n\nAlso experiment-wise for e.g. heat equation and Burgers’ equation it is not 100% clear to me what is the input to the neural network surrogate and what is the output, and where the conditioning on time comes into the game. I would have expected curves showing errors for e.g. 100 step rollouts, 200 step rollouts, and so on. Also I am missing how FNO2d is used (input and output?)  I am also missing parameter and runtime comparisons. All in all that makes it really hard to judge the experiments.\n\nNichol, Alexander Quinn, and Prafulla Dhariwal. \"Improved denoising diffusion probabilistic models.\" International Conference on Machine Learning. PMLR, 2021.\nPoli, Michael, et al. \"Transform Once: Efficient Operator Learning in Frequency Domain.\" ICML 2022 2nd AI for Science Workshop. 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6ciOz4Ne7dp", "reviewer": "Reviewer_Jaqe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Similar to as is done in done in diffusion models for U-net layers, this paper applies time-conditioning to FNO layers and demonstrates it applicability on a wide variety of tasks. The paper also proposes a method to normalize the Fourier kernel so that it's more perturbation resistant.", "summary_of_the_review": "Overall there are useful ideas in the paper. However it needs better focus on its identified contributions. Clear discussion of limitations would also be quite useful.", "strength_and_weaknesses": "**Strengths**\n\nTime-conditioning in the Fourier space as done in this paper is definitely novel. Similarly row-wise normalizing the Fourier kernel is a useful trick to know. The paper applies their method on a very wide variety of problems, demonstrating the general applicability of their approach.\n\n\n**Weaknesses**\n\nClaims abut lack of expressiveness of ODEs in the second introductory paragraph are incorrect. NeuralODEs are difficult to train in practice on higher dimension problems and there have been many different approaches to try fix some of these issues in various individual contexts. S4 [1,2] class of methods actually try to solve this problem generally while being computationally less expensive. Proposing a \"PDE\" based framework seems ill-motivated and calling FNO as a PDE solver is also incorrect. A PDE solver will solve the given equation at whatever timestep you want answers for, FNOs first need to be trained on solutions from that PDE solver to make a \"PDE surrogate\" and can still only \"solve\" that particular equation and will not generalize to change in the parameter of that equation. Similarly as this paper notices too that FNO has a notion of discrete time and is often trained to recursively make predictions and can't give you predictions at whatever timestep you want. I know the literature in this space is a mess and I don't blame the authors per se but some precision in our language is required for effectively communicating with the differential equations community if you are supposedly solving problems with their approaches.\n\nMention of U-net based diffusion models is another remiss. There too scalar time is used to condition the different layers of neural network in a manner that is not _too_ different from what this paper proposes. [3] demonstrates how applying the same techniques and architecture is quite effective for PDE modeling (orders of magnitude better than FNO) and also does time-conditioning similar to this paper. \nOverall, expressiveness of FNOs is the reason for superior performance and more details about training strategies and baselines is necessary to actually make sense of results.\n\nIn the Image classification experiments it's unclear what CNN model was used as baseline and there are methods to improve CNN style model accuracies too which would ideally be compared as well. I'm not sure if it even should be in main text of the paper, it just seems so disjoint from the title. Moreover it's unclear if this \"stabilization\" won't hurt performance on difficult 2D PDE modeling tasks like Navier-Stokes.\n\n[1] https://arxiv.org/abs/2111.00396\n\n[2] https://arxiv.org/abs/2206.03398\n\n[3] https://arxiv.org/abs/2209.15616", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4sOQFkZYcly", "reviewer": "Reviewer_FEU2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A continuous-time version of NFO is presented, which can learn both ODEs and PDEs. \n", "summary_of_the_review": "This paper presents an *DE method that achieves amazing results, but does practically nothing to explain either what is happening or what is being learned or why the method is so good. I don’t think just getting SOTA results is science, one also needs to provide insights or the \"why\". \n", "strength_and_weaknesses": "s: Fantastic results.\n\ns: perhaps a principled method (maybe, not sure)\n\nw: Weak clarity. I didn't understand the method, and hence I can't evaluate it if has value. The paper follows the convention where equations are given, but not explained, exposed or illustrated. The reader is assumed to be an expert in this domain, which limits the papers audience and thus impact. For instance, eq1 is already weakly presented, but afterwards the computation flow, Nemytskii and FNO are gibberish to me. Similarly eq 4 is just given as a blurp with no explanations. I couldn't follow the time modulations. I didn't understand the ablations: why do we care about this? (The stability stuff was very clear). Furthermore, the model itself is not even defined (!), except in fig2 that I can't follow. In experiments I couldn't follow what task was being solved, and whether table 2 was training or test errors, or what the percentages or pred/interp. settings meant in table 3.\n\nw: Poor motivation. The sec 3.1. explains motivation, but I can’t follow what problem is being solved. Apparently ODEs have some issues fitting toy cases, but then we move to PDEs instead (without citing neural PDE works, or comparing to them, eg. Iakovlev'21). The main motivation is perhaps having a model that can handle both ODE and PDEs, but it’s unclear why do we want this.\n\nw: poor contextualisation. The results are fantastic, but there is little contextualisation on how this method differs from others, or what are its limitations. Does this method solve now everything, does it have any weaknesses?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3rA_ubwahMQ", "reviewer": "Reviewer_92q6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an extension of Fourier neural operators (FNO) to have operators evolving over time. The paper proposed an architecture to cooperate the time embedding into FNO. The experiments are conducted on synthetic datasets given PDEs and on real-world time series data. The model outperforms alternative methods. There is also an experiment on the robustness to adversarial attacks.", "summary_of_the_review": "The paper did a good job describing the proposed method and the emperical results are impressive. However, it's not clear what factor leads to the effectiveness of the proposed models.", "strength_and_weaknesses": "### Strength\n- The paper is well-written, and well-structured.\n- Theoretical results support the claim on the stability of the proposed method.\n- Experiments are comprehensive, covering multiple aspects from learning dynamics and robustness to adversarial attack.\n### Weaknesses\n- Although the suggested time modulation shows impressive performance, it’s hard to justify why it works well with this specific choice (in Table 1). I’m not familiar with StyleGAN2. So if there is any good justification from StyleGAN2, it would be nice to have a brief discussion of the benefits of such an architecture design.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject