Characteristic Neural Ordinary Differential Equations
Authors: Xingzi Xu
Ali Hasan
Khalil Elkhalil
Jie Ding
Vahid Tarokh
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Neural ODE
Differential Equation
Method of characteristics

Abstract: We propose Characteristic-Neural Ordinary Differential Equations (C-NODEs), a framework for extending Neural Ordinary Differential Equations (NODEs) beyond ODEs. While NODEs model the evolution of a latent variables as the solution to an ODE, C-NODE models the evolution of the latent variables as the solution of a family of first-order quasi-linear partial differential equations (PDEs) along curves on which the PDEs reduce to ODEs, referred to as characteristic curves. This in turn allows the application of the standard frameworks for solving ODEs, namely the adjoint method. Learning optimal characteristic curves for given tasks improves the performance and computational efficiency, compared to state of the art NODE models. We prove that the C-NODE framework extends the classical NODE on classification tasks by demonstrating explicit C-NODE representable functions not expressible by NODEs.  Additionally, we present C-NODE-based continuous normalizing flows, which describe the density evolution of latent variables along multiple dimensions. Empirical results demonstrate the improvements provided by the proposed method for classification and density estimation on CIFAR-10, SVHN, and MNIST datasets under a similar computational budget as the existing NODE methods. The results also provide empirical evidence that the learned curves improve the efficiency of the system through a lower number of parameters and function evaluations compared with baselines.

Characteristic Neural Ordinary Differential Equations
Anonymous Author(s) Affiliation
Address
email
Abstract
We propose Characteristic-Neures Ordinary Differential Equations (C-NODEs), framework for extending Neural Ordinary Differential Equations (NODEs) beyond ODEs. While NODEs model the evolution of latent variables the solution to ODE, C-NODE models the evolution of the latent variables as the solution of a family of first-order quasi-linear partial differential equations (PDEs) along curves on which the PDEs reduce to ODEs, referred to as characteristic curves, This in turn allows the application the standard frameworks for solving ODEs, namely the adjoint method. Learning optimal characteristic curves for given tasks improves the performance and computational efficiency, compared to state of the NODE models. We prove that the C-NODE framework extends the classical NODE on classification tasks by demonstrating explicit C-NODE representable functions not expressible by NODEs. Additionally, we present C-NODE-basee continuous normalizing flows, which describe the density evolution of latent variables along multiple dimensions. Empirical results demonstrate the improvements provided by the proposed method for classification and density estimation on CIFAR-10 10, SVHN, and MNIST datasets under similar computational budget as the existing NODE methods. The results also provide empirical evidence that the learned curves improve the efficiency of the system through lower number of parameters and function evaluations compared with basclines.
2
11
13
15
17
19
Introduction
21 Deep learning and differential equations share many connections, and techniques the intersection have led to insights in both fields. One predominant connection is based on certain neural network architectures resembling numerical integration schemes, leading the development of Neural Ordinary Differential Equations (NODEs) S NODEs use neural network parameterization of latent variable that the solution to the learned ODE. A central benefit of NODEs is the constant memory cost, where backward passes are computed using the adjoint sensitivity method rather than backpropagating through individual forward solver steps. Backpropagating through adaptive differential equation solvers to train large NODEs will often result in memory outage, as mentioned 5 Moreover, NODES provide a flexible probability density representation often referred continuous normalizing flows (CNFs). However,
22
23
24
25
an ODE to learn a mapping from observed variables
26
27
29
31 since NODEs can only represent solutions to ODEs, the class of functions somewhat limited and may not apply more general problems that do not have smooth and one-to-one mappings.
39 To address this limitation, a series analyses based on methods from differential equations have been employed to enhance the representation capabilities of NODEs, such as the technique of controlled differential equations [24), learning higher-order ODEs [32. augmenting dynamics [0].
35
36 and considering dynamics with delay terms (55). Moreover, certain works consider generalizing the
37
ODE case to partial differential equations (PDEs), such as in [40 44). However, these methods do
38 the adjoint method, removing the primary advantage of constant memory cost. This leads us
39 to the central question motivating the work: can we combine the benefits of the rich function class of
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

40
PDEs with the efficiency of the adjoint method? To do so, we propose a method of continuous-depth neural networks that solves PDE over parametric curves that reduce the PDE to an ODE. Such curves are known as characteristics, and they define the solution of the PDE in terms of an ODE
41
43
The proposed Characteristic Neural Ordinary Differential Equations (C-NODE) learn both the characteristics and the ODE along the characteristics to solve the PDE over the data space. This allows for richer class models while still incorporating the same memory efficiency of the adjoint method. The proposed C-NODE is also extension of existing methods, as it improves the empirical accuracy of these methods classification tasks and image quality generation tasks.
15]
44
45
46
47
Related Work
48
We discuss the related work from both machine learning and numerical analysis perspectives.
50
Single Characte ristiic (ODE) Multiple Characteristics (PDE)
51 2.1
Machine Learning and ODEs
NODE is often motivated as continuous form of Resid- ual Network (ResNet) [7]. since the ResNet can be seen as forward Euler integration scheme the latent state Specifically, a ResNet is composed of multiple blocks where each block can be represented
48
Figure 1: Comparison of traditional
U++1 = u1 + f(u,. 0),
NODE (left) and proposed C-NODE (right). The solution to NODE is the solution
where 4y the evolving hidden state at time and f(u, 0) represents the gradient at time t. namely du alizing the model to step size given by At, we have:
single ODE, whereas C- NODE represents a series of ODEs that form the solution to a PDE. Each color in C-NODE represents the solution to an ODE with a different initial condition. NODE represents a single ODE, and can only represent u(2, t) along one dimen- sion, for example, u(x 0,t).
di (u)). Gener-
Ut+ = iN + f(u, O)
To adapt this model to continuous setting, we let At + and obtain:
Uc+ii - du(t)
lim
A1-
At
dt
The model can then be evaluated through existing numerical integration techniques, proposed by
53
=(t))==(( = u((a) + (/h du(())
di (u(t), = u(to) + 1.a f(u(t), O)dt.
Numerical integration can then be treated as a black box, using numerical schemes beyond the forward Euler to achieve higher numerical precision. However, since black box integrators can take an arbitrary number of intermediate steps, backpropagating through individual steps would require too much memory since the individual steps must be saved. Chen et al. [5] addressed this problem by using adjoint backpropagation, which has a constant memory usage. For a given loss function on the terminal state of the hidden state L(u(ti)), )), the adjoint att) is governed by another ODE:
da(t) aa()) di
(f(())),t,) 1,0) Du
OC
a(ti)
Ou(l1)'
that dictates the gradient with respect to the parameters. The loss L(u(tz)) )) can then be calculated by 55 solving another ODE (the adjoint) rather than backpropagating through the calculations involved in the numerical integration.
57 However, the hidden state governed by an ODE imposes limitation on the expressiveness of the
58 mapping. For example, Dupont et al. [10] describes notable limitation of NODEs is in the inability response to such limitations, many works have tried to increase the expressiveness of the mapping. Dupont et al. [10 proposed to solve
59 to represent dynamical systems with intersecting trajectories.
61 the intersection trajectories problem by augmenting the vector space, lifting the points into additional dimensions: Zhu et al. [55) included time delay in the equation to represent dynamical systems of

---

63
greater complexity; Massaroli et al. [32 proposed to condition the vector field on the inputs, allowing the integration limits to be conditioned on the input: Massaroli et al. 32 and Norcliffe et [35 additionally proposed and proved second-order ODE system can efficiently solve the intersecting
64
65
66
trajectories problem.
87
Multiple works have attempted to expand NODE systems to other common differential equation formulations. Sun al. [4]] employed a dictionary method and expanded NODEs to PDE case, achieving high accuracies both in approximating PDEs and in classifying real-world image datasets. However, Sun et al.
88
69
70
44 suggested that the method is unstable when training with the adjoint method and therefore is unable to make use of the benefits that come with training with adjoint. Zhang et al. [53] proposed a normalizing flow approach based on the Monge-Amper equation. However, Zhang et al. [53] did not consider using adjoint- based training. Long et al. [30] 31. Raissi considered discovering underlying hidden PDEs from data and predict dynamics of complex systems. Kidger et al. [24] Morrill et al.
71
72
73
74
al.
37 Brunton et
75
33 34 used ideas from rough path theory and controlled differential equations to propose a NODE architecture as a continuous recurrent neural network framework. Multiple works have expanded to the stochastic differential equations
76
78
setting and developed efficient optimization methods for them [6 22 23 25
26 28 29 49]. Salvi
et al. considered stochastic PDEs for spatio-temporal dynamics prediction. Additionally, Chen data using NODEs, and Rubanova et al. (39, De Brouwer et al. [8] makes predictions on time series data using NODEs. Physical modeling is also popular application of NODEs, as control problems are often governed by latent differential equations that can be discovered with data driven methods 77 5T 54].
80 et al. 6 models
82
84
NODE systems have also been used for modeling the flow from a simple probability density to complicated one [.. Specifically, if ((t) I²' follows the ODE du(t)/dt = f(u(t)), where fu((t) E IR",, then its log likelihood from
85
86
[3] Appendix A] is given by:
i logg p(u(t)) = tr df
aL
du(()
The trace can be calculated efficiently with Hutchinson trace estimator 13] Subsequent work 88 uses invertible ResNet, optimal transport theory, among other techniques to further improve the [6 50 [53]. CNF is desirable for having no constraints 90 on the type of neural network used, unlike discrete normalizing flows, which often have constraints
87
89
performance of CNFs [ 2 I 18
20
91 on the structure of the latent features 9 36 38. CNFs also inspire development in other generative
modeling methods. For instance, a score-based generative model can be seen as a probability flow
93
modeled with an ODE 42 47
3 Method
95
We describe the proposed C-NODE method in this section by first providing a brief introduction to the method of characteristics (MoC) for solving PDEs with an illustrative example. We then discuss how we apply the MoC to C-NODE framework. We finally discuss the types of PDEs we can describe using this method.
96
98
3.1
Method of Characteristies
100
The MoC provides a procedure for transforming certain PDEs into ODEs along paths known as characteristics. In the most general sense, the method applies to general hyperbolic differential equations; however, for illustration purposes, we will consider canonical example using the inviscid
102
103
Burgers equation. A complete exposition on the topic can be found in [15 Chapter 9], but we
104
introduce some basic concepts here for completeness. Let u(I,t) IR x R + R satisfy the
105
following inviscid Burgers equation
Ou
du + u
0,
01
Or
105
where we dropped the dependence and L. for ease of notation. We are interested in the solution of u over some bounded domain c R x R,. Consider parametric forms for the spatial component
107
:(s): [0,2] + R and temporal components t(s) [0, T] - R. over the fictitious variable E [0,
3

---

109 Intuitively, this allows us to solve an equation on curves z, ł as functions of variable s which we 110 denote (z(s). t(s)) as the characteristic. Expanding, and writing as the total derivative, we get
d u(=(())((s)) ds
du dr dr ds
du dt Ōi ds
Recalling the original PDE in 2 and substituting the proper terms into 33 for dæ/ds ss u, dt/ds 1, du /ds = O, we then recover 2). Note that we now have a system of ODEs, which we can solve to obtain the characteristics x(s)
+ zog and ((s) 8 to functions of initial conditions Tu. tu. Finally, by solving over a grid of initial conditions f Jim1 E a2, we can obtain the solution of the PDE over Q. Putting it all together, we have new ODE that is written
(1) 10
d
u(<((s), = du
+ u du
= 0,
ds
Ot
dr
II where we can integrate over 8 through
((2(()), = [' u(z((()) t(s))ds
ds
/ ds
T d
u(us + zo. s)ds,
112 using the adjoint method with boundary conditions To. to- This contrasts the usual direct integration over the variable t that is done in NODE; we jointly couple the integration through the character- istics. An example of solving this equation over multiple initial conditions is given in Figure with the contrast standard NODE integration.
114
115
116
To provide some intuition for using MoC, we note that MoC most generally applies to hyperbolic PDEs. The transport equation is example of this family of PDEs, which roughly describes the propagation of physical quantities through time. Such equations are appropriate for deep learning tasks due to their ability transport data into different regions of the state space. For instance, in classification task, we consider the problem of transporting high-dimensional data points that are not linearly separable to spaces where they are linearly separable. Similarly, in generative modeling, transport base distribution to data distribution.
118
120
122
123
3.2
Neural Representation of Characteristics
In the proposed method, we learn the components involved in the MoC, namely the characteristics and the function coefficients. We now generalize the example given in 3.I which involved two variables, a k-dimensional system. Specifically, consider the following nonhomogeneous boundary value problem (BVP)
125
127
ou + Σk 0t - (1(11, ... The u) az; [n(x(0)) = Up
du
= c(z1 .... Tke u), on x. E IR* x (0) on X E R.
(4)
Here, u R* _d is multivariate map, a, : R + E and RR - E" be functions dependent on values of u and x's. This problem is well-defined and has a solution so long as
129
Σ 1 a, dz;
au is continuous [2]
130
131
MoC has historically been used in a scalar context, but generalization to the vector case is relatively straightforward. A proof of the generalization can be found in Appendix B.I We decompose the PDE ④ into the following system of ODEs
133
de: ds
a, .... Ik. u),
(5)
du
k du dx: c(I), Ik. Tk, u).
(6)
ds
Ox. ds
194 We represent this ODE system by parameterizing dx./ / ds and 0u/Ox; with neural networks. Conse- quently, du/ds ds is evolving according to (

---

138
Following this expansion, we arrive at
- u(x(0)) + ! ds
T du
(x, u) ds
=((x(0))+ + /. [Jxu] (x, u; 22) dx ds
(x, u: 2s)
137
where we remove u's dependency on x(s) and x's dependency on s for simplicity of notation, In Equation 7 the functions J and dx ds are learnable functions which are the outputs of deep neural networks with inputs u and parameters 22
138
140
3.3
Conditioning on data
Previous works primarily modeled the task of classifying set of data points with fixed differential equation, neglecting possible structural variations lying in the data. Here, we condition C-NODE on each data point, thereby solving a PDE with a different initial condition. Specifically, consider the term given by the integrand in (7). The neural network representing the characteristic dx /x/ds ds is
142
143
144
145
conditioned on the input data z R Define a feature extractor function g(-) R E" and we
148
have
dæ; (2 Tke u: g(z))
(8)
ds
By introducing g(z) in (8), the equation describing the characteristics changes depending the current data point. This leads to the classification task being modeled with a family rather than one single differential equation.
148
149
150
3.4
Training C-NODEs
151
After introducing the main components of C-NODEs, we can integrate them into a unified algorithm. To motivate this section, and to be consistent with part of the empirical evaluation, we will consider classification tasks with data and y, its class label. In the approach we pursue here, the image zj first passed through feature extractor function g(-; (1) R + R= with parameters 1). The output of is the feature
2j R Yj E Z+. For instance, 2j may be an image,
153
154
155
gzz θ1) that provides the boundary condition for the PDE on uti,, We integrate along
156
different characteristic curves indexed by E [o, I] with boundary condition u (x(0)) u and
157
compute the end values as given by 7 where we mentioned in Section 3.2
158
u (x(T) = ub) (6) + S/ Jxu( ['
x, u)); 02)
dx
ds x, u) up (1) O2 ds
(9)
Finally, u (xT)) is passed through another neural network, (x Qa) with input u (x) and parameters Oş whose output are the probabilities each class labels for image zj. The entire learning is now is reduced to finding optimal weights (91, O2 θx) which can be achieved
159
160
182
by minimizing the loss
N
L
Σ
L ((((n
163
where L(-) a loss function choice. In Algorithm we illustrate the implementation procedure with the forward Fuler method for simplicity for the framework but note any ODE solver can be used.
164
185
3.5
Combining MoC with Existing NODE Modifications
166
As mentioned in the Section 2 the proposed C-NODES method can be used as an extension existing NODE frameworks. In all NODE modifications, the underlying expression of J f(t, u; O)dt remains
167
the same. Modifying this expression to f Jæu(x, u; O)dx0dxxsss /ds(/xsxx (x, u; uo; O)ds results the proposed C-NODE architecture, with the size of x being hyperparameter.
188
5

---

Algorithm C-NODE algorithm using the forward Euler method
for each input data 2j do
extract image feature u(s 0) g(2;; 01) with feature extractor neural network.
procedure Integration along = 0 +
for each time step Sm. do
calculate dx
da (x, u: g 2j; 11): 22) and J ux O2).
calculate da Jxu calculate u(5++1) = ((m) + du end for end procedure
de (sm+1 - 8m).
classify u(s 1) with neural network Φ(u(x(s 1)), 0a).
end for
4 Properties of C-NODEs
171
C-NODE has a number of theoretical properties that contribute to its expressiveness. We provide some theoretical results these properties in the proceeding sections. We also define continuous normalizing flows (CNFs) with C-NODEs, extending the CNFs originally defined with NODEs.
173
174
4.1
Intersecting trajectories
175
As mentioned in 10 one limitation of NODE is that the mappings cannot represent intersecting dynamics. We prove by construction that the C-NODEs can represent some dynamical systems with intersecting trajectories the following proposition:
176
177
178
Proposition 4.1. The C-NODE can represent a dynamical system on u(s). du/ds /s g(s, u)
R+ x R + R, where when u(0) = 1, then u(1) u(0) + So G(s, u)ds = 0: and when u(0) = 0,
179
then u11) = (0) + 1or ((s u)ds 1.
180
181
Proof. See Appendix B.2
182
4.2
Density estimation with C-NODES
183
C-NODEs can also be used define a continuous density flow that models the density of variable over space subject to the variable satisfying a PDE. Similar to the change of log probability of NODEs, as in I we provide the following proposition for C-NODEs:
184
185
186
Proposition 4.2. Let u(s) be a finite continuous random variable with probability density function Ou and deg are uniformly Lipschitz Su da,, Assuming Sx; ds
187
p(((s)) and let u(s) satisfy ds
duls) Eiz öz; ds continuous u and continuous in s, then the evolution of the log probability of u follows:
188
@ alogp(u -tr
0
k
Ou da.
Os
Ou
Oz ds
Proof. See Appendix B.3
190
CNFs are continuous and invertible one-to-one mappings onto themselves, i.e., homeomorphisms. Zhang et al. (52) proved that vanilla NODEs are not universal estimators of homeomorphisms, and augmented neural ODEs (ANODEs) are universal estimators of homeomorphisms. We demonstrate
192
193
that C-NODEs are pointwise estimators of homeomorphisms, which we formalize in the following proposition:
194
195
Proposition 4.3. Given any homeomorphism h Y T. T
RP, initial condition up and time Ou di such that u(I, uo) = h(vo))
196
T > 0, there exists a flow u(s, uo) E IR" following du
8z de
Ou de
+ at da
Proof. See Appendix
B.4

---

20
20
20
2.25
2.23
2.00
1.5
20
20
225
225
2.00
2.0
2.00
2.0
1.75
1.5
1.5
20
SWN
MMS1
MAD5
(a) Adjoint training:
(b) Backprop through Euler training;
Figure 2: Red: NODE. Blue: C-NODE. Training dynamics of different datasets with adjoint in Fig. 2a and with Euler in Fig. 2b averaged over five runs. The first column is the training process of SVHN, the second column of CIFAR-IO, and the third column of MNIST. By incorporating the C-NODE method, we achieve more stable training process in both CIFAR- 10 and SVHN, while achieving higher accuracy. Full-sized figure in supplementary materials.
Experiments
198
189
We present experiments on image classification tasks benchmark datasets, image generation tasks on benchmark datasets, PDE modeling, and time series prediction.
200
5.1
Classification Experiments with Image Datasets
202
We first conduct experiments for classification tasks on nigh-dimensional image datasets, including MNIST, CIFAR-10, and SVHN. We provide results for C-NODE and also combine the framework with existing methods, including ANODEs [T), Input Layer NODEs (IL-NODEs) (32), and 2nd-
203
204
205
Order NODES 32.. For all classification experiments, we set the encoder of input images for
206
conditioning to be identity, i.e., g(2) = making the input into C-NODE the original image. This
207
way, we focus exclusively on the performance of C-NODE.
208
The results for the experiments with the adjoint method are reported in Table and in Figure 2a We
209
investigate the performances of the models classification accuracy and the number of function evaluations (NFE) taken in the adaptive numerical integration. NFE is an indicator of the model's
210
computational complexity, and also be interpreted the network depth for the continuous NODE
system [5) Using a similar number of parameters, combining C-NODEs with different models consistently results in higher accuracies and mostly uses smaller numbers of NFEs, indicating better parameter efficiency. An ablation study on C-NODEs' and NODEs' parameters can be found in Appendix C.2 The performance improvements can be observed, especially on CIFAR-10 10 and SVHN, where it seems the dynamics to be learned are too complex for ODE systems, requiring sophisticated model and large number of NFEs. It appears that solving PDE system along a multidimensional characteristic is beneficial for training more expressive functions with less complex dynamics, as can be seen in Figures 2a 2b
212
213
214
215
216
217
219
220
We also report training results using traditional backpropagation through the forward Fuler solver Figure 2b The experiments are performed using the same network architectures as the previous experiments using the adjoint method. It appears that C-NODEs converge significantly faster than the NODES (usually in one epoch) and generally have more stable training process with smaller variance. In experiments with MNIST, C-NODEs converge in only one epoch, while NODEs converge
221
222
223
224
225
roughly 15 epochs. This provides additional empirical evidence on the benefits of training using the characteristics. As shown in Figures 2a 2b] compared to training with the adjoint method, training with the forward Euler solver results in less variance, indicating a more stable training process. At the same time, training with the adjoint method results in more accurate models, as the adjoint method uses constant amount memory, and can employ more accurate adaptive ODE solvers.
226
227
229
230
5.2
Continuous normalizing flow with C-NODEs
231
compare the performance of CNFs defined with NODEs to with C-NODEs on MNIST, SVHN, and CI. We use Hutchinson trace estimator to calculate the trace and multi-scale
7

---

Dataset
Method
Accuracy 1
NFE
Param.[K]
NODE
75.28 + 0.836%
131
115.444
C-NODE
82.19 0.478%
124
113.851
ANODE
89.8 0.952%
167
112.234
ANODE+C-NODE
92.23 = 0.176%
146
112.276
SVHN
2nd-Ord
88.22 # 1.11%
161
112.801
2nd-Ord+C-NODE
92.37 + 0.118%
135
112.843
IL-NODE
89.69 + 0.369%
195
113,368
IL-NODE+C-NODE
93.31 + 0.088%
95
113.752
NODE
56.30 + 0.742%
152
115.444
C-NODE
64.28 = 0. 243%
151
113.851
ANODE
70.99 + 0.483%
177
112.234
ANODE+C-NODE
71.36 = 0. 220%
224
112.276
CIFAR-10
2nd-Ord
70.84 + 0.360%
189
112.801
2nd-Ord+C-NODE
73.68 + 0.153%
131
112.843
IL-NODE
72.55 1 0.238%
134
113.368
IL-NODE+C-NODE
73.78 + 0.154%
85
113.752
NODE
96.90 1: 0.15-1%
72
85,468
C-NODE
97.56 = 0.431%
72
83.041
ANODE
99.12 + 0.021%
68
89.408
ANODE4C-NODE
99.20 1 0.002%
60
88.321
MNIST
2nd-Ord
99.35 + 0.002%
52
89.552
Ond-Ord+C-NODE
99.38 0.037%
61
88.465
IL-NODE
99.33 + 0.039%
53
89.597
IL-NODE+C-NODE
99.33 0.001%
60
88.51
Table 1: Mean test results over runs of different NODE models over SVHN, CIFAR-IO, and MNIST, Accuracy and NFE at convergence are reported. Applying C-NODE always increases models' accuracy and usually reduces models' NFE well as the standard error.
233
convolutional architectures as done in 9 13 Differential equations are solved using the Runge-
234
Kutta method of order 5 of the Dormand Coimaand-rinnee-ee Shammp-e solver and trained with the adjoint method. Although the Fuler forward method is faster, experimental results show that its fixed step size often
236
leads to negative Bits/Dim, indicating the importance of adaptive solvers. As shown in table
figure 3 using similar number of parameters, experimental results show that CNFs defined with
238
C-NODEs perform better than CNFs defined with NODEs in terms of Bits/Dim, as well as having lower variance, and using lower NFE on all of MNIST, CIFAR-10, and SVHN.
5.3
PDE modeling with C-NODEs
We consider synthetic regression example for hyperbolic PDE with known solution. Since NODEs assume that the latent state is only dependent on a scalar (namely time), they cannot model dependencies that vary over multiple spatial variables required by most PDEs. We quantify the differences in the representation capabilities by examining how well each method can represent linear hyperbolic PDE. We also modify the assumptions used the classification and density estimation experiments where the boundary conditions were constant as in We approximate the
242
243
244
245
246
following BVP:
Du De 2t
8u +
(10)
u(
0) = 21,
1 15x52.
'Thi is based on the code that the authors of 13
provided in httpp:://gghhib..co/tttthhhhottp

---

MNIST
CIFAR-10
SVHN
Model
B/D
Param.
NFE
B/D
Param.
NFE
B/D
Param.
NFE
Real NVP 9
1.05
N/A
3.49
N/A
Glow 27
1.06
N/A
3.35
44.0M
RQ-NSF
3.38
11.8M
Res. Flow 4
0.97
16.6M
3.28
25.2M
CP-Flow
21
1.02
2.9M
3.40
1.9M
NODE
1.00
336.1K
1350
3.49
410.1K
1847
2.15
410.1K
1844
C-NODE
0.95
338.0K
1323
3.44
406.0K
1538
2.12
406.0K
1352
Table 2: Experimental results generation tasks, with NODE, C-NODE, and other models. B/D indicates Bits/dim. Using similar amount of parameters, C-NODE outperforms NODE on all three datasets, and have significantly lower NFE when training for CIFAR-I0 and SVHN.
248 10
2exp(()+I We generate a training dataset by randomly sampling 200 points (I, 1), x E [1, 2]. E [0, 1]. as well as values u(2, () at those points.
has an analytical solution given by u(x, t) 2z exp(t)
249
We test C-NODE and NODE 200 points randomly sampled (x, E [1, 2] [0. 1]. For this experiment, C-NODE uses 809 parameters while NODE uses 185 parameters. C-NODE deviates
251
252
8.05% from the test dataset, while NODE deviates 30.52%. Further experimental details can be found in Appendix A.3
253
254
5.4
Time series prediction with C-NODEs
2.2
C-NODE
NODE
Finally, test C-NODEs and NODEs the time series prediction problem using the MuloCo dataset 256
1.8-
45
We follow the experimental settings in (39]
258
where we define an autoregressive model with the en-
1.6
259
coder being an ODE-RNN model and the decoder be-
1.4
ing a latent ODE As shown in Figure C-NODEs achieve lower testing mean squared errors (MSEs).
280
1.2
252
After 100 training epochs, C-NODEs achieve 10.14%
263
lower testing MSEs than NODEs.
0
10
20
Epoch
30
40
50
Figure 3: Red: NODE. Blue: C-NODE. Training dynamics of CNFs on MNIST dataset with adjoint method. We present Bits/dim of the first 50 training epochs.
Discussion
264
We describe an approach for extending NODEs to the case of PDEs by solving series ODEs along the characteristics of PDE. The approach applies
286
268
to any box ODE solver and can be combined with existing NODE frameworks. We classification tasks while also demonstrating its success in improving convergence using Euler forward method without the adjoint method. Additionally, C- density estimation tasks, while being more efficient with the number of parameters and using lower NFEs. C-NODE's efficiency over physical modeling and time series prediction is also highlighted with additional experiments.
empirically showcase its efficacy
270
NODE empirically achieves better performances
272
274 Limitations
There are several limitations to the proposed method. The MoC only applics to hyperbolic PDEs, and we only consider first-order semi-linear PDEs in this paper. This may be limitation since this a specific class of PDEs that does not model all data. We also did not enforce any particular structure to prevent characteristics from intersecting, which may result in shock waves and rarefactions. However, we believe that this is unlikely to happen due to the high dimensionality of the ambient space. We additionally note that, compared ANODE, C-NODE's training is not as stable. This can be improved by coupling C-NODEs with ANODES or other methods.
276
278
279
280
TTii is based on the code of Rubanova et al. 39 provide in
latent_ode

---

References
281
282
[1] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Styleflow: Atribute-conditione exploration of stylegan-generatee images using conditional continuous normalizing flows. ACM Transactions on Graphics (TOG), 10(3):1221, 2021.
284
[2] Jens Behrmann, David Kristjanson Duvenaud, and Hen-Henriik. Jacobsen. Invertible residual networks. In ICML, 2019.
286
[3] Steven L Brunton, Joshua L. Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, (3(5))33332 -9322, 2016.
288
290
[4] Ricky T. Q. Chen, Jens Behrmann, David Kristjanson Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. arXiv preprint arXiv: 02735, 2019.
292
[5] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. arXiv preprint
2019.
294
[6] Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. arXiv preprint arXiv:2011.04588 2020.
296
[7]
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv preprint arXiv: 04600000000000..00 2020.
298
[8] Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-observee time series. Advances in neural information processing systems, 32, 2019.
299
300
[9] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 1605.08880 2017.
302
303
[10] Emilien Dupont, Arnaud Doucet, and Yee Whye Tch. Augmented neural ODEs. arXiv preprint arXiv: 0811, 2019.
304
305
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. Advances in neural information processing systems, 32, 2019.
308
307
[12] Lawrence Evans. Partial Differential Equations. American Mathematical Society, 2010.
308
[13] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Kristjanson Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv: 1810.01.7. 2019.
309
311
[14] Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
313
[15] David F Griffiths, John W Dold, and David Silvester. Essential partial differential equations. Springer, 2015.
315
[16] Batuhan Güler, Alexis Laignelet, and Panos Parpas. Towards robust and stable deep learning algorithms for forward backward stochastic differential equations. arXiv preprint M.ii:19100.1123:: 2019.
317
[17] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint 1rXiv:1512.03385, 2015.
319
[18] Emiel Hoogeboom, Rianne Van Den Berg, and Max Welling, Emerging convolutions for generative normalizing flows. In International Conference on Machine Learning, pages 2771- 2780. PMLR, 2019.
321
323 [19] Ralph Howard. The gronwall inequality, 1998. URL. http://people.math.o edu/
Soward//lotess/grnnaaa pdf
10

---

325
[20] Chin- Huang, Laurent Dinh, and Aaron C. Courville. Augmented normalizing flows:
326
Bridging the gap between generative flows and latent variable models. ArXiv, abs/2002.07101, 2020.
327
[21] Chin-Wei Huang, Ricky T. Q. Chen, Christos Tsirigotis, and Aaron C. Courville. Convex poten- tial flows: Universal probability distributions with optimal transport and convex optimization. arXiv preprini vrXiv:20/2.05944 2021.
329
[22] Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
332
333
[23] Junteng Jia and Austin R. Benson. Neural jump stochastic differential equations. arXiv preprint arXiv:1955.10403, 2020.
[24] Patrick Kidger, James Morrill, James Foster, and Temy Lyons. Neural controlled differential equations for irregular time series. arXiv preprint arXiv:2005, 2000020005.08226, 2020.
336
337
[25] Patrick Kidger, James Foster, Xuechen Li, and Terry Lyons. Efficient and accurate gradients for neural sdes. arXiv preprint arXiv:2105. 2021.
338
[26]
Patrick Kidger, James Foster, Xuechen Li, and Terry J Lyons. Neural sdes as infnite-dimensional gans. In International Conference on Machine Learning, pages 5453-5463, PMLR, 2021.
340
341
[27] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. Advances in neural information processing systems, 31, 2018.
342
343
[28] Xuechen Li, fing-Kam Leonard Wong. Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. arXiv preprint arXiv:2001.0133 2020.
344
345
[29] Xuanging Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. Neural sde: Stabilizing neural ode networks with stochastic noise. arXiv preprint arXiv: 1906.003550 2019.
346
347
[30] Zichao Long. Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data, In International Conference on Machine Learning, pages 3208-3216. PMLR, 2018.
[31 Zichao Long. Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with numeric- symbolic hybrid deep network. Journal of Computational Physics, 399:108925, 2019.
350
351
[32] Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dis- secting neural odes. arXiv preprint arXiv:2002.08071, 2021.
[33] James Morrill, Patrick Kidger, Lingyi Yang, and Terry Lyons. Neural controlled differential equations for online prediction tasks, 2021.
354
355
[34] James Morrill, Cristopher Salvi, Patrick Kidger, James Foster, and Terry Lyons. Neural rough differential equations for long time series. arXiv preprint arXiv:2009.08290 2021.
356
357
[35] Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Liò. On second order behaviour in augmented neural odes. arXiv preprint arXiv:2006.07220,, 2020.
358
359
[36] George Papamakarios, lain Murray, and Theo Pavlakou. Masked autoregressive flow for density estimation. arXiv preprint arXiv:1705.07057 2017.
380
381
[37] M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019. ISSN 0021-9991.
382
383
364
doi: cctaacce/arttdddl//////1/1111111
10..0.....4 10.045. URL htllhtts::///// www. sciencedirect. com/
365
366
[38]
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows.
367
In ICML, 2015.
[39]
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for iregularly-sampleed time series. Advances in neural information processing systems, 32, 2019.
369
11

---

[40]
Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, 62(3):352-36 2020.
372
373
[41] Cristopher Salvi, Maud Lemercier, and Andris Gerasimovics. Neural stochastic partial differ- ential equations: Resolution-invariant learning of continuous spatiotemporal dynamics, arXiv preprint arXiv:2110. 2022.
375
376
[42] Yang Song, Jascha Sihh--Dccstten, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and
Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011 1. 133466, 2021.
378
379
[43]
Robert Strichartz. The way of analysis. Jones and bartlett mathematics, 2000.
380
[44]
Yifan Sun, Linan Zhang, and Hayden Schaeffer. Neupde: Neural network based ordinary and par- tial differential equations for modeling time-dependent data. arXiv preprint arXiv: 1908.02 03/90, 2019.
381
[45] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, Tumothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. ArXiv, htss/800.0006 2018.
384
386
[46] Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. arXiv preprint arXiv: 1 13789, 2019.
387
388
[47] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Xixiv preprint 100.05931, 2021.
389
390
[88] E. Weinan. proposal on machine learning via dynamical systems.
Communications in
Mathematics and Statistics, 5, 2017.
[99] Winnie Xu, Ricky TQ Chen, Xuechen Li, and David Duvenaud. Infinitely deep bayesian neural networks with stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pages 721-738. PMLR, 2022.
393
385
[50] Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. Ode2vae: Deep generative second order odes with bayesian neural networks. Advances in Neural Information Processing Systems,
396
32, 2019.
398
[11] Cagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. Continuous-time model-based reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 Proceedings of Ma- chine Learning Research, pages 12009-12018. PMLR, 18-24 Jul 2021. URL https //procediins..mrrprree www.prressvvieeee arees//v13/yylddi2laa..
399
400
402
403
[52] Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural odes and invertible residual networks. In ICML, 2020.
[53] Linfeng Zhang, Weinan E, and Lei Wang. Monge-ampère flow for generative modeling. arXiv preprint 1809 10188, 2018
406
407
[54] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. arXiv preprint (tti::1909.110777 2019.
408
409
[55] Qunxi Zhu, Yao Guo, and Wei Lin. Neural delay differential equations. arXiv preprint a002v:2102. 10801, 2021.
12

---

Checklist
412
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
414
415
(b) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
417
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
418
(d) Did you describe the limitations of your work? [Yes]
419
If you are including theoretical results...
420
(a) Did you state the full set of assumptions all theoretical results? [Yes]
421
(b) Did you include complete proofs of all theoretical results? [Yes]
If you ran experiments...
423
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? (ess]
424
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes]
426
427
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
429
(d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
430
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
432
(a) If your work uses existing assets, did you cite the creators? [Yes]
439
(b) Did you mention the license of the assets? [No] Assests used follow the MIT license, granting the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software.
435
436
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
437
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
439
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
13

---

A Experimental Details
442
A.1
Experimental details of classification tasks
We report the average performance over five independent training processes, and the models are trained for 100 epochs for all three datasets.
444
445
The input for 2nd-Ord, NODE, and C-NODE are the original images. In the IL-NODE, we transform the input to a latent space before the integration by the integral; that is, raise the [ dimensional input image into the I (cctp))xxx dimensional latent feature space Rl We decode the result after performing the continuous transformations along characteristics curves, back to the Rexhxx dimensional object space. Combining this with the C-NODE can be seen solving PDE on the latest features of the images rather than on the images directly. We solve first-order PDEs with three variables in CIFAR-10 10 and SVHN and solve first-order PDEs with two variables in MNIST. The number of parameters of the models is similar by adjusting the number of features used in the networks. We use similar training hyperparameters [32].
448
448
450
452
453
454
Unlike ODEs, we take derivatives with respect to different variables in PDEs. For a PDE with variables, this results in the constraint of the balance equations
8u Fu
i, (1,2, (1
drzzy Ōryz;
This can be satisfied by defining the k-th derivative with neural network, and integrate k 1 times to get the first order derivatives. Another way of satisfying the balance equation is to drop the dependency on the variables, ie.. Vi E 21.2....
457
458
du
f(
459
When we drop the dependency, all higher order derivatives are zero, and the balance equations are satisfied.
460
461
All experiments were performed on NVIDIA RTX 3090 GPUs on a cloud cluster.
482
A.2
Experimental details of continuous normalizing flows
We report the average performance over four independent training processes. As shown in Figure 4 compared to NODE, using a C-NODE structure improves the stability of training, as well as having a better performance. Specifically, the standard errors for C-NODEs MNIST, SVHN, and CIFAR-10 are 0.37%, 0.51 %, and 0.24% respectively, and for NODEs the standard errors on MNIST, SVHN, and CIFAR-10 are 1.07%, 0.32%, and 0.22% respectively.
464
466
468
The experiments are developed using code adapted from the code that the authors of 13 provided in jord
470
All experiments were performed NVIDIA RTX 3090 GPUs cloud cluster.
A.3
Experimental details of PDE modeling
472
We want to solve the initial value problem
Zu
0)
2x,
15052,
473
where the exact solution u(x,,)) ()
+1J Our dataset's input are 200 randomly sampled points
2xe
474 (z. t). E [1,2], t E [0, 1]. and the dataset's 's outputare the exact solutions at those points.
du 475 For the C-NODE architecture, we define four networks: NN t) for ag NN2(2, t) for du (t) for the characteristic path (æ(s), t(s)), (N) for the initial condition. The result
477 calculated four steps:
This is based the code of Massaroli et al. 32 provide in httt::://ggthab...o///////LLLLLt
14

---

Bits/dim
2
C-NODE
NODE
1
Epoch
Bits/dim
3.0
2.5
Epoch
6
Bits/dim
4
0
20
40
60
80
Epoch
Figure 4: The training process averaged over 4 runs of C-NODE and NODE. The first row are the results on MNIST, the second row are the results SVHN, the third row the results on CIFAR-IO.
1. Integrate Au = fo du(=(s)) ds = So Bu dt [1 as before.
Ou de ds = NN₂ NN + NN1 Or de
478
ds
28
479
2. Given x, solve equation NN + for iteratively, with 4441
481
x - NN * t. to initialized to be x.
482
3. Calculate initial value (=)(0), t(0)) (N).)))
483
4. u(æ, t) = Au + u(z(0),
484
For the NODE architecture, we define one network: N (æ, () for 8u øt
The result calculated as
u(x t) = fo du
at dt So dt.
485
486
All experiments were performed on NVIDIA RTX 3080 GPUs on a local machine.
487
A.4
Experimental results and details of time series predictions
488
A.4.1
Experimental details time series predictions on MuJoCo dataset
489
Yul hta.b//oo//llaaauaaarraaaa _atent_ode] NODE's training follows the original setup, with the dimension of the recognition model being 30, the number of units per layer in each GRU update networks being 100, the number of units per layer in ODE function being 300, the number of layers in ODE function in generative and recognition ODE both being 3.
We follow the experimental setup as described in htttp:://itthu:
491
493
494
We use C-NODE with dimensionality of 128. The number of units per layer in the network describing dx/ ds 12. For network describing 0u/Oa, the dimension the recognition model
495
is 30, the number of units per layer in each of GRU update networks is 100, the number of units per layer in the ODE function is 100, the number of layers in ODE function in generative and recognition
437
ODE is 1.
15

---

C-NODE NODE
0.08
0.06
0.04
60 80 100
Figure 5: Red: NODE. Blue: C-NODE. Training dynamics of ODE- -NNN on the MuJoCo dataset with the "Hopper" model from the Deepmind Control Suit [45]. We present testing mean squared error (MSE) training epochs 10 to 100. The C-NODE method achieves lower testing MSE while having lower variance.
499
A.4.2 Experiment results of time series predictions on synthetic dataset
We test C-NODEs, ANODEs, and NODEs synthetic time series prediction problem. We define
501
eexp(t)+1* and we sample u(x,, t) + 0.1e,, where E, NN00, 1) over I E [1, 2). / E [0, 1] to generate the training dataset. We test the performance on / E [n,, + I] with n (0, 1, ,....; 5). To make the problem more challenging, x values are omitted, and only
function by u(x, t) 2x exp((t)
502
503
504
values are provided during both training and testing. As shown in Table 3 C-NODE produces more
505
profound improvements over NODEs as time increases.
Time
[0.1]
[1,2]
[2.3]
[3,4]
[4,5]
[5,6]
NODE
0.0322
0.1764
0.4681
0.8093
1.1911
1.6202
ANODE
0,0428
0,0629
0. 0.1248
0.2778
0.5360
0.9252
C-NODE
0.0270
0.0365
0.0582
0.1474
0.3300
0.6054
Table 3: Time series prediction results for NODE, ANODE, and C-NODE at different time intervals. Errors are testing mean squared errors. Across all time intervals, C-NODE outperforms NODE and ANODE.
506 We also test C-NODEs, NODEs, and ANODEs on time series prediction with different levels of noise. Specifically, using the same function as above, we form training and testing dataset with (0 m), E {0, .J.. We test the performance on the time period E [0, 1].
507
Noise Level
0
2
3
4
5
NODE
0.0326
0.1784
0.7886
1.9685
3.7530
6.553
ANODE
0.04
0.1984
0.6035
1.0574
1.4850
2.0593
C-NODE
0.0267
0.1011
0.3294
0.7148
1.2856
2.0834
Table Time series prediction results for NODE, ANODE, and C-NODE at different noise levels. Errors are testing mean squared errors.
A.4.3
Experimental details of time series predictions synthetic dataset
2- 2I different time with [1, 2], and æ being accessible 511 the network. We also provide the network with the value of u(1,
510 We want predict u(x, t)
512 We use a dimensional C-NODE network. The result is calculated with
u(x,t) = u(1,0) + / Σ
Ou dz; ds.
dz, ds
16

---

518
NODE is calculated with
u(x,1) 1) = u(1,0) + L 8u
dt. at
514
In our experiments, C-NODEs use 1221 parameters, ANODES use 1270 parameters, NODEs use 1290 parameters.
515
516 All experiments were performed on NVIDIA RTX 3080 GPUs on local machine.
517
B
Approximation Capabilities of C-NODE
Proposition B.1 (Method of Characteristics for Vector Valued PDEs). Let u(x1, ...... rR) R + R" be the solution of a first order semilinear PDE bounded domain Q C R2 of the form
du c(21,..... Tk. u) on (II...., = X Ω.
Σ a.(a......... Fk: u)
(11)
Oz.
520 Additionally, let a (ai...., a²)²² functions. Define a system of ODEs
ggtt + RR Rktn + Re be Lipschitz continuous
dx (8) a(x(s), Us)))
de (s) c(x(s), U(s))
x(0) Xo; Xp 02
u(xo) = uo
U(0) = u1
522
where Xo and un define the initial condition, On is the boundary of the domain Q. Given initial
523
conditions Xy, Ug the solution of this system of ODEs U(s) [a, bj + IR is equal to the solution of
524
the PDE Equation [TT along the characteristic curve defined by x(s), ie., u(x(s)) U(s). The
525
union of solutions
(s) for all E On is equal to the solution of the original PDE in Equation
526
IT for all
Lemma B.2 (Gronwall's Lemma [1).. Let U C R be open set. Let f U x [0 ER be continuous function and let h1, h2 : [0, + U satisfy the initial value problems:
dhi [(,, (1). h2( X1:
dt
dh2(1) f(hz (t), h2(0) X2.
dt
If there exists non-negative constant C such that for all & E [0,7] [|f(hz(t),t) f(b t)l < C2 (t)ll, the Euclidean norm. Then, for all t [0, T].
where ||
[12(() h1 (1)ll < c| x1|l-
527
B.1 Proof' of Proposition B.1
This proof largely based on the proof for the univarate case provided a We extend for the vector
528
529 valued case.
530 Proof. For PDE with input, and n-dimensional output, have a, R R, digg and c kg + R2" In proposition B.1 we look at PDEs in the following form
E R"
Tu dx;
Σ az FR.
c(x) c( Zz. u).
(12)
https://en.wikkieddaa.
Case
17

---

532
Defining and substituting x
a = (ai....,, ax) and Jacobian J(u(x))
0u
8u
E Ru*k into Equation result in
533
00g
[u(x))aax u) c(x, u).
(13)
From proposition B.1 the characteristic curves are given by
dæ; a; Tki u)
ds
and the ODE system is given by
dx (s) = U(s),,
U(s),,
(14)
ds
535
dU (s) c(((s). U()).
(15)
ds
Define the difference between the solution to (55) and the PDE in as
(ss) ||u(x( s)) Us)] (u(x(s)) U(s)) (u(x(s)) U(s)),
Differentiating (ss) with respect and plugging in 14 we get
A'(s) == dA(s) 2(u(((s)) U(s) (J(u)x U'(s))
ds
2[[(x(()) - U(+) (J(u)a(x(ss, U(+) c(x(s), U(s))
(16)
13 gives us E a: a=( Fk, 1) au 538 and rearrange terms, we have
c(,,...., Rk, 0. Plugging this equality into 16
(s) = 2[u(x(+)) U(+] (F(m)a(x(s) U(+) c(x(s),xxx))))))))
[[)))((())) u(s)) u(x))=
539
Combining terms, we have
A' = 2(u- - U) ((((((( - - (J)))a((()a()))) c(u)))
2(u U) (J(n) (a(U) a(u) + [c(U) c(u)))
Applying triangle inequality, we have
|| < 2 U a(u)l || + ||c(U) - c(u)
By the assumption in proposition B.1 and are Lipschitz continuous. By Lipschitz continuity, we have |aa(U) a(u))ll < Al|u U| and |c<UU) - c(u))|| < Bllu u U|| for some constants A and
542
543
B in R4, Also, for compact set j0, so). 8a < co, since both u and Jacobian J are continuous mapping,
544
J(u) is also compact. Since a subspace of TR. is compact if and only is closed and bounded, J(u)
545
is bounded [43] Thus, [u()] s M for some constant M in R+. Define C = 2(AM + B), we
546
have
(s)ll 2(AM U|l Bllu U Ull
=Cli U
CC)A())|I-
From proposition B. we have u(x(0)) = U(0). As proved above, we have du(x(s))
d((s))
= ( < C1)A()))I
ds
ds
where C' 0c. Thus, by lemma
oc.
B.2 we have
|A(s)) < Ct [[(() e (u((())))) U(O) 0.
This further implies that U(s) u(x(s)) so long and are Lipschitz continuous.
18

---

0.8
0.4
1.0
3.8
0.0
U
0.6
0.2
0.8
1.0
0.0
Figure 6: Comparison of C-NODEs and NODEs. C-NODEs (solid blue) learn family of integration paths conditioned on the input value, avoiding intersecting dynamics. NODEs (dashed red) integrate along ID line that is not conditioned the input value and can not represent functions requiring intersecting dynamics.
548
B.2
Proof of Proposition 4.1
Proof.
Suppose have C-NODE given by
du du dr
du dt Ōt ds
ds dx ds
549
Write out specific functions for these terms to match the desired properties of the function. Define initial condition u(0, 0) Ug. By setting
550
dr (s, uo 0) 1,
dt (s, up 0) ugg
ds
du (w(x; 1),0)
Ou (u(a, (), -2
Ø1
551
have the ODE and solution,
du 2ug
ds
u(s; uo) (1-2w0)
0
1 1 1 [
0
u s:
-1 s.
552
To be specific, we can represent this system with the following family PDEs:
Ou
Ou
uo
2uo-
dr
553
We can solve this system obtain a function that has intersecting trajectories. The solution is visualized in Figure which shows that C-NODE can be used to learn and represent this function
554
555
G. should be noted that this not the only possible solution function G, when 0t/Os we fall back to NODE system with the dynamical system conditioned on the input data. In this conditioned setting, we can then represent & by stopping the dynamics different times as 32
557
558
559
B.3
Proof of Proposition 4.2
The proof uses the change of variables formula for particle that depends vector rather than scalar and it follows directly from the proof given [5) Appendix A]. provide the full proof for completeness.
561
19

---

583 Proof. Assume Ex=
Du de: is Lipschitz continuous in u and continuous in t, so every initial value 8e:
problem has unique solution [2]. Also assume u(s) is bounded.
565 Want
Op(u(s)) 0s
8
du dx:
lr
Iu
0my ds
Define T. u(s + c). log p(u)) log det
The discrete change of variables states that U1 f(u)) log logp(u1)
āʃ 38
568
Take the limit of the time difference between and u1, by definition of derivatives,
i log p(u(s)) Ot
logp(u(s + +)) - log p(oss((s))
lim c-
log logp(u(s)) log det du (u(tt))) log p(u(s))
lim ++0
log det P (u())))
lim
Tu
c->
8.
log det Du
8 (u(s))
lim
8 e De
e-+0
θ log det ((()ss))
θ ou
lim
log
det
Te((ss)]
lim
c-1
de
au
c de
= lim
det Te(u(s))
c-0
det
du
á (u((s)) de
Du
lim_-0+
det 8u T((ss)))
3e
lim det D4 (u((s))
det 0
c-+ de Ou ((((s))
lim
The Jacobi's formula states that if is differentiable map from the real numbers matrices, then di det A(t) = tr(adj(A(t)) dA(O), where adj is the adjugate. Thus, have
570
① lim tr adj
de du I.(((s))
θ a
Øt
--
Fu
( 0
T((u(t)))
T.(U(S)) ]
tr
lim adj
lim -+tu
Ou
de Du
tr
adj
Ou u(t)
lim
de Du (.(sss))
+
tr
lim
T. Tu((ss))
c-+0)
de Ou
20

---

Substituting T. with its Taylor series expansion and taking the limit, we have log p(u(t))
0
du
O((2) + +0())) +..))
tr
lim
u
+
0t
c-00
de Ou
ds
Ø 0
Eu dx;
+ + 0 0(0)) ))
tr
lim
+
c-0+ dc Ou
0z; ds
0
0
+0(3)+0(0) :..) + + + 0(e)) 00 + +...)
du dx;
lim
1
<-0+ Ōe
Uu
Oxg ds
0
Du dx;
lim
+-+
2u
0r, ds
k
Ou dx;
tr
Ou
Fzg ds
572
B.4 Proof of Proposition 4.3
4.3
574
Proof. To prove proposition 4.3 need show that for any homeomorphism h(.), there exists u(s, uo) E R" following C-NODE system such that u(s = T, uo) h(uo))
578
Without loss of generality, say =
577
Define C-NODE system
dn
Bu Si de
ds
σt de
de
(s, (g) = 1, (u(x, 1)) (uu)):
da du 07
de (s, uo) 110
2u 07
(u(x, t)) -1.
578 Then, du ds
= h(uo)= - up- At s = 1, have
du ('' ds
u(s 1, (o) -u(s 0, uo)
ds
Du dx
du dt
uo +
ds
Ox ds
ol ds
uo + h(uo)
1 (-1) unds
uo + h((w)) - uo
h(wo))-
579
The inverse map will be defined by integration backwards. Specifically, have
u(s 0, uo) w(s 1,,0) {,' du dd
ds
Ou dr
Su dt
h(wu))
ds
Or ds
Øt ds
M(uo)
h(uo) 1 + (-1) ugds
Iu(u))
h(uo) + uo
21

---

580
Thus, for any homeomorphism fa(), there exists a C-NODE system, such that forward integration for time is equivalent as applying ht(), and backward integration for time -1 equivalent to 581
applying h- (-).
C Ablation Study
584
Ablation study on dimension of C-NODE
We perform an ablation study the impact of the number of dimensions of the C-NODE we implement. This study allows us to evaluate the relationship between the model performance and the model's limit of mathematical approximating power. Empirical results show that number of dimensions used in the C-NODE model, the C-NODE's performance first improves and then declines, due to overfitting. We have found out that information criteria like AIC and BIC can be successfully applied for dimension selection in this scenario.
586
increase the
588
590
In previous experiments, we represent 0u/Oz; with separate and independent neural networks u, (u, 0). Here, we represent all k functions as vector-valued function [au/Oz), 0 1 We approximate this vector-valued function with neural network c(u, 0). The model is trained using Euler solver to have better training stability when the neural network has a large number of parameters. Experiment details for the ablation study is as shown in Figures 7 g 9
592 C,
594
596
C.2
Ablation study number of parameters
598
We show C-NODE's parameter efficiency over NODE with an ablation study on the image classi- fication task the CIFAR-10 dataset. Specifically, under similar training setup, we experiment with C-NODE with 95071, 55855, and 17379 parameters and experiment with NODE with 96044, 56828, and 17444 parameters. As shown in Figure T0 although C-NODE has more variance in its performance, it outperforms NODE along the whole training process in all three cases.
589
600
602
22

---

90
Accuracy
80
70
1D
4D
16D
64D
2D
8D
32D
128D
60
1D
4D
16D
64D
2D
8D
32D
128D
Test Loss
1.8
1.6
2.0
1D
4D
16D
64D
Train Loss
2D
8D
32D
128D
1.8
1.6
0
20
40
60
80
100
Figure 7: The training process averaged over 4 runs of C-NODE with 1, 2, 4. 8, 16, 32, 64, 128, 256. 512, and 1024 dimensions on the MNIST dataset. The first row the accuracy of prediction, the second row is the testing error, and the third row is the training error.
23

---

85
Accuracy
80
75
1D
4D
16D
64D
2D
8D
32D
128D
1.75
1D
4D
16D
64D
Test Loss
1.70
2D
8D
32D
128D
1.65
1.60
1.7
1D
4D
16D
64D
Train Loss
2D
8D
32D
128D
1.6
1.5
20
40
60
80
100
Figure 8: The training process averaged 4 runs of C-NODE with 1, 4, 16, 32, 64, and 128 dimensions on the SVHN dataset. The first row is the accuracy of prediction, the second row the testing error, and the third row is the training error.
24

---

60
Accuracy
50
40
1D
4D
16D
64D
2D
8D
32D
128D
30
1D
4D
16D
64D
2.1
2D
8D
32D
-128D
Test Loss
2.0
1.9
2.2
1D
4D
16D
64D
Train Loss
2D
8D
32D
128D
2.0
1.8
1.6
20
40
60
80
100
Figure 9: The training process averaged over 4 runs of C-NODE with 1, 2, 4, 8, 16, 32, 64, and 128 dimensions on the CIFAR- 10 dataset. The first row the accuracy of prediction, the second row the testing error, and the third row is the training error.
25

---

60
60
Accuracy
40
40
20
20
20
20
40
20
40
40
Test Loss
2.2
2.2
2.0
2.0
2.0
0
20
40
20
40
20
40
Train Loss
2.25
2.25
2.25
2.00
2.00
2.00
1.75
1.75
1.75
0
20
40
20
40
20
40
95k
56k
17k
Figure 10: The training process averaged over four runs of C-NODE with 95071, 55855, and 17379 parameters on the CIFAR- dataset, and NODE with 96044, 55855, and 17379 parameters. The first row is the prediction accuracy, the second row is the testing error. and the third row is the training error. Blue lines are the results for C-NODE, and red lines are the results for NODE.
26

---

D
Algorithm for continuous normalizing flows defined with C-NODE
603
We additionally provide algorithms for training and sampling CNFs defined with C-NODEs.
Algorithm 2 Algorithm for training CNFs defined with C-NODE
given probability density function of p(u(s 0)) po(-)
for each input data 2j do
u 1
3j
Given
logp(z,) - log (((())
n(0)
procedure Integrate from + to get
log (og)(z)) - log p( u (0))
for each time step s., do
calculate dx de (x, u; g(z;; 11); O2) and J Jxu(x, O2). du Jx dx de - Jxu de
calculate
calculate
dx
with Hutchinson trace estimator [13
u(sm+1) =
u(sm) log p(u(sm)))
du dt @ log pi (m()))) Os
calculate
+
(sm-1 8m)
log logp(u(sm++))
end for
evaluate Pn (n(u)))
calculate log p(z,) (log p(z,) log p(u()))) log (o(()(
optimize log p(z,) with an optimization algorithm (stochastic gradient descent etc.)
end for
604
Algorithm Algorithm for sampling CNFs defined with C-NODE
procedure sample u(s = 0) from base distribution po())
procedure Integrate from for each time step 8m do
1 to get u(s = 1)
calculate dx
da [x, u: g g(zy; 11): O2) and Jxu(x, u: O2).
calculate de calculate u(sm+1) u(sm) end for end procedure
du = J ds
dx
du (sm+1 8m).
u(s = 1) is our sample from the CNF
27


Reviews:
[{"review_id": "x213j0xqdXO", "reviewer": "Reviewer_EacU", "summary": "The authors investigate the role of the method of characteristics (MoC) on the modeling of the latent variables of NODEs, and propose a new algorithm named C-NODEs. C-NODEs parameterize the latent variables of NODEs as the solution of a family of first-order quasi-linear PDEs along the characteristic curves. The authors prove C-NODEs can learn intersecting trajectories and also are universal approximators of homeomorphisms. Experiments empirically show C-NODEs improve the learning accuracy and efficiency on multiple tasks.", "questions": "\n1. This paper uses MoC to parameterize or augment the latent variables of NODEs. However, this parameterization restrict the latent varialbes to the solution of first-order quasi-linear hyperbolic PDEs. Does this hypothesis make sense?\n\n2. Some other papers also propose ways to augment NODEs by parameterize the latent variables, such as ANODE, Second-order NODEs (SONODE). Are C-NODEs better than these methods?\n\n3. In Equation (7) (line 129), the functions $J_x u$ and $dx/ds$ are modeled by neural networks. Using $\\text{NN}_1$ and $\\text{NN}_2$ to represent neural networks, is it equivalent to $u(x(T)) = u(x(0)) + \\int_0^T \\text{NN}_1(x,u;\\Theta_2) \\text{NN}_2(x,u;\\Theta_2)ds$? If so, the structure of Equation (4) will not be preserved.\n\n4. Near the line 111, $\\frac{d}{ds}u(x(s),t(s)) = \\frac{\\partial u}{\\partial t} + u \\frac{\\partial u}{\\partial x} = 0$. If so, by integrating over $s$, we have $u(x(T),t(T);x_0,t_0) \\coloneqq \\int_0^T \\frac{d}{ds} u(x(s),t(s))ds = \\int_0^T 0 ds = 0$ ?\n\ntypo: line 3 - ``a latent variables\"", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The expressiveness of vanilla NODEs is indeed a problem due to the restriction of ODEs. The paper try to involve the family of PDEs to augment NODEs. The paper is easy to follow. The strength of this paper is the rich experiments and analysis of the properties of C-NODEs. However, the presented work raises several questions. The main weakness is the clarity and insufficient baselines. See questions below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "acm86c_GUX3", "reviewer": "Reviewer_E8Rb", "summary": "The authors propose Characteristics NODE (C-NODE) which parametrize the method of characteristics ODE of Hamilton-Jacobi equations. The authors provide density estimates for C-NODE. The authors backs up their claim of improving performance with experiments on images and time series.", "questions": "- In algorithm session, only procedures for image processing tasks are discussed. What about the other ones?\n- Also in algorithm session, what is your starting x(s=0)?\n- Equations like burgers may produce shock or rarefactions during their solution. How do you deal with those?\n", "limitations": "MOC only applies to Hamilton-Jacobi equations (not just hyperbolic), but still, it does not apply to all equations and may fail to model some type of problems. This should not be an essential problem since it could be solved with Augmentation.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of combining method of characteristics and NODE is very straightforward and new.\n- The density estimates are theoretical supported and works well on CNF type problem in their experiments\n\nWeakness:\n- Section 3 is a little bit messy. It is easy to find out the idea behind, but it is a bit confusing on things like whether they want to solve the PDE numerically or the characteristics ODE numerically; how they will do interpolation after they solve with characteristics ODE, etc.\n- Experiments on PDE modeling and time series prediction are synthetic data only. It might be converntional for PDE modeling, for time series prediction, this is not a very convincing experiment.\n- For problems without a general PDE view, like image classification and time series prediction, the model is simply combination of equations (5) and (6), which is basically more augmentations on NODE, which may not be something new. Still, this provides an interesting PDE viewpoint that is very inspiring for CNF tasks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ETJZyE1uWT4", "reviewer": "Reviewer_1nEB", "summary": "**Edit after Reviewer and AC discussion**\n\nAfter a thorough discussion with the other reviewers and AC unfortunately I will lower my score to a 6. The main reason is that the most revealing experiment is in Table 1, which is a strong experiment. Revealing how all the baselines change with and without C-NODE. But image classification is not the best experiment for NODEs. Instead this strong comparison should be carried out on all experiments especially Normalizing Flows and Time-Series.\n\n\n**Original**\n\nThis paper adapts the Neural Ordinary Differential Equation (NODE) framework to run along the characteristic curves of partial differential equations (PDES), the new model is called Characteristic Neural ODEs (C-NODEs).\n\nThis effectively is a different way to make more expressive NODE models, similar to Augmented Neural ODEs and Neural Controlled ODEs being techniques to improve expressivity and generalization of NODEs.\n\nProofs are provided to show this improves expressivity. C-NODEs are tested on classification and continuous normalizing flow tasks (density estimation) on MNIST, CIFAR-10 and SVHN. C-NODEs are also tested on time-series and PDE tasks. C-NODEs perform well on these tasks, outperforming NODEs as well as Augmented NODEs, 2nd Order NODEs and IL-NODEs. ", "questions": "My only question is how best to view these C-NODEs, as they are solved with blackbox ODE solvers. Are they more like ODEs or PDEs? And if they are like ODEs is it possible for a NODE (or an ANODE/Latent NODE) to learn the same dynamics as C-NODE. Is it better to think of C-NODE as limiting the dynamics function so it has to be in the form of a characteristic curve? Similar to how convolutions share weights across linear layers to enforce translation symmetry.", "limitations": "The paper sufficiently explores its limitations. There is no broader impact statement, it isn't really required here, however, it is good to keep in mind that the applications of NODEs and C-NODEs can include some potentially unethical ones. This is because they can generally be applied to learning time-series and now partial differential equations. As said, this is not the case in the current form of the paper, the work is incremental and not applied in those directions. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is generally very good, with good contributions and methdology, specifically:\n- The theoretical grounding of C-NODEs is good.\n- The experiments on the whole are good, experimenting against expected baselines on standard benchmark tests.\n- The paper is well written, with helpful examples for difficult/new concepts.\n\nWeaknesses:\n\nThe largest weakness is that the paper generally feels a little incomplete, the work is brilliant but there are some experiments that could make it a brilliant paper. Specific examples are:\n\n- There is this term $g(z)$ included in the dynamics, allowing the vector field to be conditioned on the input. It would be quite informative to see how much this term affects the answer. That is, can this term (a learnable function with the same structure) solve the classification problem on its own? This would make a convincing ablation if it can't.\n- This term $g(z)$ also seems to give the initial condition for the C-NODE, it might be helpful to visualise the C-NODEs time evolution, if it stays roughly the same $g(z)$ is doing the majority of the work.\n- Parameter efficiency is mentioned in the paper, but this doesn't seem to be explicitly tested, again this would make a good appendix study.\n- Results are missing for testing Continuous Normalizing Flows (and discrete flows) on SVHN.\n- The paper claims the adjoint method is better because it can be used with adaptive solvers due to memory efficiency. The Euler method is used with direct backprop. It would be helpful to show that Dopri5 fails when using direct backprop because memory runs out.\n- It is noted that coupling C-NODEs with ANODEs can make training more stable, apart from in the classification experiments it doesn't look like this has been tested.\n\nAside from this there are some minor typos or tiny mistakes that do not affect the quality of the paper but for the authors knowledge:\n\n- Line 26-27: It might help to clarify the adjoint method is only memory efficient in integration time, and not size of the dynamics function for example.\n- Between lines 52 and 53: I would change \"would take too much memory\" to \"would require too much memory\".\n- Line 85: \"then its log likelihood from Chen et al.:\" should be changed to \"then its log likelihood from Chen et al. is given by:\"\n- Line 107: Implies $s \\in [0, 1]$ when it seems that $s \\in[0, T]$\n- Line 200: \"NFE is a indicator\" should be \"NFE is an indicator\"\n- Line 230: \"Differential equations are solved using the adjoint method and a Runge-Kutta of order 5 of the Dormand-Prince-Shampine solver.\" Should be \"Differential equations are solved with a Runge-Kutta of order 5 of the Dormand-Prince-Shampine solver and trained with the adjoint method\".\n- Line 236: \"using a lower NFEs\" should be \"using a lower NFE\".\n- Line 265: \"and can combine with\" should be \"and can be combined with\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject