Learning to Decouple Complex System for Sequential Data
Authors: Zihan Zhou
Minxin Ao
Tianshu Yu
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural differential equation
sequential learning
decoupling complex system

Abstract: A complex system with cluttered observations may be a coupled mixture of multiple simple sub-systems corresponding to \emph{latent entities}. Such sub-systems may hold distinct dynamics in the continuous-time domain, therein complicated interactions between sub-systems also evolve over time.  This setting is fairly common in the real world, but has been less considered. In this paper, we propose a sequential learning approach under this setting by decoupling a complex system for handling irregularly sampled and cluttered sequential observations. Such decoupling brings about not only subsystems describing the dynamics of each latent entity, but also a meta-system capturing the interaction between entities over time. Specifically, we argue that the meta-system of interactions is governed by a smoothed version of \emph{projected differential equations}. Experimental results on synthetic and real-world datasets show the advantages of our approach when facing complex and cluttered sequential data compared to the state-of-the-art.

Under review as a conference paper at ICLR 2023
LEARNING TO DECOUPLE COMPLEX SYSTEM FOR SEQUENTIAL DATA
Anonymous authors Paper under double-blind review
ABSTRACT
A complex system with cluttered observations may be a coupled mixture of multiple simple sub-systems corresponding to latent entities. Such sub-systems may hold distinct dynamics in the continuous-time domain, therein complicated interactions between sub-systems also evolve over time. This setting is fairly common in the real world, but has been less considered. In this paper, we propose a sequential learning approach under this setting by decoupling complex system for handling irregularly sampled and cluttered sequential observations. Such decoupling brings about not only subsystems describing the dynamics of each latent entity, but also a meta-system capturing the interaction between entities over time. Specifically, we argue that the meta-system of interactions is governed by smoothed version of projected differential equations. Experimental results on synthetic and real-world datasets show the advantages of our approach when facing complex and cluttered sequential data compared to the state-of-the-art
INTRODUCTION
Discovering hidden rules from sequential observations has been essential topic in machine learning, with a large variety of applications such as physics simulation (Sanchez-Gonzalez et al., 2020), autonomous driving (Diehl et al. 2019), ECG analysis (Golany et | [20T] and event analysis [Chen et al. 202T). to name a few. A standard scheme is to consider sequential data at each timestamp being holistic and homogeneous under some ideal assumptions (i.e., only the temporal behavior of one entity is involved in a sequence), under which data/observation is treated as collection of slices at different time from unified system. series of sequential learning models fall into this category. including variants of recurrent neural networks (RNNs) (Cho et al.) 2014 Hochreiter & Schmidhuber 1997), neural differential equations (DEs) (Chen et al. 2018 Kidger et al.) 2020 Rusch & Mishra 2021 I Zhu et al. 2021) and spatial/temporal attention-ban based approaches (Vaswani et 2017] et al.) 2019 Song et al., 2017). These variants fit well into the scenarios agreeing with the aforementioned assumptions, and are proved effective in learning or modeling for relatively simple applications with clean data source.
In the real world, system may not only describe single and holistic entity, but also consist of several distinguishable interacting but simple subsystems, where each subsystem corresponds to a physical entity. For example, we can think of the movement of a solar system being the mixture of distinguishable subsystems of sun and surrounding planets, while interactions between these celestial bodies along time are governed by the laws of gravity. Back to centuries ago, physicists and astronomers made enormous effort to discover the rule of celestial movements from the records of each single bodies, and eventually delivered the neat yet elegant differential equations (DEs) depicting principles of moving bodies and interactions therein. Likewise in nowadays, researchers also developed a series of machine learning models for sequential data with distinguishable partitions QQin et al. (017). Two widely adopted strategies for learning the interactions between subsystems graph neural networks (lakovlev et al., 2021 Ha & Jeong [021 Kipf et al. 2018 Yıldız ct al. 2022
Xhonneux et al. 2020) and attention mechanism (Vaswani et al. 2017 Lu et al. 2020 Goyal et al.
2021), while the interactions are typically encoded with "messages" between nodes and pair- wise "attention scores", respectively.
is worth noting a even more difficult scenario, in which the data/observation is so cluttered that cannot be readily distinguished into separate parts. This be either due to the way of data

---

Under review as a conference paper at ICLR 2023
collection (e.g., video consisting of multiple objects), because there is no explicit physical entities originally (e.g., weather time series). To tackle this, a fair assumption can be introduced that complex observations can be decoupled into several relatively independent modules in the feature space, where each module corresponds to a latent entity. Latent entities may not have exact physical meanings, but learning procedures can greatly benefit from such decoupling, as this assumption can be viewed as strong regularization to the system. This assumption has been successfully incorporated in several models for learning from regularly sampled sequential data, by emphasizing "independence" to some extent between channels or groups in the feature space (Li et al., 2018 Yu et al. 2020) Goyal al. 2021; Madan al.] 202T). Another successful counterpart in parallel benefiting from this assumption is transformer (Vaswani et al., 2017) which stacks multiple layers of self-attention and point-wise feedforward networks. In transformers, each attention head can viewed as a relatively independent module and interaction happens throughout head re-weighting procedure following the attention scores. Lu et al. (2020) presented an interpretation from a dynamic point of view, by regarding a basic layer in the transtormer as one step of integration, governed by differential equations derived from interacting particles. Vuckovic et al.) (2220) extended this interpretation with more solid mathematical support by viewing forward pass of transformer applying successive Markov kernels in particle-based dynamic system.
We note, however, despite the ubiquity of this setting, there is barely any previous investigation focusing on learning for irregularly sampled and cluttered sequential data. Aforementioned works either fail to handle the irregularity (Goyal et al.) 2021 Li et al., 2018). or neglect the indepen- dencc/modulariity assumption in the latent space (Chen et al. 2018 Kidger et al. 2020). In this paper, inspired by recent advances of neural controlled dynamics (Kidger et al. 2020) and novel interpretation of attention mechanism (Vuckovic et al. 2020), we make a step to propose an effective approach addressing this problem under dynamic setting. To this end, our approach explicitly learned to decouple a complex system into several latent sub-systems and utilizes additional meta-system capturing the evolution of interactions over time. Specifically, taking into account the constrained interactions analogous to the attention mechanism, further characterized such interactions using projected differential equations (EsojDEs). We argued our contributions as follows:
We provide a novel modeling strategy for sequential data from a system decoupling perspec- tive;
We propose a novel and natural interpretation of evolving interactions ProjDE-based meta-system under constraints;
Our approach is flexible to be integrated into various tasks.
insensitive and more compatible to other modules, thus being
Extensive experiments were conducted either regularly or irregularly sampled sequential data, in- cluding both synthetic and real-world settings. It was observed that our approach achieved prominent performance compared to state-of-the-arts on wide spectrum of tasks.
2 RELATED WORK
Sequential learning. Traditionally, learning with sequential data can be performed using variants of recurrent neural networks (RNNs) (Hochreiter & Schmidhuber 1997 Cho et al.) 2014 Li et al. 2018)
under a Markov setting. While such RNNs are generally designed for regularly sampling frequency, a more natural line of counterparts lie in the continuous time domain allowing irregularly sampled time series as input. As such, a variety of RNN-based methods are developed, by introducing exponential decay on observations (Che et al. [2018] Mei & Eisner (017). incorporating an underlying Gaussian process (Li & Marlin 20to/Futoma et al. 2017), or integrating some latent evolution under ODEs seminal work interpreting forward passing in neural networks as integration of ODEs was proposed in Chen et al. (2018), following by series of relevant works (Liu et al. 2019 Li et al. [2020a] Dupont et al. 2019). As integration over ODEs allows for arbitrary step length, it's s natural modeling of irregularly time series, and proved powerful in many machine learning tasks (e.g., bioinformatics (Golany et al. 2021), physics (Nardini et al. 2021) and computer vision (Park et al. 22021). Kidger et al. (2020) studied more effective way of injecting observations to the system via a mathematical tool called Controlled differential Equation. achieving state-of-the-art performance several benchmarks. Some variants of neural ODEs have
(Rubanova et al. [019]] De Brouwer et al. 2019).
2

---

Under review as a conference paper at ICLR 2023
also been extended to discrete structure (Chamberlain al. 2021a Xhonneux et al. 2020 and non-Euclidean setting Chambeelain et al. 2021b).
Learning with independence. Independence or modular property serve as strong regularization prior in some learning tasks under static setting (Wang et al. 2020; Liu et al. (2020). In the sequential case, some early attempts over RNNS emphasized implicit "independence" in the feature space between dimensions or channels (Li et al. 2018 Yu et al. 2020). As independence assumption commonly holds in vision task (with distinguishable objects). Pang et al. (2020); Li et al. (2020b)
proposed video understanding schemes by decoupling the spatio-temporal patterns. For more generic case where the observations are collected without any prior, /oyal et al. (202T) devised an sequential learning scheme called recurrent independence mechanism (RIM), and its generalization ability was extensively studied in Madan et al. (2021). Lu et al. (2020) investigated self- attentioo mechanism (Vaswani et al. 2017 and interpreted it as nearly independent multi-partion system with interactions therein. Vuckovic et al. (2020) further provided more solid mathematical analysis with the tool of Markov kernel. Study of such mechanism in the dynamical setting was barely observed.
3 METHODOLOGY
3.1 BACKGROUND
In this section, we briefly review three aspects related to our approach. Our approach is built upon the basic sub-system derived from Neural Controlled Dynamics (Kidger et al.] 2020), while the interactions are modeled at an additional meta-system analogous to Self-attention (Lu et al. 2020 Vuckovic et al.) 2020), and further interpreted and generalized using the tool of Projected Differential Equations Dupuus & Nagurney
1993).
Neural Controlled Dynamics. Continuous-time dynamics can be expressed using differential equations z'(t) = dz/ dt = f(z(t), t), where z E IR and t are a d-dimension state and the time, respectively. Function f R. x IR + Rd governs the evolution of the dynamics. Given the initial state z(to), the state at any time ( can be evaluated with:
z(t1) z(to) + S/ (2
f(z(s),s)ds
In practice, we aim at learning the dynamics from series of observations or controls (x(tx) R. lk 0, 1, ...} by parameterizing the dynamies with fo(-) where / the unknown parameter be learned. Thus, generic dynamics incorporating outer signals can be written as:
I" So(z(>), x(6), s)ds
z( ((1) = z((o) +
Rather than directly injecting x as in Eq. 2 Neural Controlled Differential Equation (CDE) proposed to deal with outer signals with a Riemann-Stiellen Stieltjes integral (Kidger et al., 2020):
x (44) 2(16) + [" Fo(2(s))x'(s)dd
where Fg : Rd - dxd is learnable vector field and x'(s) dx/ds ss is the derivative of signal w.r.. time s, thus "Fo(z(s))x' (s)" matrix-vector multiplication. During implementation. Kidger et al. (220)) argued that simple cubic spline interpolation on allows dense calculation of x (t) at any time i, and exhibits promising performance. Kidger et al. (2020), is also mathematically shown that, incorporating observations/controos following Eq. (3) with greater representation ability compared to Eq. (2). hence achieving state-of-the-a performance on several public tasks.
Self-attention. It is argued in Lu et al. 2020); Vuckovic et al. (020 that basic unit in Transformer (Vaswani et al.] (017)) with silk-link consisting of one self-attention layer and point-wise feedforward layer amounts to simulating a multi-particle dynamical system. Considering such layer with n attention-heads (corresponding particles), given an attention head index E {1,2, ..., n), the
3

---

Under review as a conference paper at ICLR 2023
update rule of the ith unit depth reads:
11, 2i,; + MultiHeadAttya.. (Z1,5, [2,1 ... 21,n))
(4a)
1,4 21,; + FFN who (21,))
(4b)
where MultiHeedAAttyy. and FFN Win. are multi-head attention layer and feedforward layer with parameters W att and W system:
fin respectively. Eq. (4) can then be interpreted interacting multi-particle
(t,(t) (z, (t), 2t)).. (((()))))((( dt
where function F corresponding to Eq. 4a)) represents the diffusion term and G corresponding to Eq. 4bb) stands for the convection term. Readers referred to Lu et al. (2020); Vuckovie al. (2020) for more details.
Projected DEs. It is tool depicting the behavior of dynamics where solutions are constrained within a set. Concretely, given a closed polyhedral KC c: IR" and a mapping H K. - R", we can introduce an operator 11g: 1 x K R" which is defined by means of directional derivatives as:
Pc(a (H()))
(a Ha))
lim a-0)
a
where Px(() a projection onto K. in terms of Euclidean distance:
PP((a) - all2
inf lyy - a|2
yek
Intuitively, Eq. pictures the dynamics of a driven by function H, but constrained within X. Whenever reaches beyond K, it would be projected back using Eq. 77. By extending Eq. (
Dupuis & Nagurne (993); Zhang & Nagurney 1995) considered the projected differential equations
as follows:
da(l) [x H(a))
(8)
which allows for discontinuous dynamics on a.
3.2 LEARNING TO DECOUPLE
Our method is built upon the assumption that cluttered sequential observations are composed from several relatively independent sub-systems, and therefore explicitly learns to decouple them as well as capture the mutual interactions with meta-s seeta-system parallel. Let the cluttered observa- tions/controlls be c(t) E R at time t for t = I.... T, where T is the time horizon. We employ distinct mappings with learnable parameters (e.g., MI.P) to obtain respective controls to each sub-system: (t) = pc(((t) E I for i = 1,..., A generic dynamics of the proposed method can be written as:
dz, fi (z:(1),
[2) (!),.., (1) x; (1), for = 1,..,,
(9a)
di
da(t)
II' (a(t), g(((t) [z (f), z,, (t)))
(9b)
dt
where Eq. (9a) and Eq. (9b) refer to the ith sub-system describing the evolution of single latent entity and meta-system depicting the interactions, respectively. (t) E R7 is the hidden state for the ith subsystem, and a is tensor governs the dynamics of the interactions. Here (Ts(.) is projection operator, which projects the evolving trajectory into set S. We introduce such a operator as assumed that interactions among latent entities should be constrained following some latent manifold structure. f((-) and g()) are both Iearnable functions, and also the essential roles for capturing the underlying complex dynamics.
According to Eq. ( we fully decouple a complex system into several components. Although we found some decoupling counterparts in the context of RNNs (Li et al. 2018: Yu et al.) 2020) and attention- like mechanism (Lu et al., 2020 Goyal et al. 2021). their decoupling could not be applied our problem. We elaborate the details of implementing Eq. (9) in the followings.
4

---

Under review as a conference paper at ICLR 2023
Learning sub-systems. Sub-systems corresponding the latent entities seek to model relatively independent dynamics separately. Specifically, we employ the way of integrating x, into Eq. (9a) in controlled dynamical fashion in the sate-of-the-e of-the-art art method (Kidger et al. 2020):
(t) = F; (L), a(t) [2,(1), ..., Zn (f)) (1)
(10)
where (.) E Rexm is a learnable vector field. Concretely, if we let ( = (t),....... 2n (t) be the tensor collecting all sub-systems, the ith sub-system self-attention fashion reads:
dz = (() (dx,))))(((
(11)
where F. takes the ith slice from a tensor. Note timestamp t can be arbitrary. resulting in an irregularly sampled sequential data. To address this, we following the strategy in in(iigeer al. (2020)
by performing cubic spline interpolation on x; over observed timestamp I, resulting in x; x;()) at dense
time L. Note that for all sub-systems, different from Eq. (T0) we utilize an identical function/netwwork
F(-) as in Eq. (I), but with different control sequence x, (t) (.(())). Since in our implementation p-()) is a lite network such as MLP, this can significantly reduce the parameter size.
Learning interactions. In our approach, interactions between latent entities modeled separately as another meta-system. This is quite different from some related methods (Lu et al. 2020 Vuckovic et al. 2020) where sub-systems and interactions treated as holistic step of forward integration. Such thorough decoupling also brings about gradient with higher magnitude in the early training stage (see Figure 2). which allows for faster convergence and more significant gradient backpropa- gation. For the meta-system describing the interactions in Eq. (9b). two essential components are involved: domain and the projection operator II. In the context of ProjDEs, a system is constrained as a(t) E S for any t. In terms of interactions. common choice of S is the stochastic simplex which can be interpreted a transition kernel (Vuckovic al., 2020).
We allow follow this setting by
defining S be row-wise stochastic (n- J-simplices:
S 2 {A E gax" A1 = A, 2 0}
(12)
where 1 is vector with all I entries. A ma) is n X matrix. In the sequel, we will use the notation A throughout. Thus the meta-system capturing the interactions can be implemented as:
dA(l) ITs (A o (1), (2) ... 2. (4)))
(13)
dt
Notably, the projection operator (soft (-)) in the attention mechanism can be viewed as a row-wise projection onto the (n 1)-simllex with entropic regularization (Amos 2019):
Ps(A,,)
min - A,, - HB.))) BES
(14)
where HI(()) is the entropy function. A and B.j are the ith row and column of A and B, respec- tively. is readily derived that row-wise softmax A) is the solution to Eq. (14 by utilizing KKT condition. Comparing Eq. [14] to the standard Euclidean projection in Eq. Z7).. we note the entropic regularization H[(-) in Eq. (14) allows for smoothed trajectory without discontinuity, thus eases the gradient flow over the whole integration path.
While Eq. 13) defines a projected dynamical system directly A, switch update the system using L as follows, which is considered to further ease the forward integration. This is achieved by instead modeling the dynamies of the feature before fed into softmax-())
(t) Softmax (L()))
(15a)
L(t) L(0) + [ d (()) K (Z(s)) ds,
(15b)
0 ds
Vdn
L(l + At) L(() At d Q(Z(())) (s)
K (Z(s))
(15c)
ds
Vdz
where Q(-) and K(-) correspond to the query and key in the attention mechanism, respectively. (() = Q()))) KT (Z(0))/ /dg. We show that updating the dynamic of following Eq. 15) is
equivalent to directly updating A in Appendix A.2
We employ the standard Euler's descritization for performing the integration, by updating z and A simultaneously. We term approach decoupling-basee neural system (DNS).
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
We evaluate the performance of DNS on multiple synthetic and real-world datasets. More details about the dataset and implemetation details can be found in Appendix A.9 Throughout all the tables consisting of the results, a placeholder "x" is referred to a failure during training process after multiple trials, and s_" indicates "not applicable" (RIM cannot handle irregular cases).
Baselines. We compare DNS with several selected models capturing interactions or modeling irregular time series, including CT-RNN (Funahashi & Nakamura] 1993) modeling continuous dynamics, CT-GRU (Mozer et al. 2017) using state-decay decay mechanisms, RIM (Goyal et al. 2021) updating almost independent modules discretely, and NeuralCDE (Kidger et al. 2020) which reports of-teoof-hhe-at performance on several benchmarks. We use "method x y and "method y to indicate settings, where and are the number of the underlying modules (e.g., sub-systems in method and independent components RIM) and dimension the latent feature.
Table I: Trajectory prediction. Three body.
Table Link predition. Charged.
Square-Emrr Error (x (x10-?)
Accuracy( %
Model
Model
Regular
Irregular
Regular
Irregulat lar
CT-RNN 512
2.1291
3.0538
CT-RNN 64
61.86
61.90
CT-RNN 1024
1,9943
2.8968
CT-RNN 128
62.60
63.47
CT-RNN 2048
1.9514
3.0008
CT-RNN 256
62.54
65.40
CT-GRU 512
1.8554
2.4997
CT-GRU 64
63.13
63.23
CT-GRU 1024
1.7155
2.3020
CT-GRU 128
63.76
63.64
CT-GRU 2048
1.6421
2.1996
CT-GRU 256
64.18
65.16
NeuralCDE 512
3,9213
5.0077
NeuraICDE 64
67.90
65.83
NeuralCDE 1024
4.4934
5,4811
NeuralCDE 128
68.92
66.97
RIM 512
2.4519
RIM 64
66.46
RIM 1024
2.1865
RIM 128
69.14
DNS 512
1.8272
2,4874
DNS 5 64
67.27
66.04
DNS 1024
1.8206
2.3650
DNS 5 128
69.07
65.23
DNS 2048
1.5012
2.0994
DNS 5 256
68.37
64.99
4.1
THREE BODY
Three body problem is characterized by chaotic dynamical system for most randomly initial conditions. A small perturbation may cause drastic changes of the movement. Taking into account the problem complexity, it is particularly suitable for testing our approach. In this experiment, we consider a trajectory predicting problem given the noisy historical motion of three masses, where gravity causes interactions between them. Therefore, models need to (implicitly) learn both Newton's laws of motion for modeling sub-system dynamics and Newton's law universal gravitation to decouple the latent interaction. This dataset consists of 50k training samples and 5k test samples. For each sample, 8 historical locations for regular setting and 6 historical locations (randomly sampled from 8) for irregular setting 3-dimensional space of three bodies are given to predict subsequent locations, hence 9-dimensional observation at each time stamp. To equip with the cluttered setting, the correspondence between dimensions and bodies will not be fed into the learning models. Each model is trained until convergence and the average of mean squared error (MSE) of the last 5 epochs on the test set is reported as the accuracy. Results are summarized in Table D We can conclude that "DNS 2048" outperformed all the selected counterparts in both regular and irregular settings. Notably, although our method is built on NeuralCDE, but with the decoupling, the performance can be significantly improved.
Latent entity V.S. physical entity. We visualize dynamics A along the movements of three body system. Note set up 3 sub-systems for potentially capturing dynamics of 3 physical entities. Figure [ shows that when three bodies approach each other, the off- oii-oiagonal entries of A increase, indicating stronger interactions when physical bodies close to each other. This implies that DNS can somewhat decouple a complex three body system according to its physical meaning in the real world.

---

Under review as a conference paper at ICLR 2023
P90
2.000
8/011
2:11
0.30:
2,001
1.59
0:301
7,000
3.10
5,009
0 0.55
0:322
2.251
0.387
6:722
0.034
363
6:032
7.184
0,552
615
0.200
1:225
1130
0.002
3:314
E.482
2.366
2.742
2.163
6.122
7.27
0.3
0.199
0.600
0.333
2,003
0110
Figure 1: A figure showing the evolution over time on interactions (the upper row) between three latent sub-systems, as well as the corresponding three physical bodies in a Three Body environment.
continuous softmax
0.015
0.010
0.005
0.000
1030
1250 1500
1750
2000
Figure 2: Comparison of magnitude of gradient of A along with the training process. "Continuous softmax" and "Discrete softmax" correspond to the update rules following Eq. (15) and holistic forward integration in Lu al. (2020); Vuckovic et (2020), respectively.
4.2 SPRING
We experiment the capability of DNS in decoupling the independence in complex dynamics controlled by simple physics rule. We use simulated system in which particles connected by (invisible) springs (Kuramoto 1975 Kipf et al., (018).. Each pair of particles has equal probability of having interaction not. Our task is to use observed trajectory predict whether there are springs between any pair of two particles, which is analogous to the task of link prediction under a dynamical setting. This can be inferred from whether two trajectories change coherently. Spring dataset consists of 50k training examples and 10k test examples. Each sample has the length of 49. We also test model's noise resistance and add Gaussian noise to the observed data. We set up two scenarios "Train& Test" and "Test" (see Table 3 corresponding to injecting noise at both training and testing phases, and only testing phases, respectively. We test variety of combinations of the number of sub-systems and dimension of hidden state. Experimental results are in Table 3, From the first two blocks of Table 3 we see standard DNS stably outperforms all the selected counterparts by a large margin when the data clean. When noise exists on both training and testing phases, DNS exhibits somewhat sensitivity to the coefficients, thus needs to be fine-tuned. However, when only testing samples are noisy, DNS again maintains high stability and promising performance.
Variants of DNS. To examine the compatibility of our model, we create two variants of DNS. The first version is to replace cubic spline interpolation over xt(t) with natural smoothing spline (Green & Silverman 1993), in consideration of incorporating smoother controls. This version is termed as DNSg. Another version of DNS is without explicitly modeling the dynamics of A as in Eq. 15 but treating one step of integration as a holistic forward pass consisting of both z and A following the interpretation in Lu et al. (2020); Vuckovic et al. (2020). This variant is called DNS, Evaluation results on these two variants are summarized in the third block in Table 3 According to Table even though DNSş does not exhibit the best performance with clear data,
quite reliable the noisy cases. It seems a smoothing procedure on the controls can be helpful under massive uncertainty. This also raises one of our future research directions to investigate the way handling different controls. DNSp. with holistic forward integration, cannot achieve performance comparative to standard DNS. To analyze, we visualize the magnitude of gradient of A (with DNS and DNSp) along with the training process in Figure 2 One readily find that the proposed method DNS by fully decoupling A can derive a more significant gradient in the early training stage than DNS,- We argue that, explicitly introducing interaction dynamics A creates "highway" in parallel to each
7

---

Under review as a conference paper at ICLR 2023
Table Link prediction. Spring dataset.
Clean
Noisy
Model
Param Size(MB)
Regular
Irregular
Train& Test
Test
CT-RNN 128
71.39
76.96
70.62
69.91
0.1
CT-RNN 256
76.68
81.00
75.88
75.99
0.3
CT-RNN 512
72.44
51.68
75.09
71.46
1.1
CT-GRU 128
86.20
87.13
85.90
86.14
1.3
CT-GRU 256
88.14
87.92
88.13
88.11
4.8
CT-GRU 512
89.57
88.61
89.17
89.24
18.0
CT-GRU 1024
90.97
89.99
90.25
90.84
70.0
NeuralCDE 128
90.02
89.55
77.32
75.51
1.3
NeuralCDE 256
91.06
90.55
80.12
77.79
5.3
NeuralCDE 512
91.57
91.40
81.87
78.22
21.1
NeuralCDE 1024
91.23
90.71
80.47
79.15
84.2
RIM 128
85.77
85.82
85.74
9.9
RIM 256
86.92
86.64
86.91
25.7
RIM 512
87.43
87.42
87.43
75.4
DNS 128
93.29
93.15
91.96
87.97
2.8
DNS 5 256
93.81
93.43
92.56
86.36
10.7
DNS 512
88.07
88.04
89.74
83.64
41.8
DNS 8 128
93.59
94.43
66.72
85.40
2.9
DNS 256
94.92
95.62
93.73
87.24
11.0
DNS 8 512
95.38
95.40
76.52
89,44
42.3
DNS
93.79
93.04
93.11
86.00
3.1
DNS 10 256
94.86
95.20
93.90
88.17
11.2
DNS 10 512
92.99
92.79
91.48
87.41
42.7
DNSş 128
93,00 (5)
93.15 (5)
93.73 (8)
91.82 (8)
3.1
DNSş 256
93,99 (5)
93.43 (5)
94.15 (8)
78.65 (8)
11.2
DNSs 512
95.04 (5)
88.04 (5)
94.15 (8)
93.26 (8)
42.7
DNSp 128
92.91
93.28
91.40
87.61
2.8
DNSp 256
93.80
94.00
92.22
87.16
10.7
DNSp 5.512
88.78
91.93
89.65
81.92
41.8
sub-system, which tends ease the gradient flow throughout the training, and is thus beneficial for training on complex and cluttered sequential data.
4.3 CHARGED
Experiment setting of charged dataset follows Kuramoto (1975): Kipf et al. (2018). Charged particles attract repel each other with equal probability. Interaction strength inversely proportional the square of the distance and results more complex system compared with spring dataset. Particles are more frequently bounced back by the boundary of the container, leading to a sudden change of trajectory. Hence, intuitively, models introduced for continuous dynamics may have limited performance. Experiment results are Table this dataset, DNS achieved promising performance close to RIM and NeuralCDE. We observe continuous-time versions of RNNs (CT-RNN and CT- GRU) don't perform well under such setting with several changes boundary, yet DNS achieves good accuracy.
4.4 HUMAN ACTIONS
Recognition of human actions dataset contains three types of human actions which are hand clapping, hand waving and jogging (Schuldt et al. 2004). take 50 equispaced frames from each video. consider limbs of the character subsystems. When the character does one
For this dataset, kind of actions, subsystems interact in a specific pattern. test the compatibility of all the selected models by turning the parameters of the backhone Resnet18 8 (He et al. 2016) untunable and tunable.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Video classification. Regular and Irregular Human Actions.
Unnormalized
Normalized
Model
Regular
Irregular Trainable
Regular Frozen
Irregular
Frozen
Trainable
Trainable
CT-RNNN 32
33.33
46.17
48.00
88.33
37.22
CT-RNN 64
33.50
47.50
58.33
90.00
24.44
CT-GRU 32
63.83
58.33
56.00
96.67
66.67
CT-GRU 64
37.00
60.33
66.67
96.67
72.78
NeuralCDE 32
68.33
52.47
57.83
81.67
33.89
NeuralCDE 64
70.50
70.33
59.17
61.67
RIM 32
49.33
55,50
95.00
RIM 64
43.00
44.83
95.00
DNS 32
70.17
95.00
95.33
96.67
86.67
DNS 64
78.67
97.00
93.17
96.67
87.78
We also feed the model with data in raw format (pixel value in each channel is [0, 255)) and normalized format (pixel value in each channel follows Gaussian (0.5, 0.5)), to examine robustness of different methods under various changing scales. Experimental results are summarized Table 4 In unnormalized case, DNS significantly outperformed all other counterparts, demonstrating strong robustness for raw data collected from a large range. DNS is also more compatible to backbone of Resnet18, 8, and thus potentially being more flexible to be integrated into various tasks. In the normalized setting, although DNS and CT-GRU hold comparative accuracy, DNS delivers prominent performance under the irregular setting.
To view how the decoupling works for video recognition task, we visualize the strength the learned parameters mapping 128-D feature into latent sub-systems in Figure 3 For better view, re-organize the order of the 128-D feature. It can be seen that there are some latent structure in the parameters grouping 128-D control to the system. Each sub-system mainly focuses on small portion of the control, based on which we can infer that each sub-system models different components in inputted images.
Sub Sub
Sub 5 Sub 6
Figure 3: A figure showing the importance of each feature vector entry for subsystems
5 CONCLUSION
In this paper, we propose method for modeling cluttered and irregularly sampled sequential data. Our method is built upon the assumption that complex observation may be derived from relatively simple and independent latent sub-systems, wherein the interactions also evolve over time. We devise a strategy to explicitly decouple such latent sub-systems and a meta-system governing the interaction. Inspired by recent findings about self-attention mechanism, and further utilizing the tool of projected differential equations, we present an novel interpretation to our model. Experiments on various tasks demonstrate prominent performance of our method over previous state-of-the-ae methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Brandon Amos. Differentiable aptimization-basee modeling for machine learning. Ph. D. thesis, 2019.
Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. In ICMI,, 2021a.
Benjamin Chamberlain, James Rowbottom, Davide Eynard, Francesco Di Giovanni, Xiaowen Dong, and Michael Bronstein. Beltrami flow and neural diffusion on graphs. NeurIPS, 2021b.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In NeurIPS, 2018.
Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Learning neural event functions for ordinary differential equations. In ICLR, 2021.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409. 1259, 2014.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau, Gru-ode-bayes: Continuous
modeling of pporadically-observed time series. NeurlPS, 2019.
Frederik Diehl, Thomas Brunner, Michael Truong Le, and Alois Knoll. Graph neural networks for modelling traffic participant interaction. IEEE Intelligent Vehicles Symposium (IV), 2019.
Emilien Dupont, Arnaud Doucet, and Yee Whye Tch. Augmented neural odes. NeurIPS, 2019.
Paul Dupuis and Anna Nagurney, Dynamical systems and variational inequalities. Annals of Operations Research, 44(1):7-42, 1993.
Chenyou Fan, Yuze Zhang, Pan, Xiaoyue Li, Chi Zhang, Rong Yuan, Di Wu, Wensheng Wang, Jian Pei, and Heng Huang. Multi-horizon time series forecasting with temporal attention learning. In ACM SIGKDD, 2019.
Ken-ichi Funahashi and Yuichi Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural networks, 6(6):801-80) -806, 1993.
Joseph Futoma, Sanjay Hariharan, and Katherine Heller. Learning to detect sepsis with a multitask gaussian process rnn classifier. In ICML, 2017.
Tomer Golany, Daniel Freedman, and Kira Radinsky. Ecg ode-gan: Learning ordinary differential equations of eeg dynamics via generative adversarial learning. In AAAI, 2021.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bemhard Schölkopf. Recurrent independent mechanisms. In ICLR. 2021.
Peter Green and Bernard Silverman. Nonparametric regression and generalized linear models: roughness penally approach. Cre Press, 1993.
Seungwoong Ha and Hawoong Jeong. Unraveling hidden interactions in complex systems with deep learning. Scientific reports, 11(1):1-33, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Valerii Iakovlev, Markus Heinonen, and Harri Lähdesmäki. Learning continuous pdes from sparse data with graph neural networks. In ICLR, 2021.
10

---

Under review as a conference paper at ICLR 2023
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. In NeurlPS, 2020.
Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In JCML, 2018.
Yoshiki Kuramoto. Self-entrainment of population of coupled non-linear oscillators. In International symposium mathematical problems in theoretical physics, pp. 420-422, 1975.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indran): Building a longer and deeper mn. In CVPR, 2018.
Steven Cheng-Xian Li and Benjamin M Marlin. A scalable end-to-end gaussian process adapter for irregularly sampled time series classification. NeurIPS, 2016.
Xuechen Li, Ting-Kan Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, 2020a.
Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human-object interaction. NeurIPS, 2020b.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. Neural sde: Stabilizing neural ode networks with stochastic noise. arXiv preprint urXiv::906.022 2019.
Yanbei Liu, Xiao Wang. Shu Wu, and Zhitao Xiao. Independence promoted graph disentangled networks. In AAAI, 2020.
Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, and Yoshua Bengio. Fast and slow learning of recurrent independent mechanisms. In ICLR, 2021.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. In NeurIPS, 2017.
Michael C Mozer, Denis Kazakov, and Robert V Lindsey. Discrete event, continuous time mns. arXiv preprint arXiv: 1710.041 10, 2017.
John T Nardini, Ruth Baker, Matthew Simpson, and Kevin Flores. Learning differential equation models from stochastic agent-based model simulations. Journal of the Royal Society Interface, 11176))20000088, 2021.
Bo Pang, Kaiwen Zha, Hanwen Cao, Jiajun Tang, Minghui Yu, and Cewu Lu. Complex sequential understanding through the awareness of spatial and temporal concepts. Nature Machine Intelligence, 25552245-53,, 2020.
Sunghyun Park, Kangyeol Kim, Junsoo Lee, Jaegul Choo, Joonseok Lee, Sookyung Kim, and Edward Choi. Vid-ode: Continuous-time video generation with neural ordinary differential equation. In AAAI, 2021.
Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. dual-stage attention-based recurrent neural network for time series prediction. In IJCAI, 2017.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for ireeularly-sampled time series. NeurIPS, 2019.
T Konstantin Rusch and Siddhartha Mishra. Unicornn: A recurrent model for learning very long time dependencies. In /CML, 2021.
Alvaro
Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec. and Peter Battaglia. Learning to simulate complex physics with graph networks. In ICMI,, 2020.
11

---

Under review as a conference paper at ICLR 2023
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: local svm approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3. pp. 32-36. IEEE, 2004.
Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, and Jiaying Liu. An end-to-end spatio-temporal attention model for human action recognition from skeleton data. In AAAI, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
James Vuckovic, Aristide Baratin, and Remi Tachet des Combes. A mathematical theory of attention. arXiv preprint (iXXix2007002870 2020.
Yingyao Wang, Junwei Bao, Guangyi Liu, Youzheng Wu, Xiaodong He, Bowen Zhou, and Tiejun Zhao. Learning to decouple relations: Few-shot relation classification with entity-guided attention and confusion-aware training. arXiv preprint 4rXiv:2010.10894, 2020.
Louis-Pascal Xhonneux, Meng Qu, and Jian Tang. Continuous graph neural networks. In ICML, 2020.
Çağatay Yıldız, Melih Kandemir, and Barbara Rakitsch. Learning interacting dynamical systems with latent gaussian process odes. arXiv preprint arXiv:2205.11884, 2022.
Tianshu Yu, Yikang Li, and Baoxin Li. Rhyrnn: Rhythmic rnn for recognizing events in long and complex videos. In ECCV, 2020.
Zhang and A Nagurney. On the stability of projected dynamical systems. Journal of Optimization Theory and Applications, 85(1):97-124, 1995.
Qunxi Zhu, Yao Guo, and Wei Lin. Neural delay differential equations. In ICLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DETAILS ABOUT FIGURE
We replace the fully connected layer in the pretained Resnet18 with another neural network whose output size equals to 64. Freezing Resnet means that we do not update parameters in the convolutional layer. Parameters of the fully connected layers can still be updated. Image feature vectors fed forward by linear layer of size 64 by 128 and activated by ReLu function. Then, feature vectors are fed forward by distinct linear layer and we obtain different control signal for each subsystem. In Figure 3 gray points on the second line denotes entries of the 128 dimensional feature vector after reordering. For each subsystem, we plot the top 40 entries which has the greatest impact for control signals.
When plotting the figure of gradients change along the training process, we drop the points where gradients are relatively too large. In practice, we can avoid gradient exploding gradient problem by gradient clipping.
A.2 ON THE EQUIVALENCE OF MODELING dA AND
cL dr
Let Ltt) denotes the multiplication of key and query, i.e., L(t)
Q(1K (0) ddz the A softmax
we model the dynamics of L(l), we obtain
dL
L(t + At) L(t) + At
(16)
di
Apply the softmax function on both side of the equation, have
A At) softmax(L(()) At- dL
dt + A(t) A())
(t) + softmax(L(t)) + At dL softmax(L(()) dt
Reorder the equation, we have
A(t + At)
A(t)
softmax(L(t) + At dL softmax ((f))
Al
Al
softmax (L(t) + At softmax (((t)) dL
de
At de
dt
Take At + 0, we have
AA
dsoftmax (t)
dL
(17)
dt
dL
di
which is equivalent to the update step in Eq.
15
A.3 EXPERIMENT DETAILS
A.3.1 DNS
For the implementation simplicity, DNS with batch input requires each sample observed at the fist and last timestamp. Default control signal dimension equals to 2 input_siza size. When initilizing the Weight matrix key and query layer, control encoder and initial hidden state encoder, we use 0.01 x torch.rand and set bias equals to 0.
A.4 CT-RNN
We set the ODE solver to Euler with 3 folds for each integration, and we set T = 1.
A.5 CT-GRU
We set model's hyperparameters M
13

---

Under review as a conference paper at ICLR 2023
A.6 NEURALCDE
We two layer neural network approximate ODE function. We Fuler method to solve the CDE on three body to accelerate training process. We use torchdiffeq.oder odeiettaajooioiiii library and set the numerical precision 0.01 solve the CDE on other datasets.
A.7 RIM
We set the hyperparameters to the default values in the orginal paper. Key size input :64, value size input: 400. query size input 64, number of input heads: number of common heads: 4, input dropout: 0.1, common dropout: 0.1, key size common: 32, value size common: 100, query size common: 32.
Number of units and k: Spring: (8, 5), Charged: (8, 5), Three body: (6, 3), Human action: (6, 3).
A.8 TRAINING HYPERPARAMETERS
Table shows the default hyperparameters selection to train each model. For three body dataset, we find that for all models, setting batch size be can enhance model's performance.
three body irregular spring/charged
human action
minibatch size optimizer learning rate
128
regular: irregular:
Adam
le-3
using a cosine anncaling schedule with eta min=5e-5
clip grad max norm epochs
0.1
0.1
100
regular: 25, irregular: 50
50
Table 5: Training Hyperparameters
A.9 DATASET SETTINGS
A.9.1 THREE BODY DATASET
We use Python to simulate the motion of three bodies. We add multiplication noise from uniform distribution ((0.995, 1.005). We generate 50k training samples and 5k test samples. Three celestial bodies all samples have a fixed initial position and each pair has the sample distance. We randomly initialize the velocity so that in most samples, all three bodies have strong interactions and it is also possible that only two celestial bodies have a strong interaction and the rest one moves almost in a straight line. The dataset contains the locations of three bodies in three dimensional space, so the input size equals to 9. All samples have the length of 8. For partially observed dataset, all samples have the length of and the locations at the last timestamp are always observed. We set batch size equals to and update models after 128 forward call. We use the historical motion three bodies to predict 3 subsequent motion. We grid search the hidden size in [512, 1024, 2048] and use the mean value of mean squared error (MSE) of the last 5 epochs on the test as the model accuracy.
A.9.2 SPRING/CHARGED
We follow the experiment setting in Kipf et al. (2018). We generate 40k training samples and 10k test samples. We test model's noise resistance ability on noisy spring dataset. The noise level can be see from Figure 4 We set the number of particles to be 5. The input contains current location and velocity of each particle in two dimension, so the input size is 20. All samples have the length of 49/19 (spring regular/irregular) and 24/9 (charged regular/irreuular) Features at the first and last timestamp are always observed. We set batch size equals to 128. The task is to prediction whether there are springs connecting two particles. We grid search the hidden size in the range 128, 256, 512 for DNS and RIM and in the range of 128, 256, 512, 1024 for CTGRU and NeuralCDE. We use the mean accuracy of the last epochs on test set as the model accuracy (carly stop if the mean accuracy of the current epoch is 5% (regular) or 2% (irregular) less than the last epoch).
14

---

Under review as a conference paper at ICLR 2023
Location Feature Distrubution
Noise Distrubution
Count
Count
-0.2
0.2
Velocity Feature Distrubution
Noise Distrubution
Count
Count
-0.10
-0.05
0.00
0.05
0.10
Figure noise level
A.9.3 HUMAN ACTION
Human action dataset contains three types of human actions. There are 99 videos for hand clapping, 100 videos for hand waving and 100 videos for jogging. Videos have the length of 15 seconds in average and all videos were taken over homogeneous backgrounds with static camera. We downsample the resolution each frame to 224 224 pixels. We use mean and std equals to 0.5 to normalize images and use Resnet18 8 pretained on ImageNet (He al. (016) as feature extractors. For regular human action dataset, we set batch size equals to 4. Because the classification task is relatively easy, froze the convolutional layer parameters in Resnet18 8 and set the output size of the fully connected layer after the convolutional layer to 64. Models need to use clustered image features for action recognition. For irregular human action dataset, each video has the length of 36 50 frames and we set batch size equals to 1. We fine tune Resnet 8 and also set the dimension of image feature vector to 64. We grid search the hidden size in [32, 64] and use the mean accuracy of the last 3 epochs on test set as the model accuracy.
B CROSS VALIDATION
Table 6: Link prediction. Spring dataset with cross-validation.
Accuracy (%)
Model
Regular
Trregular
CT-RNN 256
74.583 + 2.421
53.262 + 1.013
CT-GRU 1024
90.362 0.175
89.791 = 0.133
NeuralCDE 256
91.076 + 0.121
90.521 = 0.107
RIM 512
87.450 0.083
DNS 512
94.755 + 0.584
93.695 + 2.142
We use 5-fold cross- cross-validation, leaving the test set unseen during the training process. We early stop the training process if the accuracy the training of the current epoch is 3% less than the previous best epoch. Results are summarized in Table with the mean accuracy and the corresponding standard deviation. We can readily that DNS outperforms all the other methods by large margin.
15

---

Under review as a conference paper at ICLR 2023
C ABLATION STUDY
Since our method merely incorporates an extra meta-system and a control encoder for modeling the interaction compared to standard NeuralCDE, we conduct experiments under different settings to see how different encoders and hidden state dimension can contribute to improve NeuralCDE.
To ensure the fairness, we cast a 2-layer MLP with different output sizes (2 and 16 times of input size) as in DNS to obtain the feature of controls, with varying sizes of the hidden state (128, 256 and 512). Results are summarized in Table 7 Note "DNS 8 512" incorporates MLPs, which much less than the "MLP (output=16x inpiu size)" in terms of parameter size. We see that with an extra control encoder and varying hidden dimensions, there no obvious performance difference among these settings. However, once the interaction meta-system is imposed, DNS can achieve quite significant performance gain. This turn shows the necessity the proposed meta-system for explicitly modeling the evolving interactions.
Table 7: Ablation study of DNS
Control
Accuracy
No encoding + 128
90.0209
No encoding + 256
91.0624
No encoding + 512
91.5740
M(Potuuuut=2xin size) 128
87,0086
MLPPoutpuu=22iimpu size) 256
90.8679
ML_P(output=2>xinput size) 512
91.5120
Ao.Pouuuputt1 size) + 128
91,1651
Mi.Pioupput:66xinnu size) 256
91.0800
MLP(oouput=16xxinpu size) 512
90.7029
DNS 8 512 (8 (S>MLP(ouuut=>2iinpu size)) + 512
95.38
16


Reviews:
[{"review_id": "vp9LnBjeQzs", "reviewer": "Reviewer_ateH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper solves the problem of decoupling dynamical complex systems in a data-driven manner. The authors propose an improved version of the projected differential equations. The authors conduct experiments on various important problems with synthetic or real-world datasets. The results show that the proposed DNS can achieve the best performance.", "summary_of_the_review": "The experimental part requires a lot of improvements, although the idea makes sense.", "strength_and_weaknesses": "Strong points.\n1. The proposed method and the basic idea make sense to me. The design of the meta-system is novel.\n2. The method is described well, and the paper presentation is nice.\n3. The results show the best performance of the proposed DNS.\n\nWeak points.\n1. The experimental settings require improvement. There is no sufficient hyper-parameter study of the baselines. Thus, the comparisons among the models may not be fair, which leads to unreliable conclusions.\n2. There is no study about the complexity. Compared with other baselines, which also work in a data-driven manner, the complexity of the proposed method is not clear. It seems the parameter size is far larger compared with baselines, and thus the computation time is a concern.\n3. The performance improvement of the proposed method seems to be not steady. From the results, we can observe that the performance is very sensitive to the parameter size (influenced by the hyperparameter). Therefore, it is also concerned about the generalization ability of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hWGO6zRSnn8", "reviewer": "Reviewer_koo9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at modeling complex dynamical systems into smaller subsystems. It does so by combining neural controlled dynamics(Kidger et al.) with self-attention (Vaswani et al.). Empirical evaluations are performed in multiple regression and link prediction environments from 3-body gravitation, springs, charged particles and human actions.", "summary_of_the_review": "I like the goal of this paper and believe it has many practical applications. At the same time, I feel that, at the moment, the methodology and experimental results do not support the novelty and quality of the proposed approach.", "strength_and_weaknesses": "- Strengths\n    - The problem of modeling modular dynamical systems at irregular time intervals is compelling.\n    - The paper does a good job introducing neural ODEs, neural controlled dynamics and self-attention.\n    - The paper evaluates in a wide diversity of scenarios, capturing different applications of the proposed approach.\n- Weaknesses\n    - Experimental results are not strong and some better results are missing from the table of comparisons. In particular, at least two methods achieve much better results in the charged and springs datasets. The paper obtains 68% link prediction accuracy in charged and 95.4% on springs. However, AFAIK, this is on the same datasets that Neural Relational Inference with Fast Modular Meta-learning (NeurIPS '19) obtained 88.4% and 99.9%, respectively. Furthermore, the paper where these datasets come from (Neural Relational Inference for Interacting Subsystems, ICML '18) is not shown in the table of comparisons, and that also obtains significantly better results: 82.1% and 99.9%.\n    - The proposed architecture is missing justification, in the sense that it is proposed as something we can do, but the reason why the added complexity is necessary is missing. Better ablations in the experimental section, improved explanations in the method section and a better interplay between the two sections would greatly help in this direction.\n    - Although good for many other conferences, the level of novelty (mainly combining attention-based mechanisms with neural controlled dynamics), is IMO not enough for the ICLR bar.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hLkx-nxYmW_", "reviewer": "Reviewer_gG1Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new neural ODE-based framework to model irregularly-sampled sequential data. The approach is based on neural control differential equations, and on the observation that attention models can be viewed as integrating a differential equation path. The authors evaluate their approach on a few toy examples, and it is compared to several other baselines (ncde included). The reported results show certain benefits to using the proposed method.\n", "summary_of_the_review": "This paper starts with an intuitive requirement: one wants to model complex systems using several simple and independent sub-systems. This motivates from a higher-level the method design, which turns out to be a coupled system of differential equations, with another equation dealing with interactions. While the intuition and modeling is generally reasonable, the evaluation is lacking in terms of chosen tasks, and testing protocol. I believe the paper could be significantly stronger if the authors can improve on these points.\n", "strength_and_weaknesses": "The main strength of this paper is the incorporation of two seemingly unrelated concepts: controlled differential equations and attention maps into a unified framework that models irregularly sampled time series data and their interactions. While that involved math is somewhat challenging, the authors did a good job in laying out the dependencies and relations between the various equations that appear in Sec. 3.\n\nThe main weakness of this paper is the evaluation of the proposed approach. First, the tasks chosen for evaluation seem somewhat arbitrary, and not necessarily related to the underlying assumptions. For instance, the motion of three bodies is independent as long as no interactions occur. However, the main motivation beind the coupled PDEs introduced in Sec. 3 was to model several highly independent sub-systems. In addition, the authors state that the three body problem was simulated with many interactions, so the amount of independent motion is further reduced. Second, the description of the evaluation protocol and reported results are somewhat alarming. Was there a validation set? The text suggests there was no validation set. How was grid search performed then? By reporting the best results on the (5 last epochs of) test set? If so, it might be that the results are biased towards overfitting the test sets. Moreover, the text suggest that only a single run was performed. Given that the differences in error measures are not significant, I would like to see a more extensive evaluation where testing is performed 5 or more times with different seeds, and average error metrics and standard deviations are reported. Also, are the results reported in Tabs. 1-4 reported elsewhere (for the baselines)? If not, how did achieve these results? training and testing the various methods? Was there an extensive grid search performed for the other methods? If so, how did you choose the parameter ranges? As a note, your framework uses more supervision information in comparison to other works, as you use e.g., 3 sub-systems in the three body example. Given the similarity between the proposed method and NCDE, I would like to see an extensive ablation study between these approaches. The graphs in Fig. 2 do not show any advantage to the proposed method in terms of gradient magnitudes. The issue around gradient magnitude is specifically related to exploding and vanishing gradients. Fig. 2 shows neither, and in particular, there is not even an order of magnitude difference between red and blue curves.\n\nThe second weakness is related to the modeling. Specifically, the text is dense and hard to follow at times. No proper motivation or description was provided regarding Eq. (11). Why model $dz_i(t)$ in this way? The interaction differential equation was better motivated using attention mechanisms, but still I do not necessarily see why this particular model satisfies the underlying assumptions. For instance, the interaction DE does not encourage the sub-systems to be independent. How $g$ is modeled, and what is its purpose? What is the loss function? What do you do with $z, A$ to get the output?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OcEKjM-DRwg", "reviewer": "Reviewer_gYdf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the problem of modeling data coming from a complex system with an underlying continuous dynamics. The key assumption/claim of the paper is that such a complex system is best modeled as an interaction between multiple coupled sub-systems (that has its own dynamics too). Using ideas from differential equation literature and modern attention mechanism, the paper comes up with a compelling approach towards modeling such systems. Advantages of the approach are the clear in the experimental results where the approach is compared against a variety of baselines.", "summary_of_the_review": "This is a fantastic paper and likely has a wider real world impact capability especially when modeling real world man-made physical systems which are often modular. It would be great if the paper could discuss its own limitations more carefully (especially neural DiffEq can be hard to train with increasing dimensions), any effect of diffeq solver vs Euler as used here, GPUs vs CPUs, etc. W.r.t video classification, event camera data might be ideal demonstration for irregular data. For a lot of real world applications the modular structure is partially known as well and incorporating that information would be fairly impactful future work. Similarly a good test of decoupling could be if the components could be transferred to a similar but different system.", "strength_and_weaknesses": "**Strengths**\n\nThe problem of simplifying a complex system into appropriate simpler sub-systems is one of the most important things scientists and engineers do in their daily life. Figuring out ways to automate this process directly from data can be extremely impactful. The paper very clearly formalizes the problems and identifies connections and bridges to theory of differential equations as well as continuous time interpretations of extremely successful attention mechanisms. Insights from these connections result in a very natural way of decoupling the sub-systems as well as the meta-interaction system. Results are validated on a variety of problems including human-action recognition with quite a few relevant baselines which are all appropriately modeling the task with an underlying continuous dynamics. \n\n**Weaknesses**\n\nThe paper doesn't mention this but training neural diffeq approaches can be quite inefficient, especially on modern heterogenous architectures.\nWould be great to have limitations explicated somewhere.\nThere are a lot of typos in the paper. There are also quite a few w.r.t `\\citep` vs `\\citet` issues too. Reading with fresh eyes would help fix a lot of these issues.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject