Gated Inference Network: Inferencing and Learning State-Space Models
Authors: Hamidreza Hashempoor
Wan Choi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Time Series
Recurrent Networks
Gaussian Process

Abstract: State-space models (SSMs) perform predictions by learning the underlying dynamics of observed sequence. We propose a new SSM in both high and low dimensional observation space, which utilizes Bayesian filtering-smoothing to model system’s dynamics more accurately than RNN-based SSMs and can be learned in an end-to-end manner. The designed architecture, which we call the Gated Inference Network (GIN), is able to integrate the uncertainty estimates and learn the complicated dynamics of the system that enables us to perform estimation and imputation tasks in both data presence and absence. The proposed model uses the GRU cells into its structure to complete the data flow, while avoids expensive computations and potentially unstable matrix inversions. The GIN is able to deal with any time-series data and gives us a strong robustness to handle the observational noise. In the numerical experiments, we show that the GIN reduces the
uncertainty of estimates and outperforms its counterparts , LSTMs, GRUs and variational approaches.

Under review as a conference paper at ICLR 2023
GATED INFERENCE NETWORK: INFERENCING AND LEARNING STATE-SPACE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
State-space models (SSMs) perform predictions by learning the underlying dy- namics of observed sequence. We propose a new SSM approach in both high and low dimensional observation space, which utilizes Bayesian Antering-smoothin to model system's 's dynamics more accurately than RNN-based SSMs and can be learned in an end-to-end manner. The designed architecture, which we call the Gated Inference Network (GIN), is able to integrate the uncertainty estimates and Iearn the complicated dynamics of the system that enables us to perform estimation and imputation tasks in both data presence and absence. The proposed model uses the GRU cells into its structure to complete the data flow, while avoids expensive computations and potentially unstable matrix inversions. The GIN is able deal with any time-series data and gives us a strong robustness to handle the obser- vational noise. In the numerical experiments, we show that the GIN reduces the uncertainty of estimates and outperforms its counterparts. LSTMs, GRUs and variational approaches.
INTRODUCTION
State estimation and inference in the states in dynamical systems is one of the most interesting problems that has lots of application in signal processing and time series Rauch et al. (1965). In some cases, learning state space is a very complicated task due the relatively high dimension of observations and measurements, which only provides the partial information about the states. Noise another significant issue in this scenario, where is more likely to obtain noisy observation. Time series prediction and estimating the next scene, c.g, the state prediction or next observation prediction, another substantial application that again requires the inference within the states which comes from the observations. Classical memory networks such as LSTMs (Hochreiter & Schmidhuber 1997). GRUs (Cho et al., 2014) and simple RNNs like (Wilson & Finkel] and (Yadaiah Sowmya 2006) fail to give some intuition about the uncertainties and dynamics. A group of approaches perform the Kalman Filtering (KF) the latent state which usually requires a deep encoder for feature extraction. Krishnan et al.) (017)). Ghalamzan al. (2021) and Hashemmour et al.) (2020) belong to these group of works. However the mentioned solutions have some restrictions, where they are not able deal with high dimensional non-linear systems and the classic KF approach is computationally expensive, e.g matrix inversion issue. Likewise, indirect optimization of an objective fuction by using variational inference, like the work of Kingma & Welling (2013), increases the complexity of the model. Moreover, in the variational inference approaches that usually implemented in the context of variational auto encoders for dimension reduction, they do not have access the loss directly and have to minimize its lower bound instead, which reduce the ability of learning dynamics and affect the performance of the model. KalmanNet Revach et al. (202T) and Ruhe & Forré (2021) use GRU in their structure for the state update. However, they are only able to deal with low-dimensional state space and cannot handle complex high dimensional inputs because of directly using classic Bayesian equations and matrix inversion issue. Moreover, their structure require the full, or at least partial, dynamic information.
The mentioned restrictions for KF and its variants and variational models in addition the necessity of having a metric to measure the uncertainty, motivate us to introduce the GIN, an end to end structure with dynamics learning ability using Bayesian properties for contributions of GIN are: (i) modeling high-low dimensional sequences: we show the eligibility of the GIN to infer both cases by simple adjustment in the observation transferring functions in the
The

---

Under review as a conference paper at ICLR 2023
(a) Observation
(b) LGSSM(smooth)
(c) GIN(fiter)
(d) GIN(smooth)
Figure 1: Inferred 5k length trajectories for Lorenz attractor.
proposed structure, where we conduct three experiments of high dimensional non-linear dynamics and two experiments of low dimensional chaotic observation. (ii) Learning/using dynamics: the ability of learning the dynamics(in the lack of them) and utilizing available dynamics(in the presence of them) alongside modeling high-low dimensional observations makes the GIN applicable to wide range of applications. To attain more accurate inference of observed dynamical system, we apply GRU cells that increases the modeling capability of the Kalman fltering-smoothing. By conduction an ablation study of the GIN being replaced by a linearized Gaussian state transition, we show the GIN is able for better learning state space representation with disentangled dynamics features. (iii) Direct optimization: We show that the posterior and smoothing inference distribution of the state-space model is tractable while dynamics and parameters are estimated by neural networks. Despite variational approaches, this allows us to use recursive Bayesian updates for direct likelihood maximization. (iv) Noise robustness: verified by the numerical results, inferencing for highly distorted sequences feasible with the GIN. (v) Missing data imputation: by using Bayesian properties, the GIN decides whether keep the previous information in the memory cell update them by the obtained observation. Experimental results show the out-performance of the GIN over the SOTA studies in the imputation task.
2 RELATED WORKS
To deal with complex sensory inputs, some approaches integrate deep auto encoders into their architecture. Among these works, Embed to Control (E2C) (Watter et al. 2015) uses a deep encoder to obtain the observation and a variational inference about the states. However, these methods not able to deal with missing data problem and imputation task since they do not rely on memory cells and are not recurrent. Another group of works like BackpropKF (Haarnoja et al. (016)) and RKN (Becker et al. 2019) apply CNNs for dimension-reduution and output both the uncertainty vector and observation, where they move away from variational inference and borrow Bayesian properties for the inference. However, these methods cannot handle the cases with the available knowledge of the dynamics and impose restrictive assumptions over covariance matrices, while the GIN provides principled way for using the available partial dynamics information and release any assumption over covariance. Toward learning state space (system identification) a group of works like Wang et al. (2007), Ko & Fox (2011) and Frigola et al. (2013) propose algorithms to learn GPSSMs based on maximum likelihood estimation with the iterative EM algorithm. |Figola et al. (2013) obtain sample trajectories from the smoothing distribution, then conditioned on this trajectory they conduct M step for the model's parameters. Switching linear dynamics systems (SLDS) (Ghhhramani & Hinton 2000), use additional latent variables switch among different linear dynamics, where the approximate inference algorithms can be utilized to model switching linearity for reducing approximation errors .however, this approach is not flexible general non-linear dynamical systems because the switch transition model is assumed independent of states and observations. To address this problem, Linderman et al. (2017) performs SLDS method through augmentation with a bolya-gamma-distrbbee approach employs Gibbs sampling for inferring the parameters and therefore is not scalable to large datasets. Auto-regressive Hidden Markov Models (ARHMM) explain time series structures by defining a mapping from past observations to the current observation. (Salinas et al. 2020) is a ARHMM approach, in which target values are used as inputs directly. However, this dependency of the model on the targets makes the model more vulnerable to noise. This issue addressed in DSSM (Rangapuram al. 2018). another ARHMM approach, where the target values are only incorporated through the likelihood term. Other group of works consider 3.M-based
variable and a stick-breaking process, however, this
caaattionnl-iffeennce like Structured Inference Networks (SIN) (Krishnan et al., (017). where it utilizes RNN to update the state. Kalman Variational Autoencoder (K (KV)E) (Fraccaro et al.) (017) and Extended KVAE (EK VAE) (Klushyn et al. 2021) use the original KF equations and apply both filtering and smoothing.
2

---

Under review as a conference paper at ICLR 2023
However, these EM-based variational inference methods are not able to estimate the states directly because of optimizing the lower bound of likelihood. Extra complexity is another issue with these approaches, while they are addressed by the proposed structure and direct end-to-end optimization in the GIN. We compare the GIN with these approaches in the experiment section and provide an empirical complexity analysis in appendix A.8.2 We provide detailed discussion of recent related work in appendix A.8.3
3 GATED INFERENCE NETWORK FOR SYSTEM IDENTIFICATION
In the context of System Identification (SI) the GIN is similar to a Hammerstein- Wiener (HW) model (Schoukens & Tiels) 2017) (Gilabert al. 2005). in the sense that it estimates the system parameters directly from the observations, which is in the figure 2 e(.)) and d(.) are implemented with non-linear functions, e.g. auto encoders-MIPPS. Transition block in figure represents the dynamics of the system that allows for the inference using the Gaussian state space
equations. However unlike a HW model, we employ non-linear GRU cells in the transition block that calculate the Kalman Gain (KG) and smoothing gain (SG) in an appropriate manner by circumventing the computational complexity, i.e matrix inversion issues. GRU cells empower the whole system by applying non-linearity the linearized Gaussian state space models (LGSSMs). Numerical results indicate that by the proposed structure, having a good inference for even the complex non-line non-linear systems with high dimensional observations is feasible. To achieve this, we assume the state fits into Gaussian state space models (GSSMs), which are commonly used to model sequences of vectors.
In most cases, the dynamics of the system might not be available or hard to obtain; while the process noise and observation noise are unknown (our first three experiments). Ac- cordingly, we construct GIN to learn unknown variables from data in end to end fashion, then we utilize the constructed KG and SG during inference time to obtain the filtered- smoothed states. The proposed architecture is depicted in figure In the presence of dynam- ics (our last two experiments), auto-encoder and Dynamics Network figure replaced by MLP to model the observation noise.
(
E,)
of
e()
Transition Block
d(-)
Figure 2: The GIN as a HW model for system identi- fication. By appropriate structure selection for e(.) and d(.), the GIN can handle high-low dimensional observations. The proposed architectures for each case are depicted separately with further details in appendix figures IO and [T] The relation between the internal variables, W. and x,, is simulated by the transition block.
4 PARAMETERIZATION
In this section we show the parameterization of the infer- ence model. Firstly, we refer the readers to the summary of Kalman shteringssmothingg background for the com- pleteness in appendix [A.2 In the rest of the paper we use the following notations, where the original observa- tions are 01.7, the transferred observations are W1.7 and R1.7 are diagonal matrices with r1:7 diagonal elements that correspond to the covariance of transferred observa- tion noise. XI:T corresponds the states and Qu. are diagonal matrices with qı.r diagonal elements which are the covariance of state process noise. (x, Σ, ) are the mean vector and covariance matrix of the prior state at time step i, i.e. p(x, [W:t-1), and (xx-. Σe are the mean vector and covariance matrix of the posterior state
syn-z
49%
Te-p
Xi+t
Figure 3: Graphical model for high dimen- sional observations. dyn, is the model's dynamics at time
at time step t, i.c. p(x; W1s). We define the transition matrices FI.T and emission matrices H as the dy- namics of the model, where lack of dynamics in the first three experiments means that (Fi-r, HI.r) are not know and are going to be trained model is in figure 3) while the presence of dynamics in our last two experiments means that (F 1:7; H are known(graphica models are in figures and The dynamics (Fir,, H.r) and noise matrices
3

---

Under review as a conference paper at ICLR 2023
( Pir)
Smoothing Step
Filtering Step
Prediction Step
Decoder
0.r
Encoder
oja
(Yin War)
Figure 4: The high level structure of the GIN for high dimensional observation in the lack of dynamics, while for low dimensional cases auto-encoder replaced by MLPs and dynamics are directly used. The transferred observation we and its uncertainty are obtained from the encoder(MLLPs) In each time step, the last posterior mean x is fed to the Dynamic Network to compute F, and H, In the Prediction Step the next priors (x, Σ,, are obtained by using new dynamics and the last posterions. In the filtering step, by using the priors (x; D, and the observation (w,, the next posteriors (x," ΣT) are obtained. Applying smoothing operation over the obtained posteriors (x," E, is feasible in the smoothing step. Finally, the decoder(MLP)) is utilized to produce o, which can be the high-low dimensional noise free estimates.
(R.- Q1:)) construct the system parameters 71.T (F7:, HI.T, QI:T R1.T). Given original observations 01.7 and transferred observations Wi.r, we want to find good estimate of the latent states X1.-- achieve this, we want infer the marginal distributions p(x;) (w::) for the online inference approach or filtering; and p(x,/w (,,T) for the full inference approach smoothing.
We introduce an advantageous prediction parameterization as Pr, (x, (2, -1--1. Wia-1)
((F Qt), where X Prc- (Xt- [ W(x Σ; Then, Py, (x; w1+-1)==
(F F,E, F + Q.) -V Σ,) is obtained by marginalizing out x and the Gaus- sianity of Pr. (x, Wi=-1) results from the Gaussianity of prediction parameterization. By having
Py, (x//Wi:e (itt/i and observing w., filtering parameterization is introduced as:
Pr. (x, (x/l N + H,x]), E, K, H. + +R,KK? ~V Σt) (1)
where K, is KG. After observing all transferred observations WI.Y, one can backward induction and propagate to the previous states using the chain rule. This procedure, known smoothing, can be parameterized as:
Pry (x/[W:r) .V J. F++1 Σx + Je J 1
(2)
where Je is SG and we use short handed notation N (x Egrr instead of (2) These parameter- izations give us some insight to 1-illustrate a tractable way to construct p, (x (w) and accordingly
obtain the posterior and smoothened states, based which o is constructed and 2- appropriately
modeling and KG(SG) with neural networks.
To construct the KG and SG networks, we have find appropriate inputs containing related informa- tion to attain the KG and SG. In (I) and (2). KG and SG are given by (3) and (4), respectively.
K, Σ, H H H R.
[E R.)
(3)
J, = E 4+1
F E F - Q+1] = Σ FL Din & 141
(4)
3 is proportional the prior covariance at time t, Σ, and the observation noise matrix, R4, while 14 is proportional to prior covariance matrix at time ++ Σ,+1- Our encoder(MLP) directly maps the observation noise matrix from the observation space, but the state covariance is recursive function of previous states. Consequently, consider GRUKG and GRUSG which are networks including
4

---

Under review as a conference paper at ICLR 2023
GRU that map [ ), R.] and f(E++1) to the KG and SG, respectively. GRU considers R a diagonal matrix with It elements in figure 4 as a part of its input to incorporate the effects of observation noise. In the
of high dimensional state space, due to the high dimension of E and S141 f is a convolutional layer with pooling to extract the valuable information of the covariance matrix that reduces its size, while for the low dimension of E, and Σ is the identity function.
Learning The Process Noise. In the filtering procedure, the process noise in time is obtained as
Q: = = - F
(5)
where D, F, and Σ prior state covariance, transition matrix and posterior state covariance time t. is shown that Q can be written as function of F, as
- - F F = - F. K -
R
K F
(6)
while the derivations are rather lengthy, therefore, we refer to the appendix materials A.3 From 32 the relation of the process noise with the transition matrix indicates that F, can possess the effects of Q2 if we learn it in an appropriate manner. (QQ notation means that the learned transition matrix F, comprises the effects of Qù while for the simplicity use F, abbreviation. Therefore, it is possible to rewrite as
Σ =
Another way to have a a meaningful inference about the process noise matrix is to obtain from (30 recursive function of x and Q1-1- Intuitively, g function in (30) that we call it Q Network. can be implemented by a memory cell, e.g., a GRU cell, to keep the past status of Q ,however, increases the complexity of the model. Equivalently, as stated in (32). Both these solutions can be utilized when the dynamics are known, i.e. we cannot learn the effects of Qr jointly with F, F, is not trainable.
obtain Q, directly from x with MLP
Prediction Step. Similar to the model based Kalman Filter, by using dynamics of the system and transition, the next priors obtained from the current posterior by
FIx
Σ, =
(8)
where F, is the learned transition matrix comprises the effects of the process noise from previous section. By which, is feasible to predict state mean and the state covariance matrix.
Filtering Step. To obtain the next posteriors based on the new observation (w,, rt), i.e. the output of e(.) in 2 we have to use the obtained KG matrix from GRU network and learned emission matrix H, to complete updating the state mean vector and state covariance matrix. This procedure is given by
S, = H. D, H R.,
M. = CRROKC(((E) K; E. M.M.' = + K. - H = + KK.SS
(9) (10)
In addition to avoiding the matrix inversion that arises in the computation Kalman gain and applying non-linearity to handle more complex dynamics, the architecture of KG network, KC can reduce the dimension of the input its corresponding GRU cell, and thus reduces the total amount of parameters quadratically. Additionally, positive vector and Cholesky factor consideration, M.M, p guarantee the positive definiteness of the resulted covariance matrices.
Smoothing Step. After obtaining filtered states (x+ 1 in filtering step. we employ smoothing properties of Bayesian get smoothed version of the states. In this stage, we use JI.T matrices obtained from GR network, learned transition matrices Fir and filtered states (xf E
The procedure in each smoothing step is given by:
J. = E.FT N; = SRU GRUSSG(((ETT)))
(11)
XIpT x
- F J. Enr= = Σt +
F7 F
(12)
where the first smoothing state is the last filtering state, i.e. (XTTT, ETiT) (xt, Σt)
5

---

Under review as a conference paper at ICLR 2023
Learning Dynamics. We can model the dynamics in each time step function of the transferred observations WI:--1- However, conditioning on the noisy observations can distort the procedure of learning the dynamiic. Instead, with considerable lower noise distortion to increase system's 's noise robustness, where we generate time correlated noise in our experiments show this robustness(sez (see appendix (A.5). In other words, original transition and emission equations, X fxx + and W, h(x;) It, are modeled as Fx(x 1)x-1 + q. and w, H,(x{_1)xx )x+x + rr. We learn K state transition and emission matrices F* and H and combine each one with the state dependent coefficient a* (x-_))
the state x in GSSM that includes the history of the system
K
K E a (x
Fx Σ (x
F
H(
(13)
where a separated neural network with softmax output is utilized to learn a (x{-1 that we call it Dynamics Network. This formulation enables us follow Bayesian methodology. Despite classic LGSSMs that are not able to learn the dynamics, e.g. EKF and UKF, the trainable dynamics in the GIN are function of the states. For the notation simplicity, we have used (F,, H,) instead of (F, (F, (x in the paper.
5
FITTING
For the state estimation task, by implementing p(W1:T|or (1.:), p(xx/Wi:7) and p(sr(x,) with encoder, smoothing parameterization and decoder, maximise the log-likelihood of output p(s: (0:r)
[ p(s xx xx.)(xx./w/ htpppx//w.7// wi:r)p(( (w../ow..r (dx,::) ,where is the estimated state, i.e. equal of in figure For the image imputation task, in addition to the state likelihood, we add the reconstruction seeudo-likelhood for inferring images by using Bernoulli distributions p(i, i.c. the decoder in figure maps both state S, and image i, of = [ sr]. Further details of distribution assumptions and hyper parameter optimization can be found in the appendix A.4 and A.8 After training phase,
forecasting desired number of time steps is applicable by plugging the new value x, = Fix;-1 recursively in the model, and so
Likelihood for Inferring States. Consider the ground truth sequence is defined as Sı.T. We determine the log likelihood of the states
T
[(S1.)
log NV decmean (X+/T). deC (E(TT)
(14)
where the deCmean (.) and dec (.) determines those parts the decoder that are used to obtain the state mean and state variance, respectively. We use Wishart distribution prior for our estimated covariance matrix, which pushes the estimated covariance toward a scale of identity matrix and the scale is hyper parameter. Such prior prevents getting high ligelioel due to the high uncertainty.
Likelihood for inferring images. For the imputation task, consider the ground truth as the sequence of images and their corresponding states, which are defined as (s1.7, (n:r] and the dimension of i, is D.- We determine the log likelihood:
((otT) L(S1.T
ΣΣ (k) 1o) (decs (xec)
-1( (lo()
dec (xi/T)).
(15)
dxc)(xx) defines the corresponding part of the decoder that maps the k-th pixel of i, image and À constant determines the importance of the reconstruction. The first term in RHS is obtained from (14
and we abbreviate the second term as L(i.r).
6 EVALUATION AND EXPERIMENTS
We divide our experiments into two parts, first the tasks in which the observation space is high dimensional like sequence of images, and second the applications that the observation is in low dimension by itself so there is no need to include encoder for dimension reduction. The training algorithms of both cases added in the appendix section A.11

---

Under review as a conference paper at ICLR 2023
Table 2: Pendulum state estimation. By consider n = 3m, intuitively the last part of the state dedicated to the acceleration information causing more lieklihood.
Table I: Double pendulum state estimation. (#1 23) refers to the position of the first joint, while (22, Taa) is for the second joint.
Model
S
-g
Log Likelihood
SFlt
SF
Lag Hellhhoo
LSTM (units-50)
0.163
0.171
0,148
0.167
3501 0,706
LSTM tuniss-
0.152
0.565
LSTM (units=25)
0,992
0.994
5,891 + 0.151 5,751 0.215
LSTM (units=100)
0,089
01877
(units=50)
0.179
0.369 3,976
3,886
GRU (umiss=100) 100)
0.164
0.156
0.162
0.145
0.231
GRU (unis=30)
0.095
0.089
5.986 0.168
(units-100) 100)
5,698
KVAF (m=2mm
0.193
0.188
0.178
0.149
3,709
0.101
KVAE (n:3mm)
0.171
0.159
0.151
0.162
3,501
1: 0.116
KVAF(nu2m))
5.786 0.088
KYAE (n=3m)
0.088
0.083
5.858 0.113
KKN (xu=2m))
0.134
0.129
0.139
0.118
4.176
0.294
RKN (n=2m)
0.078
0.075
6.161 0.23
LGSSMMaa (me3mm)
0.125
0.119
0,121
0.107
0,127 4,231
4.192
LGGS5 (1033))))
0,109
0.111
0.104
0.101
0.154
LGSSM
LGSSManooh
0.071
0.009
6.242 d: 0.109
(n=2m)
0.115
0.109
0.119
0.109
4.224 1.1005
GIN (143m))
0.091
0.098
0.089
4.329
0.151
GiNnu((n=2m)
0.075
0.07
6.192 01.299
GIN smonn (n==2m))
4,308
0.123
(INma((R220)
0.967
0.066
6.315 à: 0.220
GiNeroon (r23511)
0.079
0.083
0.085
0.077
4,477
0.168
GIN GiNanoth
0.065
0.067
6.292 J: 0.113
(1
6.1 HIGH DIMENSIONAL OBSERVATION WITH LACK OF DYNAMICS
We include three high dimensional experiments. The first two experiments are single pendulum and double pendulum, where the dynamics of the latter one is more complicated. The last experiment is visual odometry task. Intuitive python code in appendix A.12.I
6.1.1 SINGLE PENDULUM AND DOUBLE PENDULUM
The inputs of the encoder the images with size of 24 24. The angular velocity disturbed by transition noise which follows Normal distribution with o = 0.1 as its standard deviation at each step. In the pendulum experiment, we perform the sntering-smoothingg by the GIN where the observation is distorted with high observation noise. Furthermore, we compare GIN with LGSSM, where the GRU cells are omitted from the GIN structure and classic -eeoriingmmo equations are used, instead. The log-likelln likelihood and squared error (SE) of positions for single and double pendulum given in table 2 and I respectively. Generated samples from trained smooth-file distributions are in appendix figures [6 33
By randomly deleting the half of images from the generated sequences, we conduct the image imputation task to model by predicting those missing parts, while the missingness applied to train and test are not same, but random. The results are in table and The GIN outperforms all the other models, although the variational inference models have
complex structures in KAVE
Table 3: Image imputation task for the different models. Models contain boolean masks determining the available and missed images. For uninformed masks, a black image is considered as the input of the cell whenever the image missed, which requires the model to infer the accurate dynamics for the generation. We conduct uninformed experiment as well.
Model
Log Likelihood
E2C
-95,539 1.754
SIN
-101,268 0.567
KVAE (informed smooth)
-14.217 0.236
KVAE (unformed smooth)
-39.260 + 5.399
EKVAE (informed smooth)
-12,897 0.524
EKVAE (unformed smooth)
-29.246 + 3.328
RKN (informed)
-12.782 + 0.0160
Figure 5: Pendulum image imputation. Each figure, beginning from up to down, indicates the ground truth, uninformed observation and the imputation results of the GIN(smoothed). Missingness is ap- plied randomly for train and test.
RKN (uninformed)
-12.788 = 0.0142
L.GSSMinformed smooth)
-12.695 + 0.048
GIN (informed smooth)
-12.215 0.027
GIN (unformed smooth)
-12.246 0.029
7

---

Under review as a conference paper at ICLR 2023
Table 4: Image imputation for double pendulum.
Model
Log Likelihond
KVAE (informed smooth)
-13.917 0.294
KVAE (unformed smooth)
-38.544 6.419
EKVAE (informed smooth)
-13.917 0.414
VKVAE (unformed smooth)
-33.548 + 4.516
RKN (informed)
-13,832 0.023
Double pendulum image imputation. Each figure, beginning from up to down, indicates the ground truth, uninformed observation and the imputation results of the GIN(smoothed).
Figure
RKN (uninformed)
-13.898 1: 0.0191
LGSSM(informed smooth)
-13.775 + 0.013
GIN (informed smooth)
-13.284 0.021
GIN (unformed smooth)
-13.351 0.019
and EKVAE. We include the results using the MSE well, illustrate that our approach is also competitive in prediction accuracy (See A.
6.1.2 VISUAL ODOMETRY OF KITTI DATASET
We also evaluate the GIN with the higher dimensional observations for the visual odometry task on the KITTI dataset Geiger et al. 2012). This dataset consists of 11 separated image sequences with their corresponding labels. In order to extract the positional features, we use a feature extractor network proposed by Zhou al. in Zhou et al. (2017). The obtained features considered the observations of the GIN, i.c. (w, r). Additionally, we compare the results with LSTM, GRU, Decp Wang al. (2017) and KVAE. The results are in table and figure where the common evaluation scheme for the KITTI dataset is exploited. The results of the KVAE degrades substantially as we have reduce the size of the transferred observation prevent the complexity of matrix inversion.
6.2 Low DIMENSIONAL OBSERVATION WITH PRESENCE OF DYNAMICS
We conduct two experiments, Lorenz attractor problem and the real world dynamics NCLT dataset, where we are aware of the dynamics. Intuitive python code in appendix A. 12.2 However, the GIN is able to deal the cases in which the dynamics experiment in the appendix, where we do not give the dynamics information of Lorenz attractor and NCLT dataset the model, that they will be learned(see figures 42
not known. To show this, conduct additional
6.2.1 LORENZ ATRRACTOR
The Lorenz system is system of ordinary differential equations that describes a non-linear dynamic system used for atmospheric convection. Due to nonlinear dynamics of this chaotic system (see [A.6). can be good evaluation for the GIN cell. We evaluate the performance of the GIN trajectory of 5k length. Each point in the generated trajectories is distorted with an ob- servation noise that follows Gaussian distribution with standard deviation = 0.5. The likelihood with Gaussian distribution calculated and maximized the training phase. The mean square error (MSE) of the test data for various number of train-
5001
(1000)
Figure 7: MSE of Lorenz attractor.
Table Comparison model performance on KITTI dataset. See
35
36
37
38
39
figrres
in A.9 for the visualization results.
Doc
Tie!"
(%(
Fig
Lg (%)
[ig(")
(%6
(%)
Tigl
(RE)
8.99
4.35
9.31
3.31
8.09
6.84
12.14
1.38
1.31
3.95
6.38
3.27
2.51
06
9.66
2.8
10.99
3.22
5.42
5.82
10.93
3.98
6.70
3.51
6.14
2.90
9.83
5.48
13.70
6.52
3.91
4.60
12.73
4.68
6.59
3.49
7,21
2.98
mann
10.53
3.77
11.63
3.55
3.96
6.12
12.58
3.65
7.55
3.28
7.05
2.75
8

---

Under review as a conference paper at ICLR 2023
odus dauilaasa
X1
(a) GIN
(b) LGSSM
(c) KVAB
Figure 8: Generated samples from smoothing distribution for the joint position (x1, x2), equivalent to [ol+, o2+) in figure 4 at 100-th time step of visual odometry experiment. The ground truth is shown with black point.
ing samples are depicted in figure 7 Hybrid is graphical GNN based model Garcia Satorras et al. (2019) and DSSM Rangapuram et al. (2018) is a version of LGSSM using LSTM cells.
Due the non-linearity of the dynamics of this system, LGSSM has to linearization and then the linearized dynamics to model the transition. The DSSM model performs better for lager amount of data (>10K) because it needs learn the dynamics. The results of the Hybrid GNN and the GIN are similar, while the results of the GIN are slightly improved. Although, the core of both models is based on the GRU cell, this enhancement may come from the structure of the GIN that learns the observation and process noises separately. Inferred trajectories are in figure 1
6.2.2 REAL WORLD DYNAMICS: MICHIGAN NCLT DATASET
To evaluate the performance of the GIN on a real world dataset, the Michigan NCLT dataset CattasetCarlevvriss Bianco et al. (2016) is utilized that encompasses a collection of navigation data gathered by segwey robot moving inside of the University of Michigan's North Campus. The states each time, x, E TR. comprise the position and the velocity in each direction and the observations. y. E I2, include noisy positions. The ultimate purpose is to localize the real position of the segway robot, while only the noisy available. We apply the GIN to find the current location of the segway robot. In this experiment, we randomly select the session 2012- 01-22 captured in cloudy situation with the length of 6.1 Km. By sampling with 1Hz and removing the unstable GPS observations, 4280 time steps achieved. For the dynamics of the system, we consider uniform motion pattern with a constant velocity (see A.7). The training procedure is completed by maximizing the likelihood with Gaussian distribution assumption. The mean squared error of each approach for the test set are mentioned in the table 6, where the GIN (73.12 12 + 2.21 MSE) outperforms other approaches.tr In summary, this experiment indicates that the GIN can generalize with good performance to a real world dataset.
GPS observations
Table 6: MSE for NCLT experiment.
Model
MSE[dB]
GIN(smooth)
18.644
Hybrid GNN
20.73 0.21
KalmanNet
22.2+:0.17
DSSM
29.54
Vanilla RNN
40.21+
LGSSM
24.38++0.17
Observation
25.47:40.08
7 CONCLUSION
The GIN, an approach for representation learning in both high and low dimensional SSMs, is introduced in this paper. The data flow is conducted by Bayesian filtering-smoothingg, while, due to the usage GRU based KG and SG network, the computational issues are tackled resulting in an efficient model with numerical stable results. In the presence of the dynamics, the GIN directly use them, otherwise it directly learns them in an end to end manner, which makes the GIN as a HW model with strong system identification abilities. Insightful representation for the uncertainty of the predictions incorporated in this approach, while outperforms its counterparts including LSTMs, GRUs and several generative models with variational inferences.

---

Under review as a conference paper at ICLR 2023
BIBLIOGRAPHY
Evan Archer, II Memming Park, Lars Buesing, John Cunningham, and Liam Paninski. Black box variational inference for state space models. arXiv preprint 1rxiv:/51..0737, 2015.
2015.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 17X1::6607.0645000 2016.
Philipp Becker, Harit Pandya, Gregor Gebhardt, Cheng Zhao, C James Taylor, and Gerhard Neumann. Recurrent kalman networks: Factorized inference in high-dimensional deep feature spaces. In International Conference on Machine Learning, pp- 544 552.. PMLR, 2019.
Nicholas Carlevaris- Bianco, Arash K Ushani, and Ryan M Eustice. University of michigan north campus long-term vision and lidar dataset. The International Journal of Robotics Research, 35(9): 1023-1035, 2016.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the propertics of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409. 1259, 2014.
Marco Fraccaro, Simon Kamronn, UIrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. arXiv preprint arXiv: 171v001710.05411, 2017.
Roger Frigola, Fredrik Lindsten, Thomas Schön, and Carl Edward Rasmussen. Bayesian inference and learning in gaussian process state-space models with particle memc. Advances neural information processing systems, 26, 2013.
Victor Garcia Satorras, Zeynep Akata, and Max Welling. Combining generative and discriminative models for hybrid inference. Advances in Neural Information Processing Systems, 32, 2019.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3354-3361. IEEE, 2012.
Zoubin Ghahramani and Geoffrey E Hinton. Variational learning for switching state-space models. Neural computation, 12(4):831-866 2000.
Amir Ghalamzan, Kiyanoush Nazari, Hamidreza Hashempour, and Fangxun Zhong, Deep-Ifd: deep robot learning from demonstrations. Software Impacts, 9:100087, 2021.
Gilabert, Gabriel Montoro, and E Bertran. On the wiener and hammerstein models for power amplifier predistortion. In 2005 Asia-Pacific Microwave Conference Proceedings, volume 2, pp. 4-pp. IEEE, 2005.
Tuomas Haarnoja, Anurag Ajay, Sergey Levine, and Pieter Abbeel. Backprop kf:: Learning discrimi- native deterministic state estimators. In Advances in neural information processing systems, pp. 4376 4384, 2016.
Hamidreza Hashempour, Kiyanoush Nazari, Fangxun Zhong, et A data-set of piereing needle through deformable objects for deep learning from demonstrations. arXiv preprint arXiv:2012.00255. 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick Van der Smagt. Deep varia- tional bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint arXiv:1605.064432 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X2v:14/2.691 69800, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114 2013.
10

---

Under review as a conference paper at ICLR 2023
Alexej Klushyn, Richard Kurle, Maximilian Soelch, Botond Cseke, and Patrick van der Smagt. Latent matters: Learning deep state-space models. Advances in Neural Information Processing Systems, 34, 2021.
Jonathan Ko and Dieter Fox. Learning gp-bayesfilters via gaussian process latent variable models. Autonomous Robots, 30(1):3 23, 2011.
Rahul Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 2017.
Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. arXiv preprint arXiv:1803.02991, 2018.
Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. In Artificial PMLR, 2017.
Intelligence and Statistics, pp.
Christian Naesseth, Scott Linderman, Rajesh Ranganath, and David Blei. Variational sequential monte carlo. In International conference on artificial intelligence and statistics, pp. 968 977. PMLR, 2018.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances neural information processing systems, 31, 2018.
Herbert E Rauch, F FTTung and Charlotte T Striebel. Maximum likelihood estimates linear dynamic systems. AIAA journal, 3(8):1445-1450 1965.
Guy Revach, Nir Shlezinger, Xiaoyong Ni, Adria Lopez Escoriza, Ruud JG van Sloun, and Yonina C Eldar. Kalmannet: Neural network aided kalman filtering for partially known dynamics. arXiv preprint arXiv:2107. 10043, 2021.
David Ruhe and Patrick Forré. Self-supervised inference in state-space models. arXiv preprint 11X1v:2107.13349, 2021.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191,0021
Maarten Schoukens and Koen Tiels. Identification of block-oriented nonlinear systems starting from linear approximations: A survey. Automatica, 85:272-292, 2017.
Niklas Wahlström, Thomas B Schön, and Mare Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXiv: 2015.
Jack M Wang, David Fleet, and Aaron Hertzmann. Gaussian process dynamical models for human motion. IEEE transactions pattern analysis and machine intelligence, (0(2):283-295 2007.
Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni. Deepvo: Towards end-to-end visual odom- etry with deep recurrent convolutional neural networks. 2017 IEEE international conference on robotics and automation (ICRA), pp. 2043-2050 IEEE, 2017.
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. arXiv preprint arXiv: 07366, 2015.
Paul J Werbos.
Backpropagation through time: what it does and how do it. Proceedings of the IEEE, 78(10):1550-1560 1990.
Robert Wilson and Leif Finkel. A neural implementation of the kalman filter. Advances in neural information processing systems, 22:2062-2070. 2009.
Narri Yadaiah and G Sowmya. Neural network based state estimation of dynamical systems. In The 2006 IEEE International Joint Conference on Neural Network Proceedings. pp. 1042-1049. IEEE, 2006.
11

---

Under review as a conference paper at ICLR 2023
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1851-1858, 2017.
A
APPENDIX
A.1
GAUSSIAN STATE SPACE MODELS
In order to model the vectors of time series w = W1:7 [W1..... Wr]. Gaussian state space models (GSSMs) are commonly applied due to their Ellering-smoothing ability. In fact, GSSMs model the first-order Markov process on the state space x = [x1...., XT), which can also include the external control input u [u.,... Ur] by multivariate normality assumption of the state
Pys (X, x u,) = (x,; F,X,-1 B,u, Q), Pr. (w. xx) = (ww;; H,Z, R)
(16)
For the cases, which are not controlled via external input, B, matrix is simply 0 matrix. By Defining Tt as parameters which explain how the state state changes during the time, it contains the information the state transition, control and emission matrices. In each step, the procedure distorted via Q and R that are process noise and observation noise, respectively. is common to initial the first state X1 (O, En), then the joint probability distribution of the GSSM
of F,, B, and H, which
is
Py (w,x)l P.y
(w/x)) (
II Pro (w, (wxxx)))ppxx))
II prr (x, (2 u.). tm2
(17)
GSSMs have substantial properties that we can utilize. Filtering and smoothing are among these properties which allow obtain the filtered and smoothed posterior based on the priors and observations. By applying classic Bayesian properties, we can have strong tool to handle the missing data in the image imputation task.
A.2 FILTERING AND SMOOTIIING PARAMETERIZATION
The idea of Kalman filter applies two iterative steps, in the former one prediction is made by the prior state information, while in the latter one an update is done based on the obtained observation. By normality assumption of known additive process and observation noise, the filter can go through the two mentioned steps. the prediction step, the filter uses the transition matrix to estimate next priors (x++1 1441) which are the estimate of the the next states without any observation.
Fx, and Σn1 FE Q, and
2
(18)
In the presence of new observation, the Kalman filter idea goes through the second step and modifies the predicted prior based on the new observation and emission matrix H that results in the next posterior (x++1) Σ+ 11+1))
K E H. HE H
+R)-
and
(19)
x -x,11* E H
HE
W Hx H +R)
x Keti(w; Hx (20)
E E 1111
H + HE
(21)
The whole observation update procedure can be considered as weighted mean between the the next prior, that comes from state update, and new observation, where this weighting a function of Q and R that has uncertainty nature.
We derive smoothing parameterization, where the key idea to use Markov property, which states that is independent of future observations Wi-1:7 long x is known. However, we not
12

---

Under review as a conference paper at ICLR 2023
aware of X,+1, but there is distribution over it. So by conditioning X;-1 and then marginalizing out it is possible to obtain X3 conditioned on Wi:r-
p(xa/ (1:T) / p(xi|xx+1, p(x//x(4t Wir))((x- (W1:))))
(22)
p(x||xe41, WaS, (
By using induction and and smoothed distribution for t + 1:
p(x++1 |W::) (x E(1(()
(23)
we calculate the filtered two-slice distribution as follows:
o(x:, X (W::) =N
-N(
x
Σ
(24)
x2+1
Fr+1E
by using Gaussian conditioning we have:
X(x11X++111 (1:) N(t + + F++1x))
J, J.
(25)
where J, = F4+ [ -1 We calculate the smoothed distribution for using the rules of iterated expectation and covariance:
x = E [[[x Xe+1, Wirl [Wir]
[[x; X441, Wial [Wis]]
(26)
E [xx² + J((x++1 F++1x() |Wir]]
=x,- + J,(x+11/7 F++1);)
Egp cov [B|x, 1 W [Wi7]] + B[cov[[x Xx+1, Wirl [wir]
cov [ x Xs+1, Wial
[Wir]] + E cov |x: X1+11 Wia] [Wir]]
cov [xx Jx Fx+1x()
W + E [ J, [wi]]
(27)
J.cov [x++1 F++1 |W:: J
Σ J J
J + + J J,E, J
Σt J. (E+tiTT ETT)J
A.3 PROCESS NOISE MATRIX
As stated in (18) can claborate the process noise matrix at time t in more details
Q E F,E,
F
K.-HH, E, H + R.-11-KK7 K (28)
combining 18 into 28
results in
Q: = Σ, F,
+ Q.-1
(29)
K, H. Fr
F.
Q Q- + R -K! F
which is a function of F,, Qr 1. Network with the input of x From 20 x x is derived as function of both F and H,-1,
F,-1 and H. In the GIN, F, and H, are learned by the Dynamics
13

---

Under review as a conference paper at ICLR 2023
R-2
9--1
R-1
R--1
Teel
0
(a) Without recurrent dependency on qr.
(b) With recurrent dependency on qr-
Figure 9: Graphical models for low dimensional observations experiments.
meaning the learned F, carries the information of both H and F Therefore, rewrite the equation (29h as
Q: =g F,
where F, Dynamics Network x (H. (Hg-1, ((-1)
(30)
where g is a nonlinear function mapping x and Qt-1 to Q₂ and the graphical model for such choice of structure is in figure 9b It is possible to go one step further and simplify x more, as it has _ term in (0), combining it with (18) results in
X + [ E FT Q--1]
(31)
(H,, FT Q1-1|H?- R4-1) (ww H,-1x,1)
indicating that not only F.-1 and H,-1, but also Q: is included in x meaning that Q. can be written solely as a function of x and the graphical model for such choice is in figure 9a
Q: =g (F.(xtt)) where F. Dynamics Network x (H,-1 Fi-1, R--1).
(32)
We call g as Q Network, where g can modeled by MLP (32 or a recurrent network (30), based on the mentioned explanations. In figure 11 it is shown how the Q Network is integrated into the whole model structure.
A.4 OUTPUT DISTRIBUTION
In the case of grayscale images, consider each pixel, yı, is one with the probability of 1 - Ps respectively, meaning that P(Y y) = p" (1 - p)" 19 By re-writing the probability equation
into the exponential families form
fo(u) = h(y)) (f))- - (1 6 logl (oggloorg) { log(1-p)
(33)
- It means that by considering the last layer the decoder and applying softmax layer, p is obtained. Equivalently, one can calculate the deviance between real and estimation of it, p. which is given by
and by choosing 0 log( and (0) log p). we can obtain
D(p. p) [plog P + (1 p)log( p
--
1 P1
(34)
14

---

Under review as a conference paper at ICLR 2023
and minimize the deviance with respect
did in 15
Similarly, consider z, âh and as the ground truth state, estimated state and the model variables respectively. where the residual follows Gaussian distribution Pg e~ (á og). where dg is
the estimated variance. Then, the negative log likelihood is given by 335 as we obtained it in [4).
CC x 2 [log(do)+ + (x $0)²
(35)
206
A.5 NOISE GENERATION PROCESS
In the high dimensional observation experiments, to show the noise robustness of the system, we use time correlated noise generation scheme. It makes the noise factors correlated over time by introducing a sequence of factors ft of the same length of the data sequence. Let fo ~ 24(0, 1) and fi+1 min(max(())) fa rz), 1) with r ~ ~((-0.2, 0.2), where fo is the initialized factor and 2/ is the uniform distribution. Then by defining two thresholds, t ~ 14(0,0.25 and t2 ~ (00775 1), and the rest splitted linearly within the range of [o, 1]. The 1-th obtained observation is given by O fei. + ft) where the i, is the I-th true image and i."h is the t-th generated pure noise.
Jt are to 0 and l2 set
A.6 LORENZ ATTRACTOR DYNAMICS
There three differential equations that model Lorenz system, x the convection rate, y the horizontal temperature variation and & the vertical temperature variation.
dr
x(p - y. dy dt
dt = o x),
dt = ry -y- Bz
(36)
where the constant values o, p and are 10, and 8 respectively. To construct trajectory we use Lorenz system equations (36) with dt 10-5, then we sample from it with the step time of Al 0.01.
3.
Based on the equations of the system (36), the state is St [xt, yt, 21] and we can write the dynamics of the system A, and obtain the transition matrix Expl At] Ft. To achieve this, we use the Taylor expansion of Exp function with degrees.
10
10
0
r y
(A (t, j!
S; = AtSt
28
and
F, = I+
(37)
y
j=1
where is the degrees of expansion and is the identity matrix. For the emission matrix we use H. and for process and observation noise standard deviation, we use Q R, II respectively.
Too
o o
and
A.7 MOVEMENT MODEL DETAILS FOR THE NCLT EXPERIMENT
We assume that the segway robot is moved with constant velocity, that the equations for such dynamics are given by
0p1 J
Op2
Ov U2, 01
Oor a1
UI
0,
0, x, = [pr, 1i, F1, U2].
Y: [p]. (38)
By such assumptions for the motion's equations the transition, process noise distribution, emission and measurement noise distribution matrices can be obtained by
At
0
At
0
0
0
0
At
0
0
0
F
Q =
H
0
0
A/
0
0
A/
0
0
0
0
0
0
0
0
0
A!
(39)
[1 0
0
R =
2
1
15

---

Under review as a conference paper at ICLR 2023
where At 1 since the sampling frequency is 1Hz. Process and measurement variance parameters, and X, are unknown that the model will learn them. we split the whole sequence into training, testing and validation folds with the length of 3600 ( 18 sequences of length T = 200) 280 (1 sequence of length T = 280) and 400 (2 sequences of length = 200), respectively.
A.8 NETWORK STRUCTURE AND PARAMETERS
In all experiments, Adam optimizer Kingma & Ba (2014) has been used on NVIDIA GeForce GTX 1050 Ti. We conduct grid search for finding the hyperparameters the models being trained with the suboptimal hyperparameters. To find the initial learning rate, by conducting grid search between 0.001 and 0.2 with the increment of 0.005, we select the best one among them that corresponds to the highest log-likelihood. With an initial learning rate of 0.006 and an exponential decay with rate of 0.9 every 10 epochs, we employ back propagation through time Werbos (1990) to compute the gradients we deploy GRU cells in the structure. Layer normalization technique Ba et al. (2016) is used to stabilize the dynamies the recurrent structure and normalize the filter response. Elu activation function, ensure the positiveness of the diagonal elements of the process, noise and covariance matrices.
rule out the possibility of
In order to prevent the model being stuck in the poor local minima, e.g. focusing the reconstruction instead of learning the dynamics obtained by soaeringg-moohhngg we find it useful to use two training tricks for end-to-end learning:
1- Generating time correlated noisy sequences as consecutive observations, forces the model to learn the dynamics instead focusing on reconstruction, e.g. figure 13 and T5]
2- For the first few epochs, only learn nuto-enncoee((MPP) and globally learned parameters, e.g. F(A) and (R), but not Dynamics Network parameters a (x+-1). All the parameters are jointly learned, afterwards. This allows the system to learn good embedding and meaningful latent vectors at first, then learns how to employ K different dynamics variables.
In the lack of dynamics, for the low dimensional observations we use K 5, while for the high dimensional observations we use K = 15 as they need to learn more complex dynamics. In general, the GIN is flexible enough, tuning the parameters not difficult as the GIN capable to learn how to prune unused elements by the Dynamics Network.
To prevent the model being stuck into mode collapse, we provided two solutions:
1- By introducing & sets of Fk H where each set of F H models different dynamics, we introduce loss term with small constant factor which tries increase the distance of each pair of F* H set. Intuitively, the presence of different dynamics can easily modify the states in each update. We found this method potential solution to prevent the model go through the mode collapse.
2- Considering the negative distance of consecutive pairs of states as additional loss term with a small constant factor (the distance can be considered as euclidean difference of mean or KI. of two consecutive states). Intuitively, this solution is forcing the states to not have overlap with each other and impose them to change in each update step.
In the simulation results, we have used the first option.
A.8.1 PROPOSED ARCHITECTURE FOR HIGH AND LOW DIMENSIONAL OBSERVATIONS
The proposed structure to deal with high dimensional observations in the lack of the dtnamics(the first three experiments in the paper) is shown in figure 10 While, the proposed structure to handle low dimensional observations in the presence of the dynamics(the last two experiments in the paper) shown in the figure
A.8.2 EMPIRICAL RUNNING TIMES AND PARAMETERS
We present the number of parameters of the utilized cell structures in our experiments and their corresponding empirical running times for epoch in the table 7 and In the first row of each model
16

---

Under review as a conference paper at ICLR 2023
GIN Gell
Decoder
Our
Encoder
Hiteriing
Smoothing
0ia
Tunn
Prodiction
(8 Min)
(ham EuT
Kun Sin
Dynamics Nuturakk
Fooss
FRUSE
N
P....e B.g
Prediction Step
Mar
Filtering Step
Prootting Step
Sin Pid
Oui Migl
WaT
(i Eir
(Kig) 1
R
Figure 10: Proposed architecture for operating high dimensional observations in the lack of dynamics.
GIN Cell
1:77
X,ry
Smoothing
1:77
Filtering
Eurı
Prediction
MIP
(F H(r)
(iig Fir
Fir
GRUSC
N:r
Qui
GRUK
Prediction Step
Fir
M
Filtering Step
Smoothing Step
(Fir H(r)
(Kig) Y-
[X Mirr
(x
Wi.
(N P1
Fur
H
Figure 11: Proposed architecture for operating low dimensional observations in the presence of dynamics.
structure in the high dimensional experiments, we set the number of their parameters approximately equal to our GIN to indicate the outperformance of the GIN with the same number of the parameters, i.e. tables and while we include the empirical running time and parameters for more complex structures as well. Extra running time of EM-variational approaches, like K is due to employing classic Bayesian equations because it increases the running time substantially when dealing with higher dimensional observations, however, the GIN circumvent this difficulty. The number of the parameters of the GIN are noticeably lower than other memory cells, e.g. LSTM and GRU, and EM-variational methods dimensional covariance matrices by employing convolutional operator. allows us to reduce the parameters of internal GRU cells quadratically.
convert high dimensional sparse covariance matrices into lower
17

---

Under review as a conference paper at ICLR 2023
A.8.3 QUALITATIVE COMPARISON OF THE GIN TO RECENT RELATED WORK.
In table D we make a comparison to show whether algorithms are able to handle high and low dimensional observations, learn dynamies, use available-pan partiel dynamics, estimate state appropriately, provide model's 's uncertainty estimates handling noisy data, handle missing data and perform direct optimization. Classic LGSSMs, e.g. EKF and UKF, work based on the linearization the transition and emission equations and apply classic Bayesian updates over the linearized system with respect to the states. In other words, (F, H) in the classic LGSSMs are not data-deriven nor trainable. Despite classic LGSSMs, in the GIN we use data-driven based network to learn dynamics, i.e. Dynamics Network in the paper.
A.8.4 SINGLE-PENDULAN AND DOUBLE- EXPERIMENTS
DATA GENERATION.
Dataset consists of 1000 train, 100 valid and 100 test sequences with the length of 150. The sequences are distorted via generated noise, while in the informed imputation task half of the images are removed and boolean flags indicating the availability of the observations are passed to the cell instead. If the imputation task is in uninformed type, black images are considered as the observations instead of informing the cell with boolean flags.
ENCODER AND THE DYNAMICS NETWORK ARCHITECTURE
To design the dynamics network, we use a MLP including 60 hidden units with Relu activation function and softmax activation for the last layer. The state mean, with size of nt and number of the bases, with size of k, are the input and output of the dynamics, respectively. The structures of the encoder and decoder are in the table In the table Iū, m the transferred observation dimension that various values for this parameter are taken into account in the results. In the state estimation tasks, out dim is 4 and 8 for the single-pendulum and double-pendulung experiment, respectively. For the imputation task, number of the hidden units of the KG and SG network is set to 40 and 30, respectively. The convolutional layer applied over the covariance matrix has 8 filters with kernel size of 5.
A.8.5 LORENZ. ATTRACTOR AND NCLT EXPERIMENTS
In these two experiments that we have the knowledge of the dynamics, we employ a fully connected with the observations its input and output dimension of 3 and 2 for Lorenz attractor and NCLT experiments, respectively, to obtain the observation noise, r. The activation function is Elu 1. Similarly another fully connected with the posterior state its input and output dimension of 3 and for Lorenz attractor and NCLT experiments, respectively, to attain the uncertainty estimates, o To estimate the process noise matrix, fully connect with the posterior state the input and Elu + activation function is used. Similarly, GRU cell that maps the posterior states to the process noise matrix with 10 hidden units can be used.
Table 7: Empirical running times and parameters of high-low dimensional experiments.
Table Low dimensional experiments.
Single
Double
Param
I/EE
Param
T/
Paraa
TB::
L.TM (unite:25)
>-18k
566
<118k
<5564
4-45k
~883
LSTM (umin==100)
LSTM (units-50)
7766
98s
<766
<.96
120k
1/33k
Coll
Param
Lanonz
T/E
Param
NCLT
T/E
GRU (umiss=00)
~188
H61s
1-18
~0825
KoimanNet GNN
(anits-50)
-674
>110k
~100k
-40%
100)
-57
-85
1111
RNN
~77%
KVAF (v--00)
~/25g
-05
<-25k
-977
~633
1416
~766s
-000k
KVAE (n=200)
<336
~1144
<366
<000
166
LGSSM
RKN 1001
<25k
<338s
445
GIN
~228
~9k
~311
(n=20)
~012
<825
<112
<000
-11
LGGSMM 4m=45)
<98
<15k
<.974
<36
1366
(v-30)
<118
4.2
GIN (5:45)
~225k
-5%%
<225k
-58
4-8
-85
18

---

Under review as a conference paper at ICLR 2023
Table 9: Learning the dynamics in LGSSM is shown with x/V because general LGSSMs, e.g. UKF and EKF, are not able to learn the dynamics. However, in our setting and parameterization we use data driven-basse network for obtaining (F, H) to make LGSSMs comparable with the GIN for high dimensional observation experiments.
Model
low-d
dynamics
dynamics
uncertainty
missing-oiise
LSTM Hochrelter Schmidmaner
1997
Wahletrde
E2C
Winke of ml
X
X
X
BB-V
Archer
2015
X
SIN
Kn Kinaiin c1
DVDI
DSA
Lr& Momaall 2015
KVAL
Laaceno al 2017
ISLSD
Lindemann
2017
Deep (aliins
DSSM
Rangaimm of 2010
JybridoN' Gurcla Sulorcss el
2019
KalmanNei
Rahe
LGSSM
GIN
A.9 VISUALIZATION AND THE IMPUTATION
Graphical results of informed, uninformed and noisy observations for image imputation task for both single and double pendulum experiments can be found in [2 [3 [4 and 15 figures. Inference for the trained smoothing and filtering distributions of all high dimensional experiments are in 16
18
36 37 38 and 39 figures. where generated samples from the smoothing distribution, j (x, W and the filtering distribution, (i:) Then we fit density on the generated samples. This visualization shows the effectiveness of the GIN in reducing the uncertainty of the estimates compare LGSSM and KVAE Finally, the
20 21 22
23 24
25
26
27 28
30
31 32 33 34
35
results of NCLT experiment are in figure 47
Table 10: The structure of the encoder and decoder for single and double pendulum experiments.
Encoder
Decoder
6 x Conv, 12, max pooling 2 x 2, stride 2 2
o
fully connected: out dim, linear activation
LayerNormalizer()
o
fully connected, Elu activation
4 x Conv, 12, max pooling 2 x 2, stride 2 x 2 LayerNormalizer()
imputation task:
fully connected: 144
fully connected: 40
6 x 6 Trnns Conv, 16, stride 4 x 4
w: fully connected: m, linear activation
LayerNormalizer()
fully connected, Elu activation
4 x 4 Trns Conv, 12, stride 2 x 2
LayerNormalizer()
o : 1 Trns Cony, stride 1, softmax
19

---

Under review as a conference paper at ICLR 2023
Ground Truth
Ground Truth
Observed Sequence
Observed Sequence
LGSSM(filter
LGSSM((ilter)
LGSSM(smooth)
LGSSM(smooth)
GIN(filter)
GIN(filter)
GIN(smooth)
GIN(smooth)
Figure 12: Informed(left column) and uninformed(right column) image imputation task for the single pendulum experiments.
Figure 13: Image imputation task for the single pendulum experiment exposed to the noisy observa- tions, where the generated noise has correlation with the time. Each figure, beginning from top to bottom, indicates the ground truth, noisy observation and the imputation results of the GIN.
20

---

Under review as a conference paper at ICLR 2023
Ground Truth
Ground Truth
Observed Sequence
Observed Sequence
LGSSM(filter
LGSSM(filter
LGSSM(smooth)
LGSSM(smooth)
GIN(filter)
GIN(filter)
GIN(smooth)
GIN(smooth)
Figure 14: Informed(left column) and uninformed(right column) image imputation task for the double pendulum experiments.
Figure 15: Image imputation task for the double pendulum experiment exposed to the noisy observa- tions, where the generated noise has correlation with the time. Each figure, beginning from top to bottom, indicates the ground truth, noisy observation and the imputation results of the GIN.
21

---

Under review as a conference paper at ICLR 2023
[x1465/ W. 150
10.00
14.
12-
GIN
LGSSM
KVAE
Figure 16: Inference for the single pendulum 11 position at 100-th time step. Generated samples from smoothened distribution, fx1 100 W 1: trained by the GIN, LGSSM and KVAE, respectively. The dashed red line (2 100 w1000) is the ground truth state with distribution of 5(z1 0.7).
We calculate the sample mean and fit a distribution on the samples for further visualization and comparison purpose.
(x2 IW: 1200
GiN
LOSSM
KUAAE
Figure 17: Inference for the single pendulum x2 position 100-th time step. Generated samples from smoothened distribution, f(222000 W1:150), trained by the GIN, LGSSM and KVAF, respectively.
32-
(oreg (asss)
33
(Klcao 12p0 Wiitel
44
x1
x1
x1
(a) GIN
(b) LGSSM
(c) KVAE
Figure 18: Generated samples from the trained smoothened joint distribution the single pendulum position, (x1, x2), at 100-th time step for the GIN, LGSSM and KVAE, respectively. The ground truth is shown with black point.
22

---

Under review as a conference paper at ICLR 2023
jx1 m001
14.
12-
GIN
LGSSM
KVAE
Figure 19: Inference for the single pendulum æ1 position at 100-th time step. Generated samples from filter distribution, f [[1]ooo] [ 10:) trained by the GIN, LGSSM and KVAE, respectively. The dashed red line (x1 100
Pos (1:100) is the ground truth state with distribution of 8(x1 0.7).
12-
|x 3000
10
GiN
LGSSM
Figure 20: Inference for the single pendulum x:2 position at 100-th time step. Generated samples from filter distribution, (=2100 W1:100). trained by the GIN, LGSSM and KVAE, respectively.
(Kloce Weaso)
18-
(1l Shanshacll.nnn (12)
Selooe Keyco Lacel
X
044
x1
(a) GIN
(b) LGSSM
(c) KVAE
Figure 21: Generated samples from the trained filter joint distribution of the single pendulum position, (el,x:2), at 100-th time step for the GIN, LGSSM and KVAF, respectively. The ground truth is shown with a black point.
23

---

Under review as a conference paper at ICLR 2023
00
pa W vee!
00
GIN
UCSSM
Figure 22: Inference for the double pendulum æ1 position 100-th time step. Generated samples from smoothened distribution, f(0l (1:10) trained by the GIN, LGSSM and K respectively. The dashed red line (x1 100 |1:150) is the ground truth state with distribution of 8(x1100 0.35).
200
02
-42 GIN
LCSSM
KVAE
Figure 23: Inference for the double pendulum x22 position at 100-th time step. Generated samples from smoothened distribution, f(22100l (::10)) trained by the GIN, LGSSM and KVAE, respectively. The dashed red line / (1:150) is the ground truth state with distribution 8(x2100 - 0.35).
(Klops (e Kescel
02
44
x1
(a) GIN
(b) LGSSM
(c) KVAE
Figure 24: Generated samples from the trained smoothened joint distribution of the double pendulum first joint position, x2), at 100-th time step for the GIN, LGSSM and KVAE, respectively. The ground truth is shown with black point.
24

---

Under review as a conference paper at ICLR 2023
53
GIN
LGOGM
KVAL
Figure 25: Inference for the double pendulum x3 position at 100-th time step. Generated samples from smoothened distribution, (23310 W1:150), trained by the GIN, LGSSM and KVAE, respectively. The dashed red line (x3 W1:150) is the ground truth state with distribution of 8(23100 - 1).
05
Dor Wi 1o0!
GIN
LOSSM
KVAE
Figure 26: Inference for the double pendulum 24 position at 100-th time step. Generated samples from smoothened distribution, (0 WI:150), trained by the GIN, LGSSM and KVAE, respectively.
(50l (Waze)
X4
-02
-44
X3
13
14
X3
GIN
(b) LGSSM
(c) KVAE
Figure 27: Generated samples from the trained smoothened joint distribution of the double pendulum second joint position, (z3, r4), at 100-th time step for the GIN, LGSSM and KVAE, respectively. The ground truth is shown with a black point.
25

---

Under review as a conference paper at ICLR 2023
00
242 W) 1ns!
04
GIN
UCSSM
Figure 28: Inference for the double pendulum æ1 position 100-th time step. Generated samples from filter distribution, f[21 100 W1:100), trained by the GIN, LGSSM and KVAE, respectively. The dashed red line (x1 Pos W1:100) is the ground truth state with distribution of 8( 0.35).
GIN
LGSSM
KWAE
Figure 29: Inference for the double pendulum x2 position at 100-th time step. Generated samples from filter distribution, [ (x21 W 100), trained by the GIN, LGSSM and KVAE, respectively. The dashed red line (I:2Pos |W1:00) is the ground truth state with distribution of 8(2:100 0.35).
(ct. x2ow coo!
38
07-
01-
11-
X1
(a) GIN
(b) LGSSM
(c) KVAE
Figure 30: Generated samples from the trained filter joint distribution of the double pendulum first joint position, (x1, x2), at 100-th time step for the GIN, LGSSM and KVAE, respectively. The ground truth is shown with black point.
26

---

Under review as a conference paper at ICLR 2023
beatEd ssel
GIN
LGOGM
KVAL
Figure 31: Inference for the double pendulum x3 position at 100-th time step. Generated samples from filter distribution, (=3100 W1:100), trained by the GIN, LGSSM and KVAE, respectively. The dashed red line (I:3600 (13100/W11100) is the ground truth state with distribution of 8(x3100 - 1).
Dor W 200!
GIN
LOSSM
KVAE
Figure 32: Inference for the double pendulum x4 position at 100-th time step. Generated samples from filter distribution, (0111))) trained by the GIN, LGSSM and KVAE, respectively.
12
C6
X3
14
X3
(a) GIN
(b) LGSSM
(c) KVAE
Figure 33: Generated samples from the trained filter joint distribution of the double pendulum second joint position, (z:3, 24), at 100-th time step for the GIN, LGSSM and KVAE, respectively. The ground truth is shown with black point.
27

---

Under review as a conference paper at ICLR 2023
Is] Fo
GIN
LGSSM
KVAE
Figure 34: Inference for the visual odometry æ1 position 100-th time step. Generated samples from smoothened distribution, [[x 100 (W1:00)) trained by the GIN, LGSSM and KVAE, respectively. The dashed red line (x1 Pos W1:500) is the ground truth state with distribution of 8(x1 + 50).
bra W300007
35.0
GIN
UGSSM
K0AE
Figure 35: Inference for the visual odometry x2 position at 100-th time step. Generated samples from smoothened distribution, (=2100 W1:500), trained by the GIN, LGSSM and KVAE, respectively. The dashed red line (1:2f06 W1:500) is the ground truth state with distribution of 8(x1 - 10).
(xllgge Saaa
(Klean Maaale seel
X2
X
-30
x1
(a) GIN
(b) LGSSM
(c) KVAE
Figure 36: Generated samples from the trained smoothened joint distribution of the visual odometry joint position, x2), at 100-th time step for the GIN, LGSSM and KVAE, respectively. The ground truth is shown with a black point.
28

---

Under review as a conference paper at ICLR 2023
GIN
LGSSM
KVAE
Figure 37: Inference for the visual odometry æ1 position at 100-th time step. Generated samples from filter distribution, (10000 trained by the GIN, LGSSM and KVAE, respectively. The dashed red line (x1 (o 100 W1:100) is the ground truth state with distribution of 8(x1 + 50).
Dx2 Wa:i0o
Figure 38: Inference for the visual odometry x2 position at 100-th time step. Generated samples from filter distribution, (=2 i0:)))), trained by the GIN, LGSSM and KVAE, respectively. The dashed red line (z:2Pos (i100)) is the ground truth state with distribution of 8(x1 10).
Okl Wan
Oclass Wood)
64
-1
GIN
(b) LGSSM
(c) KVAE
Figure 39: Generated samples from the trained filter joint distribution of the visual odometry joint
position, (x1, x2), 100-th time step for the GIN, LGSSM and KVAE, respectively. The ground truth is shown with black point.
29

---

Under review as a conference paper at ICLR 2023
epocts
3.3
0.25
0.50
2.4
339
1.7
M
M
Mo W A Hov
22
epochs 130
epochs 100
epochs 100
1.50
6.5-
875
11-
15-
11
22
13-
30-
thre
9iee
tiee
1,20
1.2-
1.50
150
635
83
30
150
22
125
125
122
10
(a) LGSSM
(b) GIN(Filter)
(c) GIN(Smooth)
Figure 42: Eigenvalues of the learned transition matrix Fr and their corresponding true values in the first 100 time steps for Lorenz attractor experiment. Despite the low dimensional experiments the paper that we give the dynamics (F, H) the model, here show the GIN ability for learning the dynamics, when we do not provide the dynamics information, iee. (F,, H,) in (37
30

---

Under review as a conference paper at ICLR 2023
egocha
opocis
opoct's
1.00
3.23
-23
1.3
0.7
333-
1.4
Paasi
Your p
1 y 4444
Wy AA 1 M
0.2
3.58
2.2
1g 10 Mu 11 Ap Mmme
R0
1 mp
Ad AdAnnn Anma
0.60
138
338
173
AM
M0 Po
May Jo
0.2
0.50
spocta
apachs
opocha
3.27
1.34
1.3
1.30
3 Soitt yoad y Al Moptoo
1.3
Pealin
1.2
20 1
475
1.5
A 10 10 M Yar M 1 My
Pgn 11 M M/
130
80
29
135
0n 2
639
(a) LGSSM
(b) GIN(Filter)
(c) GIN(Smooth)
Figure 44: Eigenvalues of the learned transition matrix F, and their corresponding true values in the first 100 time steps for NCLT dataset experiment. Despite the low dimensional experiments in the paper that we provided the dynamics (F, H) for the model, here we show the GIN ability for learning the dynamics, when do not provide the dynamics information, i.e. (F,, H,) in 39
31

---

Under review as a conference paper at ICLR 2023
epochs
epechs
1.2
123
0.7
132
Saiten
Paaln
208
075
242
1.6
0.0
338
28
378
0.5
030
opacha
epochs
epechs
1.0
135-
123
130
Paaln
2. 120
130
85
21
139
639
(a) LGSSM
(b) GIN(Filter)
(c) GIN(Smooth)
Figure 46: Eigenvalues of the learned transition matrix F, and their corresponding true values in the first 100 time steps for NCLT dataset experiment. Despite the low dimensional experiments in the paper that we provided the dynamics (F, H) for the model, here we show the GIN ability for learning the dynamics, when do not provide the dynamics information, i.e. (F,, H,) in 39
32

---

Under review as a conference paper at ICLR 2023
Barmariiee
20
25
10-
10 20
40
Bastjm]
Figure 47: NCLT dataset position for the first 50 observations: ground truth positions and the generated trajectories with the GIN, LGSSM, KalmanNet and DSSM approaches illustrated.
A.10 MSE RESULTS FOR THE STATE ESTIMATION AND ESTIMATED KG-SG
We compare the learned KG-SG matrices via the GRU cells with their corresponding ground truth for the first 100 time steps of the low dimensional experiments. We calculate the element-wise squared difference of the learned KG and its ground truth, AKG, = Tr (KKG KG,) (KG, K, and take the average of all AKG while similar procedure holds for the SG. The results are provided in table 11
Table 11: Comparison of leaned KG-SG matrices and ground truth KG-SG. Lorenz attractor and NCLT experiments with dynamics refer to the situation, where are given the dynamics form, i.e.
37
39
50 epoch
80 epoch
100 epoch
200 epoch
Exp
A KG
A SG
A KG
A SG
A KG
A SG
A KG
A SG
Lorenz with dynamics
2.83
2.49
1.52
1.26
0.69
0.62
0.46
0.49
NCLT with dynamics
1.91
2.15
1.39
1.24
0.93
0.95
0.67
0.73
Lorenz without dynamics
12.31
13.26
8.55
7.93
5,40
6.29
4.19
4.58
NCLT without dynamics
9.88
9.62
6.73
7.11
5.11
4.83
3.29
3.10
The MSE results for the single and double pendulum experiments are in the table 12 and 13 In addition (7). where F matrix includes the effects of the process noise, two other mentioned solutions introduced in section 4 are included in the MSE results as well. Using GRU cell and MLP for mapping xt, their input, Q, their output, where the former is shown by GRU(Q) and the latter one by MLP(Q) in the tables.
33

---

Under review as a conference paper at ICLR 2023
Table 12: MSE for single pendulum experiment.
Model
MSE
LSTM (units = 25, m 15)
0.092:10.003
LSTM (units = =25, 20)
0.092:00.005
LSTM (units = =25, m = 40)
0.090=0.005
LSTM (umits -100,
15)
0.089=00002
LSTM (units = 100, m 20)
0.089=00.005
LSTM (units 100,
40)
0.090=0.004
GRU (units = 30, m 15)
0.095
GRU (units = 30, m 20)
0.093=0.002
GRU (units = 30, m 40)
0.094110.005
GRU (units = 100,
15)
0.091=000002
GRU (units = 100. m 20)
0.092=0.004
GRU (units 100. 40)
0.091=0.008
Model
F(Q)
MLP(Q)
GRU(Q))
LGSSM filter(m 15, 30, K 10)
0.089
0.088110.005
0.088=00000
LGSSM filter(m = 15, n 30, = 15)
0.081
0.087110.007
0.086110.00
LGSSM filter(m 15, 45, K 10)
0.085+10.004
0.08411.007
0.084
LGSSM filter(m 15, n = 45, K = 15)
0.084
0.083:100004
0.082+0.004
LGSSM filter(m 20.
40, K 10)
0.084=0.009
0.082:0.014
0.082=0.011
LGSSM filter(m = 20, n = 40, = 15)
0.012
0.081+00.005
0.080+-0.014
LGSSM filter(m = 20,
60, K 10)
0.05
0.078+00012
0.076=000009
LGSSM filter(m = 20, n = 60, K = 15)
0.007
0.075+0.001
0.074
LGSSM smooth(m 15, 30, K 10)
0.086=0.011
0.08310.004
0.084
LGSSM smooth(n 15, n 30, K 15)
0.081
0.084110.008
0.083
LGSSM smooth(m 15, 45, K = 10)
0.081
0.080+10.009
0.079100003
LGSSM smooth(m = 15, n = 45, K 15)
0.081
0.078
0.077+0.011
LGSSM smooth(m 20, 40, K 10)
0.082#0.005
0.07810.004
0.076=0.013
LGSSM smooth(m 20, n 40, == 15)
0.080+0.003
0.076+-0.006
0.074+0.010
LGSSM smooth(m 20,
60, K 10)
0.076+0.008
0.073++0002
0.070+00.009
LGSSM smooth(m = 20, π 60, K 15)
0.073+0.013
0.071
0.068
GIN filter(m 15, 30, K 10)
0.018
0.076:10.005
0.07510.004
GIN filter(m 15, π 30, K = 15)
0.078
0.075+10000
0.07410.012
GIN filter(m 15, 45. K = 10)
0.014
0.073+10008
0.072100009
GIN filter(m 15. π = 45. K = 15)
0.073+0.015
0.074=100011
0.071+0.005
GIN filter(m 20. 40. K 10)
0.072=0.005
0.072+0.008
0.070+0.002
GIN filter(m = 20, 40, K = 15)
0.071
0.071+00004
0.071+00009
GIN filter(m 20, 60, K 10)
0.067
0.066+00005
0.065+00006
GIN filter(m = 20, 60, K = 15)
0.065+0.013
0.06410.009
0.06310.010
GIN smooth( m 15, 30, K 10)
0.071
0.070-1.003
0.06910.009
GIN smooth( m = 15, n 30,
15)
0.070+10000
0.068+10.011
0.068110007
GIN smooth(rr 15, 45, K 10)
0.065+0.011
0.06510.009
0.06410.012
GIN smooth m = 15, п = 45, = 15)
0.064=0.008
0.066110.007
0.063+0.009
GIN smooth 20, 40, K 10)
0.064=0.005
0.065+0.003
0.062+00008
GIN smooth(rr m = 20. n 40, = 15)
0.063+0.004
0.064+10.011
0.063+00007
GIN smooth(m 20, 60, K 10)
0.059
0.061+0.012
0.057+00.06
GIN smooth(m = 20, n = 60, K = 15)
0.058
0.057+0.009
0.056+10.004
34

---

Under review as a conference paper at ICLR 2023
Table 13: MSE for double pendulum experiment.
Model
MSE
LSTM (units = 50, m = 15)
0.172+00012
LSTM (units 50,, m 20)
0.166=000009
LSTM (units = 50, m = 40)
0.167=0.011
LSTM (units 100, m 15)
0.16410.006
LSTM (units 100,
20)
0.162=0.009
LSTM (units = 100, m = 40)
0.159+0.010
GRU (units = 50, m = 15)
0.194
GRU (units 50, m 20)
4.189+0.01
GRU (units = 50, m = 40)
0.188
GRU (units 100.
15)
0.179=00009
GRU (units = 100, m 20)
0.169=0.014
GRU (units 100, m 40)
0.166
Model
F(Q)
MLP(Q)
GRU(Q)
LGSSM filter(m 15, 30, K 10)
0.054
0.159+10.021
0.153+00009
LGSSM filter(m = 15, n 30,
15)
0.052
0.153++0.010
0.152+0.012
LGSSM filter(m 15.
45,
10)
0.14410.011
0.14110.015
0.13910.013
LGSSM filter(mn = 15. n = 45, K = 15)
0.042
0.13110.012
0.137+0.017
LGSSM filter(m 20, n 40,
10)
0.044
0.137+0.009
0.138+0.013
LGSSM filter(m = 20. n = 40, K 15)
0.041
0.137#0.008
0.136=00.016
LGSSM filter(m 20, n 60, = 10)
0.129#0.009
0.126:100014
0.122+10.155
LGSSM filter(m 20, 60, K 15)
0.121
0.124
0.119100009
LGSSM smooth(n 15, n 30, K 10)
0.049
1..014
0.144
LGSSM smooth(m 15, n 30, K = 15)
0.001
0.146-
0.142
LGSSM smooth(m 15, n 45, K 10)
0.139
0.13610.009
0.137
LGSSM smooth(m = 15, n 45,
K = 15)
0.137
0.135+0.017
0.13310.012
LGSSM smooth(m 20, n 40, K 10)
0.136+0.014
0.131+0.022
0.132+0.011
LGSSM smooth(m = 20, 40, K = 15)
0.134+0.011
0.129+0.014
0.129=0.022
LGSSM smooth(m 20, π 60, K 10)
0.129
0.116100016
0.115
LGSSM smooth(1 20, 60, K 15)
0.120+0.010
0.112
0.108
GIN filter(m 15, 30, K 10)
0.126
0.0251
0.125
GIN filter(m 15, 30, K = 15)
0.005
0.124
0.009
GIN filter(m 15, 45, K 10)
0.115
0.11440.015
0.110
GIN filter(m 15. 45. K = 15)
0.114
0.112+00.000
0.11010.011
GIN filter(m 20,
40. K 10)
0.113+0.013
0.111+0.009
0.111=0.013
GIN filter(m 20. 40. K = 15)
0.111=0.009
0.109+0.018
0.108+00000
GIN filter(m 20, π 60, K = 10)
0.099
0.09411.011
0.095110021
GIN filter(m 20, 60, K = 15)
0.097
0.093
0.091
GIN mm 15, 30, K 10)
0.015
0.116
0.113
GIN smooth(1n m = 15, n 30,
15)
0.015
0.113
0.112
GIN smooth 15, n 45,
10)
0.105
0.10110.014
0.101+00015
GIN smooth( m = 15, n = 45, K = 15)
0.10210.013
0.100#0.011
0.09810.008
GIN smooth( m 20, n 40,
10)
0.101
0.098+0.0100
0.0941-0.015
GIN smooth(nn m = 20,
40, K 15)
0.098
0.095+0.014
0.092+00000
GIN smooth(rn m 20, n 60, 10)
0.086
0.081
0.07910.009
GIN smooth(m 20, 60, K 15)
0.083+0.009
0.079400006
0.076.10.013
35

---

Under review as a conference paper at ICLR 2023
A.11 ALGORITTMM
Algorithm High-Dimensional Observations Training
Input:
Ground Truth gt Observations 01.T, last posteriors (xt.7, Et.r),
initial posterior (x 2
01:7 Dynamics Network (x DT
Obtain F 1:2 and HIT by 13
[x Firr) = Prediction Step ((x -11 F1r))
(Wi:7; F1:T) encoder (01:T)
K1.r = Σ 1:T Hir M 1. M GRU KG (Co
J1.7 1 FT FT.
1:7 N 1:
NIT GRUSG (E
(x 2Tr)) = Filtering Step (XT:r> ET.r, KIT, WI.r, Hir)
(X1:TIT, E1:ryr = Smoothing Step (x Si J1., FIT)
o = decoder (x::7/T) E1:T(r)
LTT Likelihood (gt, O7.))
Backward Propagation ()
Algorithm Low- Now-Dimensional Observations Training
Input:
Ground Truth gt1:r, Observations Y1.r, last posteriors (xt:r, Di.r)
initial posterior (xx+ Σt)
Dynamics are not known then
01.7 Dynamics Network (x
D:T
Obtain F 1. and Hrr by
13
(W1.7, F1.7) MLP (y1-7)
(x Σ 1.T) Prediction Step (X E F..
Ki.r Hi. M MI. GRUKG 1.r FIT
J1.7 D Fiz NI.NNN: NIT GRUSG (Σ
(x Eir)) = Filtering Step (xT.T: 2T.T, KITT WI.T, H1T)
(X177TT Σ I.TTT) Smoothing Step (xirr: Ei Jusr Fir)
o = ML.P (x1.TTT+ +, E
L1 Likelihood (gt).r: T
Backward Propagation ()
else
Q network = MLP (x or GRU (x QI.T)
(Fi., H1rr) =
Fi. = trainable layer(y):r)
q1.r Q network(x*trr1))
(x ΣT.r) Prediction Step ((x -1: Σ 1)) Qui, F1a)
K1T= = M M L.T.
M GRUKG ( FIT)
JI.T =E 1 i.T.
NiT GRUSG (EiTT)
(x Σ -1.r) Filtering Step (xT.T En.ry KIT,,1.1 Hir)
(X1.TTT. Ei.ryr) Smoothing Step (xtrr: s Jur, Fir)
0:T/7 = Trainable Layer (Ei:1r)
P [x1:rTr,0:771]
L = Likelihood (gt,.r: (
Backward Propagation ()
end
36

---

Under review as a conference paper at ICLR 2023
A.12 PYTHON INFERENCE CODE
To demonstrate the simplicity of our proposed GIN, we include intuitive inference code with Tensor- flow library for both the high dimensional and low dimensional experiments. The code runs with Python 3.6+. The entire code to reproduce the experiments are available in Github repository.
A.12.1 PYTHON INTUITIVE CODE FOR HIGH DIMENSIONAL EXPERIMENTS.
import tensorslow keras as
import Prediction
import Filtering
import Smoothing
import DynamicaNettorkk
import Encoder
import Decoder
import DataGen
10
class GIN_CELL layer):
11
def _init_ (self, initial_states)
12
self.x_tml_t self.Sigma_tmll inttial_states
felf.filter_states
xxeeff self. f.8t gma_tml_+]]
def call(selE, inputs):
15
w w11:I, r_1:1 inputs
16
for w_t, r_ in (w w_if, x]1:T):
17
F_hat_t, H_hat_t - DynamicsNetwork (se1f.x_mm__)
x_t_-, Sigma_t_- Prediction (F (F_hat_t, iLhat_t,...
19
self.x_tml_t, self.Sigma_tmL
20
x_t_t, Sigma_t_ w_t, r_t, H_hat_t)
Filtering(x____. sigma_t_-r...
21
salf.filter_states..pedd (x____,, Sigma_t_t])
self.x self.x_tml_t, self.Sigme Sirffiiigaammm_____ tml_+ x_t,,, Sigma t_t
24
x1:iT_, Sigma_1:T_r Smoething (sclf. (soethiin
25
sigma_1:I_-, F F_hat_i:T)
26
return x1:___, Sigma_1:T3
class GIN (k. Model)
29
def _init_ initial_statest
30
self.x_0 +, self.Sigma 0 initial states
self CELL OBJ__BJJ---------- self.GIN BEM__EELLC (self.x. +_ self.Sigma 0_-)
seloeEncoder Encoder
33
self.Decoder Decoder
34
35
def call (sclf, o_1:T)
w_1:T, r_ = self Self.Enne Encoder (o []]]]
x_I:__, Sigma_l:T_T self.GIN LON_E_EL_LBBB 2elf.GIN_CELLLOBI [w]l:T, r_l:T)
38
<_1:T_+ self. Decoder (x_1:T_T, Sigma_1:T. 1:T_T)
39
return 1
40
Data (ataGen
0 =T (Data)
37

---

Under review as a conference paper at ICLR 2023
A.12.2
PYTHON INTUITIVE CODE FOR LOW DIMENSIONAL EXPERIMENTS.
import tensorflov weeerrloovkkkss.s... keras as k
import Prediction
import Filtering
import Smoothing
import MLP ENC
import MLP_DEC
import DataGen
import Dynamics
class CIN_CELL(k,.1 layerssllly
def init _init_ (self, initial_states)
12
self.x_tml_+, self.Sigma_tm1_ saitial_states self.filter_statte
13
[ Salf.Sigma_tm1111
14
def call(self, inputs)
16
w_l:T, r_l:T, F_1:T, HIT:T inputs
17
for w_t, r_t, F_t, H_t in (4_1:T, r_leT, F_1:T, H_1:T):
18
x_t_-, Sigma_t_- Prediction (KLt, H_t, self.x_tm x_tmm_____ __, self selg.Sigma tml_+)
x_t,,, Sigma t tx Sigma t_,, wtt, r_t, t self.filter aptedd [xx___, Sigma t +]+
21
22
self.x_tm1_t, self.Sigma_tml_+ - x_t_,, sigma_t_*
23
x11:T_T, Sigma_1:i_r_ sigma_1:T_-, F_1:T) x_1:T_T, Sigma_l:T_j
26
27
class GIN (tt.cc.els..odelll
28
def init__it_ss__ (self, initial_states Dynamics):
self. self.Sigma_0.. 0_+ initial_states
30
sel2.F_1:I, self. Dynamics
31
self.GIN_CHIIL_ON www.se..fllee. C.....NNCCUUU self fal..igma_0_-)
32
self.MLP_ANe MLP_ENC
self.MLP_DEC MLP_DEC
35
def
call (self, 0 1:TT:
36
r_1:T = self.MIA ENC (c__1:)
37
x11:T_I, Sigma_1:T_T = wwwf. so1f..IN_CCELLLO0 (o r_liI,...
sclf.F_1:I, sc:f.H_1:T)
39
sigma_o_1:i_+ seII.ME_BCCKSLgm__1111
40
x_c_l:T_t x_1:T_T
41
return x_o_1:T_t, Sigma_c_l:T_+
Data DataGen
46 Mynanice_Matrices
Matrices Dynamics ()
45 x o 1:T +, Sigma o 1:I + = GIN (Data, Dynamics Matrices)
38


Reviews:
[{"review_id": "tmA4btmEJht", "reviewer": "Reviewer_SVRC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper suggests a state space model to handle both high and low dimensional observation spaces with highly nonlinear observation model, such as animated images of physical systems. The internal state representation is probabilistic, taking account uncertainties. The model is a nonlinear extension of the Hammersein-Wiener model, where the state transition is also allowed to be nonlinear.", "summary_of_the_review": "The paper is well written and the method seems quite novel.  The claims against Variational Methods should be a bit more clarified. It seems the likelihood itself missing a regularization what a variational objective would have provided. Clarifying what is happening there would be quite illuminating.", "strength_and_weaknesses": "Strengths:\n- Ability to handle high dimensional inputs, such as images directly\n- The method builds on existing knowledge from Systems theory, utilizing inductive biases from the field. \n- The experimental setup and the method is very well described\n- No costly matrix inversion in the dynamics part\n\nWeaknesses:\n- The case of observational uncertainty is solved in an ad-hoc way. While a Kalman filter uses some prior assumption about noise levels, here the un-encoded observations are point observations. It is clear that in the case of high dimensional input this problem is hard to solve better than it is addressed here, for low dimensional cases it may be possible.\n- It is not exactly clear how the likelihood without regularization / priors is sufficient to learn a probabilistic representation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OjozQvzO5qU", "reviewer": "Reviewer_jiSL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Gated Inference Network, a recurrent architecture for handling noise, missing data, filtering, and smoothing in time series. The methods build on the literature on Kalman Filters and use specific parameterizations for the gain and smoothing matrices representing them as nonlinear transformations of their inputs parameterized by GRUs. Comparisons are done against deterministic architectures such as LSTM and GRU and stochastic models with variational approaches. Both high-dimensional and low-dimensional synthetic datasets are considered for the comparisons showing the scalability of GIN compared to alternatives.", "summary_of_the_review": "The paper could largely benefit from a clear organization and logical flow. Comparisons are not comprehensive and some methods are not considered. The methods presented in the paper lack a solid theoretical and statistical grounding, making it difficult to compare against the existing statistical literature on time series models.", "strength_and_weaknesses": "**Weaknesses**\n\n**Notation and wording**\n\nOverall, the notation and wording of this paper are confusing and not standard. Please add a notation section and clarify what each variable is representing. More specific comments are below.\n\n* What are latent noisy observations? If a variable is latent it means it's not observed, this wording is confusing for the reader, please fix this.\n* What does the superscript + mean? This notation is not common in the statistics literature.\n* My current understanding is that $x$ corresponds to dynamics, $w$ corresponds to transformed inputs, and $o$ denotes inputs. Since there are so many letters and new notations used in the paper it would largely benefit from a notation section and will help the reader understand the main contributions better.\n\n**Statistical Grounding**\n\nFrom a statistical viewpoint, after reading the paper multiple times it is still unclear to me what the generative model and its corresponding graphical model are in this framework. What inference algorithms are used and what approximations are made? More specific comments are below.\n\n* What is the graphical or statistical model?\n* Do we start by a model of $x_t|x_{t-1},w_{1:t-1}$?\n* If so, does this determine a full joint distribution over all the variables?\n* Starting from $x_t|x_{t-1},w_{1:t-1}$ do we first marginalize out $x_{t-1}$ and get $x_t|w_{1:t-1}$ and then to get $x_t|w_{1:t}$ after observing $w_t$?\n* What happens if we don't want to use Gaussian distributions? For example, what if the input data is not continuous? Does this framework still hold? What parts do we need to change to adjust for different generative assumptions? If this works only for the Gaussian case this should somehow be reflected in the abstract to clarify the scope of the presented method.\n* The Kalman framework seems to rely on linear approximations of the dynamics (where the matrices are given by nonlinear transformations of their inputs). Is the linear approximation a good enough local approximation? \n* The paper seems to lack a solid theoretical or statistical grounding. What type of inference algorithm is used and what is the underlying statistical model? The inference algorithm (if theoretically valid) has to be approximate since the model has nonlinearities and we cannot integrate them out. It looks like the KG and SG matrices are approximated using GRUs for the high-dimensional cases. How do we know if the GRU functions are finding the right KG and SG? What is the justification for using identity functions for low-dimensional cases? How is this reflected in the inference algorithm?\n\n\n**Presentation**\n\nThe presentation and the flow of the paper can be largely improved. I would recommend the authors use the following flow for their presentation.\n\n* First write a summary of Kalman Filtering for completeness to set the context and familiarize the readers with notation (you can add a notation section before this).\n* Write a statistical model, draw a graphical model, include parameters, random variables, and noise, and use a better naming convention (latent vs. observed, which variables are following what type of dynamics or equations.\n* Write your parameterization (eq. 7-11) and which functions are approximated by GRUs and how this fixed the issue with KF such as matrix inversion, high-d data, covariance parameterization, include a note about what are the approximations to what cost function and how this relates to VI and other approximations.\n\n**Comparisons**\n\n* The method includes lots of matrix multiplications, how does the time complexity and scalability of your method compare to alternatives e.g. VAEs? Can you include a paragraph about the time complexity and report some results on the wall clock time for your methods and comparisons with others?\n* The comparison against E2C sounds like a strawman, E2C is an RL algorithm where the representations are learned for a completely different purpose.\n* Many other comparisons aren't included both in the introduction and in the compared methods such as SVAE, ARHMM, SLDS,\n* For the results reported in tables, did you run all the models until convergence? \n* How is the model complexity accounted for? How does the number of parameters across different models correspond to the number of units?\n* You are comparing stochastic and deterministic models. This is not a fair comparison. There are various methods in the VAE literature with specific latent and approximating family structures and algorithms for improving the inference such as importance-weighted autoencoders for tighter bounds, etc. For other models such as ARHMMs and SLDS models, there are specific inference algorithms designed such as Laplace-EM to help with convergence and generalizations. These are not included in the comparisons.\n* Test log-likelihoods are not comparable across different models.\n* I don't quite understand the sample generation plots, do we want the variance to be small and more densely concentrated around the mean? Although a single parameter has generated the data, we might expect a larger set of parameters to produce similar observations. How should this be quantified properly?\n\n**Minor Issues**\nEq 3, 4: what do you mean by proportional to?\nEq 5, \"can be written as a function of $F_t$\". This needs to be written down in the main text, it's an important piece of what follows in the manuscript. \nI don't understand eq. 6, it is not a standard notation. Can you clarify this?\n\n\n## Post Rebuttal\n\n**Presentation** I thank the authors for their detailed responses and clarifications. After reading the author's responses and comments from other reviewers I still think that the paper will largely benefit from a new organization to help clarify it from a statistical viewpoint. Specifically, the graphical models and the notation is not consistent with the literature on state space models (R1, R2, R3, R4) and it makes it hard to identify the contributions and follow the notations. A common strategy for organizing a state space model is to first include a generative process (accompanied by its corresponding graphical model) where there is a clear distinction between random variables, parameters, choice of distributions, functions, etc (R5). Then determine the inference strategy (variational inference, max likelihood, max a posteriori, sampling-based inference, etc.) and determine what approximations are performed. This organization will clearly resolve questions such as the linearity of the model vs. approximation (R7). Including the literature of Kalman filtering as the base notation in the main text will help determine where this method lies in the literature of SSMs (R8). After reading this paper multiple times I still don't have a clear picture of what the generative process is and how this method lies in the statistical models of time series literature.\n\n**Comparisons** My comments and concerns regarding the comparisons did not spark the introduction of new results and comparisons which is against my expectation (R11, R12). In addition, the existing results require more information to be accompanied for completeness such as (effective) model complexity (R13). I also didn't find some of my original questions to be addressed fully such as R15. \n\nI do believe that the paper poses a really interesting perspective and contribution, and it would be beneficial work to share with the community. Hence I increase my score slightly and encourage the authors to do another iteration of refinement. However, I think the paper is not quite ready for prime time as it is.\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DkTbGrb2CcU", "reviewer": "Reviewer_dayt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a novel Gated Inference Network (GIN) to infer and learn state space models in both high and low dimensions. They use concepts in Kalman filtering to obtain estimates of the dynamics. They show that GIN is better at learning state space representations with disentangled dynamics features than existing approaches, they are also robust with noise and able to impute missing data. GIN is efficient in handling the case of unknown dynamics, the models are learned directly from the observations which are fed into encoders to obtain latent observations. ", "summary_of_the_review": "Due to the lack of clarity and the weaknesses mentioned above, I am not recommending acceptance in the current state of the submission.", "strength_and_weaknesses": "1. For many case studies, the authors do not estimate the latents but assume the latent form. In a typical dynamics and latent estimation study, we are not provided the latent form (for example, position and velocity in the Michigan NCLT dataset). What happens if the authors want to estimate the latents as well as the dynamics?\n2. The paper sets out to perform system identification and learn the parameters of the state space model, but the authors only compare the predicted / estimated states with the ground truth. Certain properties of the recovered matrices can be compared with ground truth, for example, their eigenvalues. The authors need to show that their methods can recover ground truth dynamics.\n3. The paper lacks motivation for the architecture of GIN. For example, is the goal of the encoder and decoder simply to reduce dimensionality? What if the model does not have an encoder and decoder, for example in the case of a low-dimensional input (the authors state that they use an MLP in this case, but is this necessary)?\n4. The data that the authors used to perform the imputation task is simple although it outperforms other methods shown in the paper, the data has a high amount of temporal correlations, so it may not be hard to achieve an imputation task. Imputation in a complex real-world dataset would be more helpful to show the ability of GIN.\n5. The “lack of dynamics” needs to be explained more clearly: it is not clear whether the dynamics are not being inferred or there is no succinct generative dynamical model for the simulations.\n\nMinor comments:\n1. The notation of x_{t+} and x_{t-} appear on page 3 first, but their definitions are in the caption of Figure 3 which is later shown.\n2. The introduction does not clearly state the problem that the authors are considering. There are many typos and colloquial sentences, e.g., “a bunch of approaches”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject