Gated Inference Network: Inference and Learning State-Space Models
Authors: Hamidreza Hashempoor
Wan Choi
Venue: NeurIPS 2024 poster (2024)
Keywords: Time Series and Recurrent Networks
Primary area: other
TL;DR: Our algorithm efficiently models dynamical systems by observing high-dimensional noise-affected data, outperforming state-of-the-art counterparts in state estimation and image imputation tasks.

Abstract: This paper advances temporal reasoning within dynamically changing high-dimensional noisy observations, focusing on a latent space that characterizes the nonlinear dynamics of objects in their environment. We introduce the *Gated Inference Network* (GIN), an efficient approximate Bayesian inference algorithm for state space models (SSMs) with nonlinear state transitions and emissions. GIN disentangles two latent representations: one representing the object derived from a nonlinear mapping model, and another representing the latent state describing its dynamics. This disentanglement enables direct state estimation and missing data imputation as the world evolves. To infer the latent state, we utilize a deep extended Kalman filter (EKF) approach that integrates a novel compact RNN structure to compute both the Kalman Gain (KG) and smoothing gain (SG), completing the data flow. This design results in a computational cost per step that is linearly faster than EKF but introduces issues such as the exploding gradient problem. To mitigate the exploding gradients caused by the compact RNN structure in our model, we propose a specialized learning method that ensures stable training and inference. The model is then trained end-to-end on videos depicting a diverse range of simulated and real-world physical systems, and outperforms its ounterparts —RNNs, autoregressive models, and variational approaches— in state estimation and missing data imputation tasks.

Gated Inference Network: Inference and Learning State-Space Models
Hamidreza Hashempoor
Seoul National University
Department of Electrical and Computer Engineering hamidr hamidrezz hwshempmssuu...mmmmmmm ac. kr
Wan Choi
Scoul National University
Department of Electrical and Computer Engineering wancho
fanchoi@snu.. @snu ac kr.kr
Abstract
This paper advances temporal reasoning within dynamically changing high- dimensional noisy observations, focusing latent space that characterizes the nonlinear dynamics of objects in their environment. We introduce the Gated Inference Network (GIN), an efficient approximate Bayesian inference algorithm for state space models (SSMs) with nonlinear state transitions and emissions. GIN disentangles two latent representations: one representing the object derived from a nonlinear mapping model, and another representing the latent state describing its dynamics. This disentanglement enables direct state estimation and missing data imputation as the world evolves. To infer the latent state, we utilize deep extended Kalman filter (EKF) approach that integrates novel compact RNN structure compute both the Kalman Gain (KG) and smoothing gain (SG), completing the data flow. This design results computational cost per step that is linearly faster than EKF but introduces issues such as the exploding gradient problem. To mitigate the exploding gradients caused by the compact RNN structure our model, we propose a specialized learning method that ensures stable training and inference. The model is then trained end-to-end on videos depicting a diverse range of simulated and real-world physical systems, and outperforms its counterparts RNNs, autoregressive models, and variational approaches- state estimation and missing data imputation tasks.
Introduction
A state-space model is type of graphical model that effectively represents noise-afflicted data I. Typically, the objective is to conduct inference, which involves obtaining accurate estimates of the posterior distribution of latent states or noise-free measurements. In videos, inferring the state space presents a complex challenge owing to the high dimensionality noisy observations, which only offer partial information about the states. Previous methods have proposed various inference approaches such as sampling 2 variational inference [3), or belief propagation 4 Within the framework of a hidden Markov process. classical methods like the celebrated Kalman filter (KF) and smoother [). along with their extensions like EKF and UKF, [7 have been employed to address the posterior inference problem. However, the reliance of these methods accurate knowledge of the underlying dynamics, coupled with computationally intensive matrix inversions, poses challenges in scalability to high-dimensional latent spaces.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To address these issues, this paper distinguishes between the sensory observation and its dynamics by disentangling two representations: a transformed observation w, obtained by mapping the original sensory observation or through a nonlinear function modeled by neural networks, and the latent state space x, describing the dynamics of W; at time step f. To infer high dimensional state space (dynamics) Xt, we introduce filtering and smoothing techniques that build upon classical EKF updates but incorporate flexible function estimators without relying on a constrained graphical model. We model temporal dynamics with linearized Gaussian SSM, which is adapted to accommodate complex dynamics. To address non-linearity and the multiplicity of dynamics, we learn multiple transition and emission matrices, each modeling unique dynamical scenario. Then, the weight of each matrix during inference is determined by the past state x;-1 through a neural network. To prevent mode collapse and ensure the system's ability to capture diverse dynamics, we introduce loss term proportional to the KI. divergence of the transition matrices during training. This design enables our model to handle various state scenarios, similar to approaches found in the switching linear dynamics systems (SLDS) literature [9]. Unlike EKF, which requires precise characterization and modeling of the underlying dynamics, operate under the assumption that noise statistics and the underlying physical dynamics of SSM are entirely unknown.
The main drawback of the EKF is that it takes O(n²) time per step, because we need to invert the posterior covariance matrix with size of n. This makes the method slow to use. Moreover, we identify the computation of the KG and SG in the (extended) KF as critical components that rely on noise statistics and domain knowledge. In this paper, we efficiently model KG and SG using GRU cells, bypassing the need for matrix inversion in the KF flow. Consequently, the GIN algorithm achieves a time complexity of ((32n2) per step, making it linearly faster than EKF, To capitalize on the sparsity of the covariance matrix, we employ convolutional approach that further reduces the size of n relative to 3, where 0 < B < 1. However, the nonlinear structure of GRU cells can lead to gradient explosion [IO]. phenomenon that occurs when the dynamies undergo significant changes as GRU parameters traverse bifurcation points [1] during the learning process. To address this issue, we propose a learning method based on analysis of GRU cell dynamics. This method aims to prevent parameter values from reaching bifurcation points by stabilizing the dynamics of GRU cells, thereby addressing the issue of gradient explosion.
To summarize, our primary contribution is a novel algorithm designed for the recursive inference of SSMs with arbitrary nonlinear (and possibly -au-saas))) emission and transition models, with high-dimensional noisy stimuli observation. This algorithm utilizes a linearized Gaussian dynamics approach, with parameter estimation conducted through neural networks. This means that we can apply the recursive Bayesian updates, akin to the Kalman filter and Kalman smoother. We introduce a likelihood for inferring the states (dynamics) and another likelihood for inferring the high-dimensional images (of video). The objective is maximized in a supervised manner, depending on the task. We propose learning schemes for GRU cells to address issues related to gradient explosion and instability. Finally, we introduce a loss term proportional to the KL divergence of the learned transition matrices to prevent the system from becoming stuck in mode collapse.
To substantiate our claims, we conduct five experiments. First, we simulate a nonlinear dynamic system using the pendulum sequence video, common benchmark in this literature, demonstrate our model's ' ability to infer both dynamics and images. Second, we introduce a more challenging experiment of simulated nonlinear double pendulum, where the video sequence is heavily distorted with noise, showcasing our model's resilience of dynamics estimation and image imputation to noisy observations. Third, we present a switching dynamics irregular bouncing ball experiment illustrate our model's ss ability to handle multiple dynamic scenarios. Fourth, we perform visual odometry using the KITTI dataset, demonstrating the practical applicability of our methods in real-world scenarios. Finally, we assess the effectiveness of our proposed gradient explosion handling scheme by evaluating convergence across various simulation seeds.
2 Related Works
EKF and UKF represent early extensions of the original Kalman filter, allowing for nonlinear
transitions and emissions. These methods, categorized as model-based (MB) algorithms, rely on precise knowledge and modeling of the underlying dynamics within a fully characterized SSM. Consequently, the performance of these MB methods is heavily contingent on the accuracy of domain knowledge and model assumptions. Recent approaches, such as BackpropKF [12] and SIN [3],
perform EKF in the latent state using deep encoders for feature extraction. However, they face similar

---

Generated sequences from GIN in the irregular polygon environments. The videos shown as single images, with color intensity representing the incremental sequence index.
Figure 1:
limitations EKF, including computational complexity and scalability. To address these issues, common approach involves considering diagonal covariance matrix in the KF flow as 14 and
15]. However, aim to find richer approximation the posterior.
A significant body of work, including Variational Auto-Encoders (VAE) [3, Embed to Control (E2C) [6]. and Importance Weighted VAE (IW'AAE) [77. integrates deep learning with variational inference (VI), but lacks memory cells and recurrent structures for handling imputation tasks. To address this, EM- based VI approaches like Structure VAE (SVAE) [18], Kalman VAE (K (AE))) [19), Disentangled VAE (DVAE) [20), Extended KVAE (EKVAE) [2T]. Robust VAE [22]. and Markovian VAE (MVAE) 23 original KF equations for filtering and smoothing. However, they cannot directly optimize states (dynamics), as noted RKN [4] and CRU 25). Classical memory networks like LSTMs (26], GRUs [27). and simple RNNs 28 infer latent states but fail to provide insights into uncertainties and dynamics.
LatentNet [29]. KalmanNet [30] and SSI-SSM [31 utilize GRU in their structures for state updates, similar to the parameterization used in the GIN, However, these models typically necessitate access to complete or partial dynamics information, i.e. they are MB, and their reliance vanilla GRU cells can cause instability of the SSM. Liu et al. [32 proposes a disentanglement model for audio data, which shares similarities with the GIN. However, their parameterization and objectives differ from those the GIN, and they do not utilize compact RNNs in their inference structure.
We cover the most relevant works in this section. A more detailed discussion, including comprehen- sive comparison of recent related works like System Identification (SI) with details of EM algorithm, auto-regressive (AR) and SLDS models, is provided in Appendix [A.8.3 and and Table This builds upon the findings of the RKN with additional complements. We also conduct an empirical complexity analysis in appendix A.8.2 to evaluate the computational efficiency of our method compared to the discussed approaches. This analysis involved measuring the execution time per iteration using a clock on the wall as the benchmark.
Background
Linear Gaussian state space models. Linear Gaussian state space models (LGSSSMs) are [W1...., wr].
commonly employed to model vectors of observed time series, denoted LGSSMs excel in filtering and smoothing tasks. They model the first-order Markov process on the state space = [X1,... ....
.T,,,.,,], XT). which may also incorporate external control inputs [u...., Ur] under the multivariate normality assumption of the state:
Pr, (x, u.) = (x;; F,X,-1 + B,u, Qe), Px. (w.lx)) = N(w,; H. R,).
(I)
Tt. represents the parameters of the system at time I, encapsulating information from matrices F., B,, H., Q₁ and R,, which correspond to the state transition, control, emission, process noise covariance, and observation noise covariance, respectively. At each time step, the transition and emission procedures are subject to distortion caused by process noise and observation noise, following distributions of (0, Q,) and (0, R,), respectively.
The goal of filtering is to infer the posterior dis- tribution Pro (xt [Wi1; U1:)) while, the smoothing aims to infer the smoothing distribution Prur (x: ( unrT) given the whole observations. Without loss of generality, we drop the in- put variable in the rest of paper, while one can condition all of the distributions on the input variable the case of existence. Considering the prior state parameterization Paz., (x//// <x//w:e-1)= N (ptj-1, E and relying on [33,, the filtering and smoothing parameterizations are
Filtering and smoothing algorithms.

---

017
Man
Bingur)
N--
Ne41
e(.)
Transition Block
d()
Figure 3: The GIN as HW model for system identification. By appropriate structure selection for e(.) and d(.). the GIN can handle high di- mensional observations. The relation between the internal variables, W. and X,, is simulated by the transition block.
W--1
We+1
9t
Figure 2: Graphical model. Dashed nodes are
task dependent output.
Pria (x)lW1:)) = N [1 +K, H14ppill E -
= -N Eeje)
(2)
Pou (xe(WI))) =N [ell +JJeuu+1 F44149)
Eijt + J.S,
-N Eger)
(3)
K, = Seje-HH H H H + Rs
J, E. .+1 E
(4)
with y HH H + R. and S | Eejj]] In (4) K, and J, are KG and SG, respectively. See appendix A.I for full derivation of filtering-mmootte smoothing distributions.
Gated Inference Network
The background section defines key elements for the GIN: original observations 01:7, task-dependent output which can be either the original denoised-imputed observations 01.7 or physical system's 's states $1:7 transferred observations W1.7 E RmxT latent states (dynamics) X1.7 E R noise covariance of transferred observations R with diagonals F1:7 E RT*T" noise covariance of states process Q Q.. with diagonals qı.T IR** and the parameters of the SSM 7u.T We consider the dimensions of the transferred observation and state at each time step to be m and n, respec- tively. The states evolve through transition distribution Prur (x1:7)
(((x)) II Pr, (x, (x,(x.1).. Each W: is assumed to be drawn from noisy emission probability Pr, (w; x2) then the gen-
erative process is assumed to factorize as P WI:T) = pn. (Xi:) II Pv, (w.)x,) This factorisation imposes emission conditional independence like w, 1 (w-6, x-)) where (w-.. = (W (W(1z-1,+1:7)- xx ((at-11++1))) Common models that assume conditional indepen- dence include linear dynamical systems, hidden Markov models, but also nonlinear SSMs with higher-order Markov chains in latent space. Graphical model of the GIN is in figure 2 where output (one of colored nodes) is generated, contingent upon the task. In all of our scenarios, the parameters of the SSM, 71:7, are completely unknown.
From SI perspective, the GIN operates similarly to Hammerstein- ing non-linear transfer functions e(.) and d(.)) (Figure 3. Leveraging encoder-decoded structures for e(.) and d(.). the GIN conducts the transfer between original sensory observations 01:7 and lower- dimensional representations W1-7 and task dependent output. The transition block in Figure evolves states using the proposed deep EKF approach, efficiently approximating Eltering-smoothing distribu- tions while ensuring system stability with the imposed constraint. Further details and formulations provided in the inference section. The transition block depicted in Figure
(HW) model 34 employ-
5 Parameterization and Inference
Given original observations o and transferred observations Wi.r, our aim to infer the latent states XLT. To accomplish this, we seek to infer the marginal distributions

---

[Wit-2
We-1
1/11/11-11
Wz Xel
Ny X//Waz
We+1
Xet W
Dynomic Net
Dynamic Not
Dynomic Net
Miteriig
24-2
W
X Xt-1/Wir
Xt-1
Xt-1
X₁ W
xg/We.
Nt+1 /
Smocthing
Smoothing
Smoothing
Figure 4: Transition block figure 3 in details. each time step, the last posterior X;- | is fed to the Dynamic Net compute Tt. In the filtering steps, by using the last posterior X l Wir-1 and the observation We, the next posterior X₂ W1:2 is obtained. Having posterior X₃ Wi1: and the next smoothing state W applying smoothing for the current state is feasible so that the smoothing state X2 W1.1 is obtained.
Prui (xg W1:e) for the online inference approach (filtering) and P (Xp (X:l) for the full in- ference approach (smoothing). We introduce advantageous prediction parameterization as Py, [3,1x,11>> W ia-1) = (x,,,,,11- Q²). where X,-1 is sampled from the last filtering distribu- tion, P.u. (xt-1 Wi:--)). Then, obtain the prior distribution time L, Pyi. (Xel Wi:t-1)
(( F,
Q+) (, integration. The Gaussianity of Pm i (x Wi:t-1)
by marginalizing out X;-1 from
Pr, (xt
(x-
results from the Gaussianity of prediction parameterization. After obtaining Py:: (X, Wi:-11) and
observing W,, it is feasible to derive the filtering parameterization using equation 2 Once all
transferred observations WI.T available, backward induction can be employed to propagate to previous states using the chain rule. This procedure, known as smoothing, is parameterized with (3). These parameterizations provide valuable insights into two key aspects: 1) appropriately modeling y using neural networks, and 2) illustrating a tractable method to parameterize KG and SG and construct distributions approximations. These approximations serve the basis for constructing output.
Learning To handle multiple dynamic scenarios, we learn K sets of state transition and emission matrices Fs and H each representing a distinct dynamic status. These matrices are combined with state-dependent coefficients a (x:-1) as
F Σ (x H. > ax
(5)
We model a (x(-1) with a K dimension softmax output neural network called the Dynamic Net. takes the last filtering state X;-1 input, containing the system's 's history with lower noise distortion than the transferred observations Wt-1- This choice enhances noise robustness, demonstrated in our experiments with time-correlated noise (See Appendix(A.6)
In the graphical model shown in Figure 2 we observe two paths for belief propagation from x,-1 to X₁. The first path, X - Xt, linked with F,, The second path involves an intermediate variable q. ~ (O, Q:): X;-1 + q: ~ Xt. Since learned F, transfers information from X;-1 to x,, we argue that it can capture effects similar to those of Qt, both are intended convey message between X and x, [1. By incorporating F, from equation (5) into the prior state distribution described earlier as Prr. (xx] (xe|Wi:e-1) = (F F, E + Qt), we can neglect the
Q term in the covariance, as its influence is accounted for by the learned Fr. There are two other meaningful parameterizations for the process noise matrix. First involves direct parameterization by x using neural network. Second approach to parameterize recursively using x, and q:-1, resulting in new graphical model with an edge from qt-1 - q (See Appendix A.2). Both parameterizations are included in our results, presented the appendix, to show their effectiveness.
Finally, the diagonal elements of the transferred observation noise vector r, are directly mapped from the original observation space using the encoder function e(.) shown in Figure The mapping employs an activation function, elu+1, to handle the positivity of the diagonal elements.
Filtering and Smoothing Approximation.
We consider GRUKC KG network to approximate KG as:
K, = HHI-HH,,M..
M, = G ((Conv((CnV//nn (Celov(Eee--):.. r.))
(6)
5

---

Adje = Asje-1 Ke-[w, H. Σ.1 = Eqj-1 + E Eft- R.J.K. (7)
In Equation (6. Conv(.) represents a zero bias convolutional layer with pooling, employed to deal with sparsity of covariance matrix and extract its relevant information while reducing its size. The [.] symbol denotes the concatenation operator. Furthermore, the presence of a positive ry vector and the consideration of the Cholesky factor, M,M, in 6 ensure the resulting covariance matrices maintain positive definiteness. This parameterization construct a new filtering distribution Iyu. (x,|ve Ww:s) = (igge Enee) that is an approximation of (2). Consequently, we consider the approximated prior distribution as Iyn (x, w Wit--) N -/t- ([ - E(jt-1), where ( 1) are used in
After obtaining filtering states from O (x, Wi:),, we use GRUSG network to approximate SG in similar way we used GRUKG in
as:
J. = 1 FT. 141 N.N.
N. = =GGUUUSG (Conv(E
(8)
Pur = Auee + J. [e - Fetiden), EØr= = Eup + J.. - Fets FT (9)
where the first smoothing state is set to the last filtering state. The smoothing distribution Iur (Xel Wi.r) N ((rr Eyr) an approximation of the exact smoothing distribution in (3). For
a GRU cell with an input size of and a hidden state size of h, the computational complexity in + 3)), which scales linearly with the input size (35]. Considering Conv(E) 8 as the input of our GRU cells in 6 and (8) with
the forward pass is (3hh(h
[0, as the pooling ratio,
the forward pass of GIN for one time step has time complexity of (3h,52n2) where > h.
Compared with LGSSM matrix inversion time complexity of (n') GIN is faster by factor of 3h.32, which is crucial in high- -iseniinnsio regimes.
6 Fitting
In the state estimation task, the output from d(.)) in 3 s, the physical system's states. However, in the imputation task, output of d(.) is same as o since the original observation is recon- structed (see decoder structure for each task in A.8.1) The conditional distributions p( wirloorr) and Pn.r (Xi:/(Wi are modeled using an encoder e(.) and smoothing parameterization, respec- tively. Mcanwhile, the conditional distributions p(s1:r |x1:T, W1:7, 01:7) and p(o1:T) X1:7, W1.r) are represented by the decoder d(.) for the tasks of state estimation and imputation. These distributions modeled using multivariate Gaussian and Bernoulli distributions, respectively. Depending on the characteristics of the observations and states, alternative likelihood distributions can also be em- ployed. For instance, the beta likelihood for data in the unit interval, mixtures for multiple marginal distributions, and the negative- binomial likelihood for positive count data (See appendix[A.3]
Likelihood for Inferring States. The following theorem defines objective for inferring physical system's S states. The proof is provided in appendix A.4.I
Theorem 1. The lower bounded conditional log likelihood of the physical system's 's states given original images is determined as:
N
N log N (s a (S))))))
L(SIT/OLT) = Σ log p(sI E. OI:T)
(10)
N
1=1
where the d.(()) and d.(.) determines those parts of the decoder obtaining the state mean and variance. N sequences of (x w ~ Quue (XLT, Wi.T Wi:T(orr) are sampled for Monte Carlo integration estimation.
Likelihood for Inferring Images. For the imputation task, consider the ground truth as the sequence of images 01:7 with the dimension of D., The following theorem defines objective for inferring images. See appendix A.4.2 for the proof.

---

Theorem 2. The lower bound of log likelihood of the original images is:
Da oi ( log (1l7)))))))) (0) )og((
[(o1.))= = 1/ N Σ log p(our |XLTT
1
(11)
with N sequences of (x I:T 1: ~ 4u (XLT, WLT ( for Monte Carlo integration estimation. dg(.) defines the corresponding part of the decoder that maps the k-th pixel of Or image.
We use Wishart distribution as prior for our covariance matrix of states in 100 and 11.. which pushes the covariance toward a scale of identity matrix. Such prior prevents getting high oog-liielihood due to the high uncertainty. We shrink this scale toward zero as time passes, as we expect the model to finally perform with very little uncertainty, approaching deterministically.
Mode collapse handling. To address the issue of mode collapse, where the model becomes stuck in the same state, we propose a loss term on (i,j) pair of transition matrices as follows:
mme (i, = F Diag(m,) )l? ||F" F'1 - |[F' Diag(m,)ll
(12)
with m, and my being distinct hyper parameter vectors, and ||.1| is the Frobenius norm. The term Ee 2j=1 Im (i. can be added into (10) and (11 This addition ensures that each F captures a unique transition, distinct from others. From statistical perspective, (12) represents
a term proportional to - K (p: (P.(F)))))))))))))) (F))((F)
+ KL (F)(lrrrr (See appendix A.5)).
(F) ~ MN nxn (F', I) and pj (F) MN nxn F I). are matrix normal distributions with priors (F) ~ MM nxx (Diag (m,), M.,,,., I, and pri, (F)
(Diag (my) I, respectively.
Gradient explosion handling. Training TTPp and
with SGD can be disrupted by a gradient
explosion. Specifically, we limit the optimization in TT)) and (11) subject to a (U,) < 2, where
a,(.) represents the i-th largest singular value and U ha denotes the weight matrix of the hidden state E A with h dimension in the GRU cell (see eq.(5) in [3] or (53 in appendix A.4.3)
Theorem 3. When 01 (U,)
GRU cell is locally stable at a fix point h 0.
We use Lyapunov stability and two lemmas for the proof of Theorem 3 in appendix A.4.3 Solving 11) and (11 with SGD gives an updated U, in each iteration that may not satisfy a (U,) < 2. We offer two solutions satisfy the constraint outlined in theorem 3. The first solution relies the spectral theorem, offering higher accuracy but with increased computational cost. Alternatively, the second solution employs the Gershgorin circle theorem, providing lower accuracy but at reduced cost. Due to space limitations, we provide the details of the second solution in appendix [A.455] For the first solution, we modify U, in three steps: (i) Decompose Ü, by singular value decomposition (SVD) such that U, WAV. (ii) Replace the singular values of diagonal A that are greater than 2 with the threshold 2 - 5 and obtain = Diag(min( 2 6),... min(an, 2 13-0) 6)). (iii) Reconstruct U, + WAV. Employing SVD approach, the cost of this solution is proportional to (hh)). In the next theorem we show that constructed U, is the optimum solution. The proof is in Appendix[A.4.4 Theorem 4. The modified weight matrix U, obtained from (iii) step above, is a solution of the following optimization problem: minu, |Ū, - U s.A. (1() < 2 - 5.
7 Evaluation and Experiments
The first two experiments are single pendulum and double pendulum, where the dynamics of the latter one is more complicated. The third experiment is ball bouncing in irregular polygon to demonstrate the ability of the GIN when faces irregular environments with multiple underlying dynamics. The fourth experiment covers visual odometry task for real world data. The last experiment shows the effectiveness of theorems 3 and 4 for gradient explosion handling. Intuitive python code and training algorithm are in appendix [1 Detailed explanations regarding hyperparameter optimization, network structure, and empirical solutions aimed at avoiding poor local minima are in appendix A.7
Single Pendulum and Double Pendulum.
In the pendulum experiment, observations are the image sequences comprising 100 time steps, each has the size of 24 x 24 distorted with time correlated noise. We per- form the Eltering-smoothing by the GIN. In state estimation task, the log-likelihood and
7

---

Table I: Double (left) and single pendulum (middle) and polygon (left) state estimation results. (81, 82, 83, 84) are estimated position of the joints of double pendulum. For single pendulum and polygon, (81, 82) denotes the estimated single joint and ball position. sampled from eq. (10).
Modal
Se
SE
SE Poa
Log Likellhood
Log Likelihood
SPag
Log Likelthood
VAR
0.212
0.275
0.192
0.265
1,953
+: 306
0.211
0.247
2.191
+: 0.451
0.769
0.782
2,094 = 0.392
TW-VE
0.266
2.113
0.291
2.328
0.291
VAE-RNN (LSTTM)
0.154
0:147
0.134
0.152
4,051
0.565
0.111
0.099
5.601
0.151
0:447
0411
3.157 = 0.123
VAF-RNN (GRU)
0.164
0.156
0.162
0.145
3,776
4: 0.231
0.109
0.101
5,740 1: 0.168
0.499
0.415
3.188 =: 0.099
SVAE (LDS)
0.190
0.211
0.179
0.159
3
0.425
0.109
0,096
5.736 11661:1000 0.168
0:124
0.438
3.229
1: 0.112
KVAE
0.188
3,679
0.101
3,194
FKVAR
0.171
0.159
0.151
0.162
3,501
0.116
0
0.093
5,58
0.113
0.340
0.499
3.276
0/064
MVAE
0.168
0.161
0.139
0.149
3,927
1: 0.226
0.067
0.067
5938
1: 0.137
0.310
0.401
3.305
1: 0,069
DeepAR
0.175
0.189
0.157
0.147
3,646 1: 0.294
0.107
0.094
5.746 1: 0.204
0.447
0.434
3.071
2: 0.
RKN
0.134
0.129
0.139
0.118
4.176
0.294
0,078
0.075
6.161
0.23
0.314
0.437
3.324 d: 0.039
CRU
0.317
0:127
0.116
0.104
4,209
0.237
0.074
0,009
6,348
+ 0.19
0.294
0.378
3,300
0,039
Encede Decoder
0.291
0.284
0.342
0.317
1,577
+: 0.427
0.248
0,191
2,271 + 0.215
0.969
0.914
1,984 4: 0.384
AE RNN (LSTM)
0.163
0.171
0.148
0.167
3,901
0.706
0.089
0,087
5.751
1: 0.215
0,349
0.319
3.331
1: 0.124
AE-RNN (GRU)
0.183
0.177
0.369
0.085
5.796
0.205
3,267
LGSSMmmr
0.125
0.119
0.121
0.107
4.191
0.127
0.077
0.073
6.211
0.265
0.448
0.312
3,171
+: 0.365
LGSSM Mynooh
0.109
0.111
0.104
0.101
4,231
1: 0.154
0,071
0.069
6.242 1: 0.109
0.327
0.317
3,544
1: 0.029
GIN (1:2m))
0.105
0.099
0.109
0.099
4,524 1: 0.105
0.063
0.06
1.192 1: 0.239
0.203
0.208
4,125 1: 0.146
GIN (ram
0.083
0.081
0.888
0.079
4,879
1: 0.151
0,57
0.056
7,315
0.220
0.198
0.199
4,254
# 0.187
emoott (m=22mn)
0.082
4.708
0.123
0.057
7.292
0.173
GIN Croon (re:3er))
0.069
0.073
0.075
0.067
4.977
0.168
0.049
0.047
7,445
0.165
0.154
0.147
4.342
0.051
/
Figure 5: Pendulum imputation. Rows from up to down are the noise distorted ground truth, observation with uninformed missing frames and the imputation results the GIN(smoothed).
Figure 6: D-pendulum imputation. Rows from up to down are the noise distorted ground truth, observation with uninformed missing frames and the imputation results of the GIN(smoothed).
squared error (SE) of the estimated states are given in table [ With and m representing state and transferred observation dimensions) as shown in Table [
3m (n
intuitively states contain information about position, velocity, and accelera- tion, increasing the likelihood com- pared to n = 2m, where only po- sition and velocity are allocated to states. For the imputation task, we randomly remove half of the images from the generated sequences and per- form image imputation by predicting the missing parts. The numerical re- sults are in table 2 and imputed im- ages are in figures and 6 In table 2] the models with informed boolean masks are aware of available and miss- ing images, while uninformed masks use a black image as input for missing images. This encourages the model to accurately infer the dynamics for image generation.
Table 2: Image imputation task log likelihood for single pen- dulum (left), double pendulum (middle) and polygon (right). The approaches without good results are not included.
Model
Single Pendulom
Double Pendulum
Irregslar Polyzon
VAE (informed)
.36.751
7,227
-44.264 + 4.232
(informed)
-34.211
5.421
5.421
RNN (informed)
-14.243
0.328
0.425
SVAE (LDS) (infoemed smooth)
-14.763
0.419
-16.978 4: 0.551
-16.578
d: (155)
KVAE unfermed smooth)
-14.217
0.236
-15.917
4: 0.294
-16,377 4: 0.688
KVAE tunfonned smooth)
-39.260
-38.544
6.419
EK (infomsed smooth)
-12.97
0.524
-15.916
EK VAL (unfocmed smonth)
-29.246
3.328
-33,548 4: 4.516
MKVAE pinformed amooth))
-12.723 4 0.428
-13.889
+ 0.298
-15.698 + 0.344
MKYAE tunformed smooth))
-21.524
1,008
-26.773
-316634
2
DeepAR (infommed)
-14.019 0.486
-15.881 4: 0.511
-19332
0.418
RKN (nformed)
-12.782 0.016
-13.832 1: 0.023
1662257 J: 0.313
(unfommed)
-12.789
-13.91
0.031
-66,588 0,482
(informed) CRU (unformed)
0.023. -12.495
-13668 + 0,088
-15.884
0.397
-12.554
0.029
-13.739 1 0.061
-15.972 1: 0.394
Fnooder- Decoder
Fnedder-Deeder (informed)
-64.569 2: 146
-75. +: 31.527
AF-RNN (GRU) (anformed)
-14.173 2: 0.0K6
-16013 + 0.346
-17.903
0.43)
LGSSMüdfoemmd smooth)
-12.395 0.048
0.013
-15.851
0.215
For ablation study, first we compare GIN with simple encoder-decoder without latent parameterization, and
GIN (infeemed smooth) GIN (unformed smooth)
-11.215 2: 0.027
-12.584
0.021
-14.678
0.226
-11.246 2: 0.029
-12,65
+ 0.019
-14.923 1: 0.384
8

---

then with LGSSM, where the GRU cells are omitted from the GIN structure and classic filtering- smoothing equations are used instead. We also check the usefulness of Altering-smoothing parameter- ization by replacing the GIN's transition block with GRU and LSTM cells and directly parameterize output distribution (like AE-RNN (36] and (37]. We compare various VI models, including VAE, VAE, VAE-RNN (38 which utilizes VAEs with recurrent cells, and EM-based VI approaches such as SVAE, KVAE, EKVAE and MVAE. Our comparison also encompasses DeepAR [39], an AR model. Our thorough comparison, covering the CRU, RKN, group of VAEs, and AR models, enhances the depth of our analysis. The GIN consistently outperforms all other models. Additionally, provide results using the MSE metric to highlight the competitive prediction accuracy of our approach (See appendix A.10).
Bouncing Ball In Irregular Polygon. subjected it more complex environment. We generated 1000 sequences, each comprising 70 time steps, depicting ball moving within an irregular polygon featuring random shapes. The initial position and velocity of the ball were randomized. No external forces acted upon the ball, and collisions with the walls were elastic. In addition to the aforemen- tioned models, we include latent SLDS structure from the SVAE approach for the comparison. SLDS is particularly relevant in scenarios such as this one, where sep- arate dynamics for each state are consid- ered, when the ball bounces off the wall. We exclude simple baselines, such as encoder-decoder, VAE, and IW-VAE, from the comparison due to their lack of compet- itive performance. The numerical results are in tables [ and 2 In Figure 7 we il- lustrate samples obtained from the trained smoothing distributions of GIN, LGSSM, and KVAE, showcasing their proximity the ground truth. For this demonstration, we specifically select the 35th time step for sampling, denoted as L( [($33)01:70). Here, s1 and s2 represent the estimated xy posi- tion of the ball within the polygon. Gener- ating 20 frames with GIN four different irregular environments is shown in figure
To demonstrate the adaptability of the GIN, we
20
plgtl,s) 8tsl 25)
pigtlast 12]
A35.00 29.15
Aluur 11.86
12 24.91
Ma5 22
44smo Basie
Toggo 14.11
Jasno 12:51
Aas o2 10.07
Masize m 10.73
Man
St
pigt23s) = 6152 10)
p(gt2agl = 6(52 - 11)
GiN
LGSSM
KWAE
KVAE
Figure 7: State (position) estimation in irregular poly- gon (6 edges) 35-th and 45-th time steps. The upper row shows the ground truth ball position in 35, 39, 42 and 45-th time steps, respectively.
I We have created animated files that demonstrate both sequence generation and imputation tasks within irregular polygons. Please visit: httpp:://stts..gnttt wttpgl...om//ivviiieeee wwwp:///wwwww//.//innnu////222.
Visual Odometry of KITTI Dataset. We also evaluate the GIN with the higher dimensional
observations for the visual odometry task on the KITTI dataset 44) This dataset consists of 11 separated image sequences with their corresponding labels. We use a feature extractor network
proposed by Zhou et al. in [41] to obtain the transferred observations of the GIN, i.e. (w,r), Additionally, we compare the results with AE-RNN(LSTM), AE-RNN(GRUU, DeepVO (42 and KVAE. The numerical results are in table 3 where the common evaluation scheme for the KITTI dataset is exploited. The results of the KVAE degrades substantially as we have to reduce the size of the transferred observation to prevent the complexity of matrix inversion. Sampling visualization from smoothing state distribution are in appendix A.9
Table 3: Comparison of model performance on KITTI dataset.
AL-RNNILSIM)
AERRNN(URU)
DecpVV
KVAB
LUSSM
GIN
1sg((%)
(%)
"ig!"
(%6
Firl"
Agg (%)
Fial
(RE)
rial")
Yigl
03
3.59
4.53
9.34
3.31
8.49
0.89
12.14
1.38
7.51
3.95
6.9%
3,27
3.44
12.36
2.89
6.97
13.17
2.64
9.14
9.66
2.5
10.99
3.22
3.42
3.82
10.93
3.98
6.70
3.51
6.14
2.90
07
9.83
5.48
13.70
6.32
3.91
4.60
12.73
4.68
6.59
3.09
7.21
2.98
13.58
3.19
13.37
3.25
8.83
14.79
2,90
8.37
2.59
muzm
10.53
3.87
11.63
6.12
12.53
7.55
3.28
7.05
2.75

---

Gradient Explosion Experiment. Table lists the log likelihood and its standard deviation for three experiments: single pendulum, double pendulum. and irregular polygon. These experiments were trained under different settings for gradient explosion handling: the conventional Gradient Clipping (GC), our first solution using Singular Value Decomposition (SVD), and our second solution using the Gershgorin Circle Theorem (GCT). The table empirically demonstrates that our method outperforms conventional gradient clipping.
this table, represents the threshold for gradient clipping as introduced in [0), and ő is the threshold in our method that keeps the spectral radius less than 2, i.e., (U,) + 2. As shown in table gradient clipping failed to train for high 0, while approach achieves lower perplexity and higher log likelihood, ensuring stability compared to gradient clipping by constraining the GRU to be stable. We provide a comprehensive discussion on the á variable, including how to tune it in different environments and its impact model performance, in the appendix 1
Table 4: Gradient explosion handling: comparison between GC, SVD and GCT.
Single Pendulum
Double Pendulum
Irregular Polygon
Objective
Success
Objective
Success
Objective
Success
8 == 0.1
7.445
0.165
100 %
4.977 1 0.168
100 %
4.342 1 0.051
100 %
SVD
8 == 0.4
7.281 + 0.149
100 %
4.847 0.241
100 %
4.191 + 0.094
100 %
(Ours)
8 3=00 0.7
7.127 0.197
100 %
0.237
4.692 + 0.237
100
4.019 0.214
100 %
8
6.935 0.271
100 %
4.471 = 0.341
100
3,839 = 0.200
100
8 = 0.1
7.318 +
100 %
4.849 + 0.197
100 %
4.217 + 0.114
100 %
GCT
8 0.4
7.184
100 %
4.711 + 0.179
100 %
= 4 0.124
100 %
(Ours)
8 0.7
6.981 0.283
100 %
4.562 + 0.193
100
3.957 = 0.117
100 %
8
6.787 0.321
100 %
4.291 0.281
3.672 + 0.176
100
0 == 5
7,221 + 0.254
100 %
4.765 1 0.349
100
4.172 1 0.274
100 %
0 = 10
7,301 + 0.495
60 %
4.834 + 0.592
50 %
4.222 + 0.570
50 %
GC
0 15
7.331 + 1.141
20 %
N/A
%
N/
0 0%
8
N/A
0%
N/A
0%
N/A
0%
Conclusion
This paper introduces the GIN for representation learning in high-dimensional SSMs. The data flow is managed by Bayesian filterig--smott sheriig-smoothing, and the use GRU-based KG and SG networks addresses computational issues, resulting in efficient and numerically stable model. The GIN learns system dynamics andtt--end making it high-performance model with strong system identification capabilities. It also provides insightful uncertainty representations for predictions and outperforms various counterparts, including generative models with variational inference, autoregressive models, and other baselines.
9 Acknowledgments
We thank Yasin Abbasi-Yadkori for useful discussions and suggestions that contributed to this work.
Bibliography
[1] C. M. Bishop and N. M. Nasrabadi, Pattern recognition and machine learning. Springer, 2006, vol. 4, no. 4.
[2] R. M. Neal et al., "Meme using hamiltonian dynamics," " Handbook of markov chain monte carlo, vol. 2, no. 11, p. 2, 2011.
[3] D. P. Kingma and M. Welling, "Auto-encoding variational bayes," arXiv preprint 112.6:1312.6114, 2013.
[4] D. Koller and N. Friedman, Probabilistic graphical models: principles and techniques. MIT press, 2009.
[5] R. E. Kalman, "A new approach to linear filtering and prediction problems," 1960.
10

---

[6] H. E. Rauch, F. Tung, and C. T. Striebel, "Maximum likelihood estimates of linear dynamic systems," AIAA journal, vol. 3, no. 8, pp. 1450, 1965.
[7] E. A. Wan and Van Der Merwe, "The unscented kalman filter for nonlinear estimation," in Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373). leee, 2000, pp. 153-158.
[8] L. Ljung, "Asymptotic behavior of the extended kalman filter as parameter estimator for linear systems," IEEE Transactions on Automatic Control, vol. 24, no. 1, pp. 36-50, 1979.
[9] S. Linderman, M. Johnson, A. Miller, R. Adams, D. Blei. and L. Paninski, "Bayesian learning and inference in recurrent switching linear dynamical systems," " Artificial Intelligence and Statistics. PMLR, 2017, pp. 914-922.
[10] R. Pascanu, T. Mikolov, and Y. Bengio, "On the difficulty of training recurrent neural networks."
International conference on machine learning. Pmir, 2013, pp. 1310-1318.
[11] K. Doya et al., "Bifurcations the learning of recurrent neural networks 3," learning (RTRL), vol. 3, p. 17, 1992.
[12] T. Haarnoja, A. Ajay, S. Levine, and P. Abbeel, "Backprop kf: Learning discriminative de- terministic state estimators," in Advances in neural information processing systems, 2016, pp. 1376-4384
[13] R. Krishnan, U. Shalit, and D. Sontag, "Structured inference networks for nonlinear state space models," " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1, 2017.
[14] G. Chang, P. Murphy, and M. Jones, "On diagonal approximations to the extended kalman filter for online training of bayesian neural networks," in Continual Lifelong Learning Workshop at ACML 2022, 2022.
[15] M. Jones, T. R. Scott, M. Ren, G. Elsayed, K. Hermann, D. Mayo, and M. C. Mozer, "Learning in temporally structured environments," in The Eleventh International Conference on Learning Representations, 2022.
[16) M. Watter, J. T. Springenherg, J. Boedecker, and M. Riedmiller, "Embed to control: A locally linear latent dynamics model for control from raw images," arXiv preprint arXiv: 1506.07365, 2015.
[17] Y. Burda, Grosse, and R. Salakhutdinov, "Importance weighted autoencoders," arXiv preprint arXiv: 1509.00519, 2015.
[18] M. J. Johnson, D. K. Duvenaud, A. Wiltschko, R. P. Adams, and S. R. Datta, "Composing graph- ical models with neural networks for structured representations and fast inference," Advances in neural information processing systems, vol. 29, 2016
[19] M. Fraccaro, S. Kamronn, U. Paquet, and O. Winther, "A disentangled recognition and nonlinear dynamics model for unsupervised learning," arXiv preprint 1rxiv:1710.057411 2017.
[20] Y. Li and Mandt, "Disentangled sequential autoencoder," arXiv preprint arXiv: (tt::/803.09999 2018.
[21] A. Klushyn, R. Kurle, M. Soelch, B. Cseke, and P. van der Smagt, "Latent matters: Learning deep state-space models," Advances in Neural Information Processing Systems, vol. 34, 2021.
[22] Tonolini, N. Aletras, Y. Jiao, and G. Kazai, "Robust weak supervision with variational auto-encoders," in International Conference on Machine Learning. PMLR, 2023, pp. 394 34 408.
in International Conference Machine Learning. PMLR, 2023, pp. 42 998-429966 961. [23] H. Zhu, C. Ralsells-Rodas, and Y. Li, "Markovian gaussian process variational autoencoders,"
[24] P. Becker, H. Pandya, G. Gebhardt, C. Zhao, C. Taylor, and G. Neumann, "Recurrent kalman networks: Factorized inference high-dimensional deep feature spaces," in International Conference Machine Learning. PMLR, 2019, pp. 544-552.
11

---

[25] M. Schirmer, M. Eltayeb, S. Lessmaan, and M. Rudolph, "Modeling irregular time series with continuous recurrent units," in International Conference on Machine Learning. PMLR, 2022,
pp. 19388-19400 405.
[26] Hochreiter and J. Schmidhuber, "Long short-term memory," Neurul computation, vol. no. 8, pp. 1735-1780, 1997.
[27] K. Cho, Van Merriënboer, D. Bahdanau, and Y. Bengio, "On the properties of neural machine translation: Encoder-decoder approaches," arXiv preprint arXiv: 1259, 2014.
[28] R. Wilson and L. Finkel, neural implementation of the kalman filter," Advances in neural information processing systems, vol. 22, pp. 2062-2070, 2009.
[29] Buchnik, G. Revach, D. Steger, R. J. Van Sloun, T. Routtenberg, and Shlezinger, "Latent- kalmannet: Learned kalman filtering for tracking from high-dimensional signals," JEEE Trans- actions on Signal Processing, 2023.
[30] G. Revach, N. Shlezinger, X. Ni, A. L. Escoriza. R. van Sloun, and Y. C. Eldar, "Kalman- net: Neural network aided kalman filtering for partially known dynamics," arXiv preprint vrXiv:2177.10443 2021.
[31] D. Ruhe and Forré, "Self-supervised inference state-space models," arXiv preprint arXiv:2107113349 2021.
[32] T. Liu, K. A. Lee, Q Wang, and H. Li, "Disentangling voice and content with sulf-supervision for speaker recognition," Advances in Neural Information Processing Systems, vol. 36, pp. 50 2012205022 236, 2023.
[33] M. P. Deisenroth and H. Ohlsson, "A general perspective gaussian filtering and smoothing: Explaining current and deriving new algorithms," in Proceedings of the 201 American Control Conference. IEEE, 2011, pp. 1807-1812.
[34] P. Gilabert, G. Montoro, and E. Bertran, "On the wiener and hammerstein models for power amplifier predistortion," in 2005 Asia- Microwave Conference Proceedings, vol. 2. IEEE, 2005, pp. 4-pp.
[35] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, "Empirical evaluation of gated recurrent neural networks on sequence modeling," arXiv preprint arXiv:1412.3555, 2014.
[36] N. Srivastava, Mansimov, and R. Salakhudinov, "Unsupervised learning of video represen- tations using Istms," in International conference on machine learning. PMLR, 2015, pp. 843-852.
[37] H. Li, S. Yu, and J. Principe, "Causal recurrent variational autoencoder for medical time series generation," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 2023, pp. 8562-8570.
[38] Chung, Kastner, L. Dinh, Goel, C. Courville, and Bengio, "A recurrent latent variable model for sequential data," Advances in neural information processing systems, vol. 28, 2015.
[39] D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski, "Deepar: Probabilistic forecasting with autoregressive recurrent networks," International Journal of Forecasting, vol. 36, no. 3, pp. 1181-1191, 2020.
[40] A. Geiger, P. Lenz, and R. Urtasun, "Are we ready for autonomous driving? the kitti vision benchmark suite," in 2012 IEEE conference computer vision and pattern recognition. IEEE, 2012, pp. 3354-3361.
[41] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, "Unsupervised learning of depth and ego- motion from video," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1851-1858.
12

---

[42] S. Wang, R. Clark, H. Wen, and N. Trigoni, "Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks," in 2017 IEEE international conference on robotics and automation (ICRA). IEFE, 2017, pp. 2043-2050.
[43] K. Sohn, H. Lee, and X. Yan, "Learning structured output representation using deep conditional generative models," Advances in neural information processing systems, vol. 28, 2015.
[44] D. Angeli, E. D. Sontag, and Y. Wang, "A characterization of integral input-to-state stability," IEEE Transactions on Automatic Control, vol. 45, pp. 1082- 2000.
[45] R. A. Horn and C. R. Johnson, Matrix analysis. Cambridge university press, 2012.
[46] D. P. Kingma and Ba, "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980, 2014.
[47] J. Werbos, "Backpropagation through time: what it does and how to do it," Proceedings of the IEEE, vol. 78, no. 10, pp. 1550-1560, 1990.
[48] J. L. Ba, J. R. Kiros, and G. E. Hinton, "Layer normalization," arXiv preprint arXiv:1607.06550. 2016.
[49] J. M. Wang, D. J. Fleet, and A. Hertzmann, "Gaussian process dynamical models for human motion," IEEE transactions on pattern analysis and machine intelligence, vol. 30, no. 2, pp. 283-298, 2007.
[50] Ko and D. Fox, "Learning sp-bayesfilters via gaussian process latent variable models," Autonomous Robots, vol. 30, no. 1, pp. 3-23, 2011.
[51] R. Frigola, F. Lindsten, T. B. Schön, and C. E. Rasmussen, "Bayesian inference and learning gaussian process state-space models with particle meme," Advances neurul information processing systems, vol. 26, 2013.
[52] M. Schoukens and K. Tiels, "Identification of block-oriented nonlinear systems starting from linear approximations: A survey," Automatica, vol. 85, pp. 272-292, 2017.
[53] R. Li, S. John, and A. Solin, "Improving hyperparameter learning under approximate inference in gaussian process models," in International Conference on Machine Learning. PMLR, 2023, pp. 19 596511615.
[54] N. Wahlström, T. B. Schön, and M. P. Deisenroth, "From pixels to torques: Policy learning with deep dynamical models," arXiv preprint arXiv:/502.0222 502.02251, 2015.
[55] E. Archer, I. M. Park, L. Buesing, J. Cunningham, and L. Paninski, "Black box variational inference for state space models," arXiv preprint drXiv:1511.07367, 2015.
[56] M. Karl, M. Soelch, J. Bayer, and P. Van der Smagt, "Deep variational bayes filters: Unsu- pervised learning of state space models from raw data," arXiv preprint arXiv: arXiv:1605.0m 1605.0643, 2016.
[57] C. Naesseth, Linderman, Ranganath, and D. Blei, "Variational sequential monte carlo," in International conference on artificial intelligence and statistics. PMLR, 2018, pp. 968-977
[58] S. S. Rangapuram, M. W. Seeger, J. Gasthaus, L. Stella, Y. Wang, and T. Januschowski, "Deep state space models for time series forecasting," " Advances in neural information processing systems, vol. 31, 2018.
[59] V. Garcia Satorras, Z. Akata, and M. Welling, "Combining generative and discriminative models for hybrid inference," Advances in Neural Information Processing Systems, vol. 32, 2019.
13

---

A Appendix
A.1 Filtering and Smoothing Parameterization
The Kalman filter operates through two iterative steps: prediction and update. In the prediction step. the filter uses prior state information to make an estimate, while in the update step, it adjusts this estimate based on new observations. Assuming normally distributed additive process and observation noise, the filter can effectively perform these steps. During the prediction step, the filter employs the transition matrix F to estimate the next priors Pria1 (X++1 (i:e) = (14+1]6: S(+1)), which represent the next state estimates without incorporating any new observations.
Mt+1/t F F and E.11 F F Qr+1. and Q++1 2241
(13)
When new observation is available, the Kalman filter proceeds the second step, where it updates the predicted prior using the new observation and the emission matrix H4+1. This results in the next posterior, (p++1]e+1, E++1(++).
K++1 E4nt E + Re41)
(14)
M4414411 = Pc+1je + Keti(we- He+10e-11)) E441441 Estije K.+i (He41 1 E4 H + R R K++1-
(15)
The entire observation update procedure can viewed as a weighted mean between the next prior, derived from the state update, and the new observation. This weighting is influenced by Q and R. reflecting their inherent uncertainties. We derive the smoothing parameterization by leveraging the Markov property, which states that x; is independent of future observations Wr+1: given X++1. Although X is unknown, there is distribution over it. By conditioning on X and then marginalizing out, obtain x, conditioned on W1.7-
f Pvee (Xe X WLT (Xe+ (Wir)dxx++1
Pur (Xt (WiT)
(16)
= Pri (Xr) Wia, Wa Prir (xe+ (Wir)/li))dda
By using induction and and smoothed distribution for &
Pur (X++1 (i:T) ((M+ijT E(1/T
(17)
we calculate the filtered two-slice distribution as follows:
Prii (Xe, X [1++) = Nr( (( Pele
( Edee F Egje
EeeFTh Eet1le
(pettle)
(18)
by using Gaussian conditioning we have:
Price X(+1, W1:)) N(uge Jx(x++1 F+4144)) Egj: J.Een1J??)
(19)
where J, = Fet ( -1. We calculate the smoothed distribution for using the rules of iterated expectation and covariance:
Mgj
E X X1+1, Wirl [wir]]
E E X X1+1, Wial [wir]]
(20)
E Maje + Je(x++1 F(-1MH)) [Wir]
= Maje Je(4411/ F F(+1M9))

---

2Tp cov B Xx+1 Wi:T]
Wi.7] E Elcov cov(cx x,, Wi:T] [Wir]]
cov
[ 1 Wia
WI:T] + I2 [eoooo x Xx+1, Wi:e] [Wir]]
cov
(ugp + J. (x+11 Px41/40))) |WiT]] + R E J |Wir]]
(21)
J,cov [x(+1 F 1441/e |Wi.]] + Eqje J, J.
-J Eqee J.
Eqi + J. E1+1/T [.111e)J7
A.2 Process Noise Matrix
As stated in
we can elaborate the process noise matrix at time t more details
Q Eije1 F = -1p-11 10--1-Fiy [ K, H. E, 1/1 2 H.
R R
| F?
-'K
(22)
combining
13 into
22
results
Q: = Enp-1 F, [F 1-1 E, 2 + Q:-1]
(23)
- [ F E, -1 FF + Q Q.-1 Ba-1]-KK___ F?
which is function F., Q.-1, F and H the GIN, F, and H; are learned by the Dynamic Network with the input of X4-1 From [15,, M is derived funetion of both F,-1 and
HІ-1. meaning the learned F, carries the information of both H4-1 and FŞ-1. Therefore, one can rewrite the equation 23
Q: gg (P ...1)) where F, Dynamic Ner 11-1 (H--I--((((....-))) F..1)
(24)
where g is a nonlinear function mapping 14-1 and Q to Q and the graphical model for such choice of structure is in appendix figure 8b It possible go step further and simplify /4-1|tt-1 more, as it has E.-it-1 term 15 combining it with (13) results in:
Pt-1jt-1 = P1-1/t-2 Fr Ex- F + Q.-1]
(H.
Q Qu-IHH?1 + R4-1) ?w, H4-1/4-
(25)
indicating that not only F,-1 and H,-1, but also Qr- is included in Pt-1jt-1- meaning that Q2 can be written solely as function of Me-1je-1 and the graphical model for such choice is in figure 8a
Q = (F where F, = Dynamic Ne ( Fr-1, Q Q..1).
(26)
We refer to as the Q Network, which can be modeled either by an MIP, as shown in (6 or by recurrent network, as detailed in (2).
A.3 Output Distribution
In the case of grayscale images, each pixel is either one or zero with the probability 1 p. respectively, meaning that P(Y = y) = -- This probability equation can rewritten
the form of an exponential family as follows:
15

---

M
Wp-1
We-t
a Without recurrent dependency qe modeled with (b) With recurrent dependency modeled with
26
Figure Graphical models for different parameterizations of the process noise.
fo(y) hry))erp(O.. (0)) dog(t" e lool [5o(==1)+ tog(1-p),
(27)
By choosing = log( T and (0) = log(1 - p) and h(y) = 1, we can obtain p I+e-0- This means that by considering as the last layer of the decoder and applying sigmoid layer, p is obtained and one optimize p such that the likelihood is maximized
did in (1I.
Similarly, consider x as the ground truth state, f as the estimated state, and as the model variables. 28 + N (â ,ā); where da is the estimated
The residual follows Gaussian distribution: variance. Then, the negative log likelihood is given by (28) as we obtained it in 10).
(log(os) (x 5o
-Cog(C) x 2
(28)
25g
A.4
Proof of theorems
We restate the theorems for the ease of readability and then provide the proofs.
A.4.1 Proof of theorem
Theorem. The log likelihood of the states given the original observations is determined as:
T log s, (1
C(s1r/o11:7) N log p x 1
( 1 OIT)
1
N
where the d.).) and do(.) determines those parts of the decoder that obtain the state mean and variance, respectively. N sequences of (x11) w(() (8)
(ir) Iur (XI.T, Wirloo1:) are sampled for
Monte Carlo integration estimation.
16

---

Proof.
log Peur (S1T/011)
=/ n ( Wirro1r7 (177) log Par (S11: W1.7. X1.7 01:1)
dxirdwier
(29)
Perr (1:T WI.7 $1:7; 01:7)
= =f (x(x,, Wi.roorr, (o W
-x Wi S1:T) log 96 (XI.T WT: (1:7,111
(xLT, Wi.Tor17 SET) log Pri:r (1:77 W1:7 W
dxirdwIr
(30)
Prir (1.7) W W S1.7,, 01.T)
-KI Wirlorr S1.T) Pm (XirT, Wir1
+ +f (X.T. Wir 11.7, log
P (1 WT: VI.T,
(31)
9 (XITT, Wi:7 (1T,
Pyr (81.7, WI.7, X1:T(o1)))
2 (x
log
(32)
46 (x17 W1.7 S1:T)
Py ($1.T WI.7 X1:T: 01:7) O1.) Phr ( (1.
log
(33)
46 (xrT, W1.7 01:1,
=
log Pm ($1:T W X1.7, 01:7]
(34)
Eg
log Pour (51.T W1.7 X1 X1.r) 01:T)
(35)
E or (ei (wir) (wrl
log Pru.r W1,
(36)
N
(2) log pl
XIT. w 1:7 01.T),
X1.T
(2)
9m (x w wir),
w 1:
-p(Wirloo (1:r)
(37)
N
log.V St (m
(38)
N
L($1.7 (LTT)
(39)
The inequality in 322 arises from the positivity of the Kullback-Leibler (KI.) term. The approx- imation in (34) results from assuming equality between the prior and the approximated posterior, i... P (W1.T, X1:7 (o.T)) 4o (X1T, 01.T. $1.)). This assumption is very restrictive
as explained further in 43 The approximation in 335 is coming from the equality assumption
of 90 (X1:T, W1.7 01:r, S1:7) and fur (X1:7, W1.7 01:7 This assumption implies that given the
original observation 01.7, our latent representation (X1.7, WI.T) is independent of the ground truth states $1.7. In simpler terms, $1:7 concealed within 01:7, allowing us disregard $1:7. The approximation in (37) is because of Monte Carlo integration estimation.
A.4.2 Proof of theorem 2
Theorem. The (lower bound of) log likelihood of the original images is:
De o (8) log (1-o{ (A(()))
L(OITT) N
log 1og
N
(40)
1:r) 9m (XLT, Wi.loour for Monte Carlo integration estimation. dx(.) detines the corresponding part of the decoder that maps the k-th pixel of image.
with N sequences of (x
1.T
(
17

---

Proof.
log Praer (01:T)
/ Our (XI.T, (irlooiirro.r... log
Prir (W1:7, X1:7; 01:7)
dxp:TdWi:T
(41)
Peur (XI,T, W (1T
f Inr (Xi:T, W1:7 log ynre (xir1 W1:7 01:1)
-nir (X1r,, Wirr o1: log 9m XI:T Wi:2011:1
+ qni (Xi:r Wirr (1:r) log Pr Wi.r, X1:7, 01:7)
dx1:rdW1:1
(42)
Pyee X1:T, W1.7 (1:T)
(L (Xie,, Wirroorr (P (XI7) Wi.r 01:T))
+ Una (XEr, Wirrlo1:7 log Pri.r Wi:Tr X1:2; 01:r)
dx
(43)
Unr (X::7 W1.r 01:7
[log Peur ( Wi.T; X1:7; O1:T)
2 E (XLT (irloor)))
(44)
Unr (X1:7, WT.7 01:7)
llog Pri.r (O1:2 W1.77 X1:7 (aa:))Pmm.r Prer (Wirs X1:1)
E (Kir. (irloorr))
(45)
Pur (XITT Wi:7/1:7)
2 y ( wirlorr))
log Pai (O1.T WI.,, X1:)]
(46)
E E (x (wt) Pogrr (0::T WI:T, W1:r)
(47)
(1) log logp(orr) w(T)), X1.7 Ynr (X (0) wit p(WI.(o1)))
(48)
D. o log(dd
(k)
(1 o, (k) log(( (x(
(49)
((1TT)
(50)
The inequality in (44) arises from the positivity the Kullback-Leibler (KL) term. The approxima- tion in (46) results from assuming equality between the prior and the approximated posterior, i.e.,
Pyur Wi:r,X1:21 = 2 (x1:1 W1. 01:r). Although it feasible to optimize (44 directly, we found out that by setting prior Pr.r (X1-7, WI.1) equal to approximated posterior 97 (X1:7 W1.r 01:1),
more stable results with concise improvement are achieved from the numerical results (similar to
19 21 43
The approximation in (48) is because of Monte Carlo integration estimation.
A.4.3 Proof of theorem 3
First we provide brief review on the GRU cell equations from 35 Then we introduce two lemmas, by which we can proof theorem 2.
GRU Cell Review.
The GRU cell holds the following equations:
2t Sigmoid( U Xx + U,h:-1);
(51)
Г; Sigmoid( U,y X₁ + U,h::1),
(52)
h; 2, h1-1 (1 2z) he
(53)
h, tanh(( U Un(r, @ h.-1))
(54)
where x, and h, are the input and the hidden state vectors, respectively. Ur.. U., U U, Uzh and U, are all weight matrices.
Lemma 1. A GRU cell has fixed point at h- 0, if the input variable x, 0.
Proof. After substituting x, = and h into (1 and 52). the update gate z, and reset gate
are both J. Substituting these values along with x 0 and h 1-1 11 0 into (54) gives the candidate
18

---

state hy = O. Finally, substituting h = 0, 2, 3. and h, 0 into (53) results in the new state h, 0. Thus, h, h O. indicating that GRU a fixed point h 0.
Lemma 2. Let be an h x h identity matrix, A²(-) shows the i-th largest absolute eigenvalue, and A = U + I. When the spectral radius [Ar) < 1, a GRU cell with input x; 0. can be approximated by the following linearized GRU near h, 0:
h Ah,-1
(55)
and the fixed point h" 0 is locally stable.
Proof. By emploving Taylor series expansion and linearizing h, with respect to h at h
X; = 0. A in (55) is obtain as _ + I. Then, from (55) we get that h; = A A'ho- Since A dependent on the t-th powers of the eigenvalues of A, the behavior of the linearized GRU is determined by the eigenvalues of A. According to the Martman-Grobmin
theorem, the behavior of a dynamical system near a hyperbolic fixed point is homeomorphic to the behavior of the linearized system. Therefore, when (A)) < 1, the fixed point h' = becomes a hyperbolic fixed point. Thus, GRU cell can be approximated as h, A hoo Then, local stability determined by the spectral radius [A1(A) < and then we have hh, holt-so = 0 and the fixed point h" = 0 is locally stable.
Theorem 3. When o1 (U,) < 2, a GRU cell is locally stable at a fix point h" = O.
Proof. Relying on Lemmal and Lemma2, we need to satisfy (AI + +1Dn < (AI(UU ! (U,) + bl due to the properties of eigenvalues. From the triangle inequality we also have (Ux)) + & ( 2. Using Weyl's inequality for its singular and eigenvalues, we have (U,)) ( and therefore (U,,) + 1/4 S (U,)) + 2. Thus, we need to satisfy (U,) + 2 < condition and conclude (U,)) + I ( + 2 < 1. This can be achieved by (U,,) < 2.
The GRU cell is considered as nonlinear dynamical system with h, fh, xx) model, where one can drive I from 51 51). We have shown the conditions for local stability of h f(h, 0) in the last paragraph. Now we show = (h, x) is locally stable if x, + 0 as t - co.
As we have the stability condition for h [h, 0), there exists V(h) that is Lyapunov function for this system such that:
dV f(h 0) < WWhh)
V(h) 2 0, and V(h)
(56)
dh
for a positive definite Wh). When the input variable x is not zero, the system equation becomes h = fhh x) and the derivative of Vhh) becomes
v(h) = dV dh f(h x)
(57)
but for small x,, we can write:
V(h) < WWhh) +e,
(58)
dV
c= dh
(f( f(h,
(59)
and 0 as x, + O, utilizing Taylor series expansion of [(, x) near x = 0 [4]. Then, as - 0, for any e 2 0, there exists a T such that for all 2 T, |xel < e and therefore we have:
(hh < - W(h) +
(60)
for sufficiently small c, (hh remains negative.
Therefore, we need to demonstrate that the inputs of the GRU cells in (6) and 8 approach as
( + xo. As discussed in the main script, we apply a Wishart prior distribution on the covariance matrices, guiding them towards scaled identity matrix. Over time, this scale diminishes towards zero, causing the covariance matrices-ant consequently, the zero bias Conv(.) operators in (6) and
19

---

to converge to 0 as + 30. This implies that the system tends to behave deterministically after prolonged observation, while the stability of the GRU cell is guaranteed. Informally, Theorem 3 requires us to asymptotically shrink the input covariance matrices of our GRU cells toward zero. While this shrinkage is reasonable as we expect the system to behave deterministically in the long run, it imposes limitation on posterior inference. We acknowledge this as an open problem for future research.
8
A.4.4 Proof of theorem 4
Theorem. The modified weight matrix Un obtained from (iii) step above, is solution of the following optimization problem: minu, |U, Unl s.t. (1(U)) < 8.
Proof. Relying on 45 due to the matrices properties we have:
o, < Ü Unl
(61)
Assuming that the first singular values of U, are greater than 2, we have > i= (o;(Un)
(U,) = Ei= (U,) (2 6)]. According to 61p. we need to show that Σ (o:(Un)
(2 [2] is equal to its upper bound Ü U l1 i.c the optimum solution.
Using the singular value decomposition properties, we have | U
[or::tr(VVM - (A))) - (A - X) = tr((A - () A)) -
A.4.5 Proof of theorem 5
Theorem. We can satisfy the constraint introduced in Theorem by applying the Gershgorin circle theorem as follows.
(i) In the i-th row of Un, sum up the non diagonal elements is Ei Úx(i,)) Do (i) for the all rows.
(ii) For each (i, i) assign circle cluster centered (i, with ia radius in the complex plane.
(iii) Merge the clusters which have intersection and repeat step (iii) until no cluster can be merged any more. Finally the set of clusters C [C1, .... C.) with length of k S h is obtained.
(iv) Define the radius of Cj cluster as de, = max (i, +ia VU (1, Cý). Calculate the radius of all & clusters step (iv).
(v) For each cluster C,, (i, i) + de, (i, E C2) > shift (t, a) + de, 5) for a small 8. Otherwise, keep Ú (i, unchanged. Do step (v) for all clusters and their elements, i.c. (i,
These five steps, ensures the new U, follows the constraint introduced in Theorem 3.
Proof. The five steps elaborated above are direct results of the generalized Gershgorin circle theorem. This approach is faster than SVD method introduced in the paper, while it is not the best solution of the optimization problem introduced the Theorem 4.
A.5 Discussion About KL Term of Mode Collapse
The term introduced in the mode collapse handling section is:
K (L Ip priir (F)) + (L(p((
(62)
Here (F) MN xx (F. III) and p,(F) MN nxn. I,), are matrix normal distribu- tions with priors (F) respectively. We can expand 62 as:
MN axn (mag and pri,(F) -MN (Diag(m,),1,
20

---

=f PF(F) log (F))pri,
log Ps(F)
(Fi dF pj pi(F).pri,(F)) dF
(63)
(F(
Pi(F)( (lo
(64)
pFi,(F)
= p. p.(F) (B))
of + pi (1o(pr)))))) oF - pi(F) (logpri(E)) dF
(65)
Relying on 45 we can simplify each term (65) as:
f pi(FF (1gg)) dF = - (vec(FF)) vec(F
(vec(F")--eee (vec))) + (r(T)] + const
(66)
2 = | -P'-F. 11
(r( const
(67)
2
2
f pi ()(F)(oogrri,)))) oF
1
[(vec(F") -ee((aag((() (vee((")))))) (vecF")-vvoDDagmm + tr(I)
(oc(Diag(mm)))
const
(68)
| Diag (m;) 2 (I( + const
(69)
f pi (i((F) (1ogmmi))))()))
1 (vec(F)) vec(DDag(mm))) (vec(F") vec(Diag(m;))) + (r(()) + const
dF
2
(70)
F' Diag mm))1
2 tr(())
(71)
2
Thus, we can rewrite 62 as:
F (m. )|
2
F - Diag (m;
||F' F
trr(1) const
(72)
2
which is half of (12) plus constant.
A.6 Noise Generation Process
In our experiments, we use a time-correlated noise generation scheme demonstrate the system's noise robustness. This method introduces a sequence of factors. ft, of the same length as the data sequence, making the noise factors correlated over time. Let fo U(0, 1) and se+1 min(max(O, ft rz), 1) with rt ~ U( 4(-0.2, 0.2). where fo is the initialized factor and U is the uniform distribution. Two thresholds, t (1, 0.25) and 1o ~ (0.7755 1), are defined. Values of fe less than & are set to O, values greater than 12 are set to 1, and the remaining values are linearly scaled within the range [0, 1]. The original observation at time t, Or, is then given by o, = fci, + (1 121-f() f.) where i, is the true image at time and i!Th is the pure noise generated for time f.
A.7 Hy perpaaaarsts and training
In all experiments, we used the Adam optimizer [46] on an NVIDIA GeForee GTX 1050 Ti with 16GB RAM. To ensure optimal hyperparameters, we conducted a grid search. We searched for the initial learning rate in the range of 0.001 to 0.2 with increments of 0.005, selecting the one that yielded the highest log- log-likelihood. We chose an initial learning rate of 0.006 with an exponential decay rate of 0.9 every 10 epochs. Backpropagation through time 47 was employed to compute gradients, given the use of GRU cells in our structure. The gradients are applied to GRU cells with the constraint explained in theorem 3, where we use the spectral theorem mentioned the main script
We applied the layer normalization technique [8] to stabilize the dynamics within the recurrent structure and normalize the filter response. The Elu activation function was used to ensure the positivity of the diagonal elements of the process noise and covariance matrices.
21

---

GIN
Encoder
Filtering
Smoothing
Decoder
[0
Tin."aa)
Prediction
Puror- Surbur-
04, erear S m
Dynamics
Care
GRIJAG
NuT
Netwerk
Pun Der
THr
Mi7
Filte ring Stop
Smoothing Step
Prediction
PPrpp Buro7-)
Per-jee--
7-13711
P Peren
Figure Proposed detailed architecture.
To prevent the model from getting stuck in poor local minima, such focusing excessively on reconstruction rather than learning the dynamics obtained through filteriggssmoothing we find beneficial to employ two training techniques for end-to-end learning:
1- Generating time correlated noisy sequences as consecutive observations, forces the model to learn the dynamics instead of focusing on reconstruction, c.g. figures and appendix.
2- During the initial epochs, the system focuses learning the auto-encoder and globally learned parameters, such as F and H while excluding the Dynamic Net parameters a,(x;,) 1). Afterwards, all parameters are jointly learned. This approach enables the system to initially acquire good embeddings and meaningful latent vectors before integrating the learning of K different dynamics variables.
In our experiments, we set K = 15 to accommodate the complexity of the latent space, with each latent representing a unique dynamical scenario. Generally, parameter tuning is not challenging when the GIN is sufficiently flexible, it can learn to prune unused elements through the Dynamic Net.
A.8 Proposed architecture, qualitative comparison with the SOTA and empirical complexity analysis
A.8.1 Architectures
The proposed detailed structure is shown in figure 9 To design the dynamic network, we use an MLP with 60 hidden units and ReLU activation function, and a softmax activation in the last layer. The state mean, with a size of n, is the input to the dynamic network, which outputs & coefficients. The structures of the encoder and decoder are shown in Table In this table, m represents the transferred observation dimension, with various values considered in the results. For state estimation tasks, the output size (out)
4, and for the single-pendulum, bouncing ball, double- pendulum and KITTI visual odometry experiments, respectively. For the imputation task, the number of hidden units in the KG and SG networks set to 40 and 30, respectively. The convolutional layer applied over the covariance matrix has filters with kernel size of 5 and zero bias.
22

---

Table 5: The structure of the encoder and decoder.
Encoder
Decoder
if state estimation task:
6 x 6 Conv, 12, max pooling 2 x 2, stride 2 x 2 LayerNormalizer))
Sy: fully connected: out, linear activation sg: fully connected, Elu + zero bised
4 x 4 Conv, 12, max pooling 2 x 2, stride 2 x 2 LayerNormalizerr)
if imputation task:
fully connected: 144
fully connected: 40
6 x 6 Trrss Conv, 16, stride 4 x 4
w: fully connected: vn linear activation
LayerNormalizer()
fully connected, Elu 1 activation
4 x 4 Trrss Conv, 12, stride 2 x 2 LayerNormalizer()
o, 1 x 1 Trms Conv, stride 1 x 1, softmax
Gradient Explosion Experiment.
Referring to table higher values of ó deteriorate performance because the convergence speed of the linearized GRU hidden state depends ai (Un)) 1.) (see Eq. (55) in the appendix), which is projected to 2 - 8. Therefore, longer time dependencies in the data are eliminated for high 6. On the other hand, smaller values of 8 close to zero make the model more prone to noise [[O]. Therefore, is necessary
to tune an appropriate 5 within the range 0 < 6 < 2, which is much easier than tuning the unbounded
A.8.2 Empirical running times and parameters
We present the number of parameters for the utilized cell structures in our experiments and their corresponding empirical rumning times for epoch in Table In the first row of each model structure, we set the number of parameters approximately equal to our GIN to demonstrate the GIN's superior performance with the same parameter count. The extra running time of EM- approaches, like KVAE, is due to the use of classic Bayesian equations, which significantly increase running time for nigher-dimensions parameters in the GIN is noticeably lower than in other memory cells, such as LSTM and GRU, and EM-variational methods. This efficiency achieved by converting high-dimensional sparse covariance matrices into lover-dimensional covariance matrices using a convolutional operator.
observations. However, the GIN avoids this issue. The number of
A.8.3 Qualitative comparison of the GIN to recent related work.
Switching Linear Dynamical System models (SLDS) decompose complex, nonlinear time series data into sequences of simpler, reusable dynamical modes. Fitting an SLDS to data enables us to learn flexible nonlinear generative model and parse data sequences into coherent discrete units. [9] is a SLDS model that incorporates additional latent variables to switch among different lincar dynamics. However, this approach relies on Gibbs sampling for inferring the parameters, rendering impractical for large datasets due to scalability issues.
Auto-regressio state space models (ARSSMs) are class of dynamic models commonly used in time series analysis and forecasting. In these models, the evolution of a system over time is described by set of states, by observing all data. Auto-Regressive (AR) Hidden Markov Models (AR-HMM) explain time series structures by defining mapping from past observations to the current observation. 39 presents an ARHMM approach where target values are used directly as inputs. However, this reliance on target values makes the model more susceptible to noise.
Toward learning state space (system identification), a number of works including those by [49 53 have proposed algorithms for learning Gaussian Process SSMs (GPSSMs) through maximum likelihood estimation using the iterative EM algorithm. where
with a set of fixed parameters y is conducted (E step), and then updating the set parameters y is performed such that the obtained likelihood is maximized (M step). Frigola et al. [51] obtain sample trajectories from the smoothing distribution and then, conditioned on this trajectory, conduct the step for the model's parameters. In the context of System Identification (SI), the GIN is akin to a Hammerstein- Wiener (HW) model [ as estimates system parameters directly from observations while transferring the observations through nonlinear functions.
23

---

Table Empirical running times and parameters of experiments.
Single
Pend
Double Pend
Bouncing Ball
Visual Odometry
Cell
Param
T/E
Param
T/E
Param
T/E
Param
T/E
LSTM (m==5)
~ 18k
556
~ 18k
~56s
~ 18k
33s
455
~838
LSTM (n=50)
~36k
70s
~36k
71s
~36k
~68
70k
~95
LSTM (n=1000
766
98s
~66
~96
~66k
933
~120
131s
GRU (n=30)
18k
~6s
~ 18k
62s
~ 188
~59s
422
~99
GRU (n=50)
~27
~65
-27
~67
27k
64s
~53k
84s
GRU (n=100)
~57
~66
~57k
~55
557
84s
~
99k
llll
VAE (n=40)
112k
550s
133
52s
122
44s
29k
70s
IWVAE (n=40)
133k
50s
~13
54s
11k
47s
30k
755
VAE-RNN (1=40)
24K
~59s
25k
~6s
24k
575
49k
899
SVAE (x=40)
27k
~67s
27k
~69
26k
665
68k
149s
KVAE (n=40)
255
~95s
25k
~97
25k
-94
62k
~ 141s
KVAE (n=60)
~36k
1144
366
111s
36k
~155
80k
165s
EKVAE (n=40)
266
~98s
~
26k
~99
26k
~
~ 65k
1458
EKVAE (n=60)
~37k
~166
38k
113s
35k
112s
83k
170s
MKVAE 40)
344
~
1122
34k
~ 110s
333
105s
~ 77k
153s
RKN (n=
~55
577
25k
58s
25k
56s
45k
79s
CRU (n=100)
244
555
244
555
233
54s
466
78s
Linode-Decoder (n=30)
100k
445
~ 10k
44s
~ ~9
444
21k
6638
LGSSM (n=30)
12k
~2s
122
~84
~ 12k
800
33k
1175
LGSSM (n=45)
~ 15k
998
~ 15k
975
~ 15k
975
~36
~ 136s
GIN (n=30)
18k
~55
18k
555
18k
554
42k
80s
GIN (n=45)
25k
~59s
25k
558
25k
557
488
~83
In Table 7 we compare various algorithms to evaluate their ability to handle high-dimension observations, Iearn dynamics, estimate state accurately, provide uncertainty estimates while handling noisy data, manage missing data, and perform direct optimization. Classic LGSSMs, such as the EKF and UKF, rely on the linearization of transition and emission equations and apply classic Bayesian updates over the linearized system with respect to the states. In other words, classic LGSSMs are masel- contrast, the GIN uses data-driven network to learn parameters, offering a different approach from the traditional moddel-based methods.
Table 7: Learning the parameters in LGSSM is shown with /V because general LGSSMs, e.g. UKF and EKF, are not able to learn the parameters. However, in our setting and parameterization we use data driven-based network for obtaining (F, H) to make LGSSMs comparable with the GIN for high dimensional observation experiments.
Modal
high
learn dynamics
missing imputation
state est
uncertainty
noise handling
dir
LSTM GRU
P2T
4
RXC
B8-VI
SIN
DVBF 56
VSMC 57
XVAF
EKYAK
MVAF 3
SSLSD 0
DeepAR
HybridGNN
KalmanNet
LGSSM
X/V
GIN
24

---

A.9 Visualization and The Imputation
Graphical results of informed, uninformed and noisy observations for image imputation task for both single and double pendulum experiments can be found in II
I and 13 figures. Inference for the
trained smoothing distribution of all experiments are in 14
15 T6
19 20
21 22 23 and
24 figures, where we generated samples from the smoothing estimated states distribution, (St O1.T) Then we fit density on the generated samples. This visualization shows the effectiveness of the GIN in reducing the uncertainty of the estimates compare to LGSSM and K
25

---

Ground Truth
Ground Truth
Observed Sequence
Observed Sequence
LGSSM((ilte)
LGSSM((ilter)
LGSSM(smooth)
LGSSM(smooth)
GIN(filter)
GIN(filter)
GIN(smooth)
GIN(smooth)
Figure 10: Informed(left column) and uninformed(right column) image imputation task for the single pendulum experiments.
Figure 11: Image imputation task for the single pendulum experiment exposed to the noisy observa- tions, where the generated noise has correlation with the time. Each figure, beginning from top bottom, indicates the ground truth, noisy observation and the imputation results of the GIN.
26

---

Ground Truth
Ground Truth
Observed Sequence
Observed Sequence
LGSSM((ilter
LGSSM(filte)
LGSSM(smooth)
LGSSM(smooth)
GIN(filter)
GIN(filter)
GIN(smooth)
GIN(smooth)
Figure 12: Informed(left column) and uninformed(right column) image imputation task for the double pendulum experiments.
Figure 13: Image imputation task for the double pendulum experiment exposed to the noisy observa- tions, where the generated noise has correlation with the time. Each figure, beginning from top bottom, indicates the ground truth, noisy observation and the imputation results of the GIN.
27

---

platicue)
Figure 14: Inference for the single pendulum s1 position 100-th time step. Generated samples from smoothened distribution, 100 trained by the GIN, LGSSM and KVAE, respectively. The dashed red line p(g[l 100) is the ground truth state with distribution of 6(s1 100 -..7). We calculate the sample mean and fit a distribution on the samples for further visualization and comparison purpose.
Vig2 882 -0
LOGSON
KIKE
Figure 15: Inference for the single pendulum s2 position at 100-th time step. Generated samples from smoothened distribution, (s2100 01:150), trained by the GIN, LGSSM and KVAE, respectively.
Mob ARtE Est
05
(a) GIN
(b) LGSSM
(c) KVAE
Figure 16: Generated samples from the trained smoothened joint distribution the single pendulum
position, (sl, s2), at 100-th time step for the GIN, LGSSM and KVAE, respectively. The ground truth is shown with a black point.
28

---

High dist -0.00)
LGSSM
Figure 17: Inference for the double pendulum s1 position at 100-th time step. Generated samples from smoothened distribution, [sl 11:10) trained by the GIN, LGSSM and KVAE, respectively. The dashed red line p(9(l100) is the ground truth state with distribution of 8(s1 0.35).
2x2 dis2
LGSSOM
Figure 18: Inference for the double pendulum s2 position at 100-th time step. Generated samples from smoothened distribution, (s2100 01:150), trained by the GIN, LGSSM and KVAE, respectively. The dashed red line p(gt2100) is the ground truth state with distribution of 8(s2100 0.35).
24
(a) GIN
(b) LGSSM
(c) KVAE
Figure 19: Generated samples from the trained smoothened joint distribution of the double pendulum
first joint position, (s1, s2), at 100-th time step for the GIN, LGSSM and KVAE, respectively. The ground truth is shown with black point.
29

---

pig1 653
Figure 20: Inference for the double pendulum s3 position 100-th time step. Generated samples from smoothened distribution, [s3 01:10) trained by the GIN, LGSSM and KVAE, respectively. The dashed red line p(9(3100) is the ground truth state with distribution of 8(s3100 1).
Figure 21: Inference for the double pendulum s4 position at 100-th time step. Generated samples from smoothened distribution, (s4100 01:150), trained by the GIN, LGSSM and KVAE, respectively.
53
(a) GIN
(b) LGSSM
(c) KVAE
Figure 22: Generated samples from the trained smoothened joint distribution of the double pendulum second joint position, (s3, s4), at 100-th time step for the GIN, LGSSM and VAE, respectively. The ground truth is shown with black point.
30

---

42.0
-42.5
p 0(s1 50
Phossoo 43
-000
50.32
Viieisoo 52.25
-55.0
69.0
23.0
35.0-
12.86
32.5
10.08
Mooscoo 8.46
7.5-
p(g]1 = 5[sl 10
GIN
LGSSM
KVAE
Figure 23: Inference for the visual odometry and 82 positions at 100-th time step. Generated samples from smoothened distribution, (s1100, 22 01:500), 01,000)))00) trained by the GIN, LGSSM and KVAE, respectively. The dashed red line 100 gt2100) is the ground truth state with distribution of (8(s1100 + 50), 8(52100 - 10)).
A.10 MSE Results for The State Estimation and Estimated KG-SG
The MSE results for the single pendulum, double pendulum, and bouncing ball experiments are shown Tables a 9 and 10 addition to the dynamics equation (5), where the F matrix includes
the effects of the process noise, two other solutions introduced in Section 5 are included in the MSE
results. Parameterizing the process noise as in (24) with a GRU cell indicated by GRU(Q), while another parameterization, shown MIP( Q), is done using (26
31

---

123
(a) GIN
(b) LGSSM
(c) KVAE
Figure 24: Generated samples from the trained smoothened joint distribution of the visual odometry joint position, (s1, s2), at 100-th time step for the GIN, LGSSM and KVAE, respectively. The ground truth is shown with black point.
A.11 Algorithms and python inference code
Algorithm Training the GIN
Input: Observations 01.7, last posteriors (10-T-110:T111
(Xo.T- -[0---1()--))- = Sample (Bo..
10.T-1)0TT1))
01.7 Dynamic Net (Xo:T- -10---)
Obtain 1.T and H by (5)
(A1:roo:--1: E1:rorr--1 1) = Prediction (Po:T-1J0:T-11 1j:
(WITT, [TTT) encoder (01.T)
KiT = 1 M HTT M = Com (1 riT)
J - Epppr F F. F.. NIT GRUSG (Conv( Coonv(E1:10:77 (21:100011))
(ALTITT {.TTIT) = Filtering (P1.TTo:T-1, E - KIT, WI.,, H.))
(ĀITII, Σ1.ryr Smoothing (fTT 1:T: EnTTT, JI. FiT))
if (image imputation) then
01.7 decoder (ÁITTT, Σ1.7IT)
L(o = Lower bound likelihood (our) by using 10 end if
if (state estimation) then
S1.7 = decoder (pI.TY: ΣirT)
C(s1. (o.r) Lower bound likelihood (s1:r) by using
end if
Backward Propagation by updating GRU cells with the steps in Theorem 3 or Theorem
32

---

Table 8: MSE for single pendulum experiment.
Model
MSE
LSTM (units 25, m 15)
0.092+00003
LSTM (units = =25, m 20)
0.092=0.005
LSTM (units =25, m 40)
0.090=0.005
LSTM (units = -100,
15)
0.089=0.002
LSTM (units = 100, m 20)
0.089=0.005
LSTM (units = 100,
40)
0.09011.004
GRU (units 30, m 15)
0.095
GRU (units = 30, m 20)
0.093+000002
GRU (units 30, m 40)
0.094+00000
GRU (units = 100. m = 15)
0.091=0.002
GRU (units 100.
20)
0.092-10.004
GRU (units = 100, = 40)
0.091
Model
F(Q)
MLP(Q)
GRU(Q))
LGSSM filter(m = 15, n 30, K = 10)
0.089-100009
0,088+10.00
0.080
LGSSM filter(m 15, 30, K 15)
0.088
0.087+:0.007
0.086:+00004
LGSSM filter(m = 15, n = 45, = 10)
0.085+00000
0.084+10.007
0.084+00009
LGSSM filter(m 15.
45,
15)
0.08410.005
0.08310.004
0.082+0.004
LGSSM filter(mn = 20. n = 40, K = 10)
0.084=0.009
0.082:10014
0.082+0.011
LGSSM filter(m 20.
40,
15)
0.083+0.012
0.08110.005
0.080100011
LGSSM filter(m = 20, n 60, K = 10)
0.079=000055
0,078:0,012
0.076100009
LGSSM filter(m 20, n 60,
15)
0.077
0.075+00011
0.074
LGSSM smooth(m 15, n 30, K 10)
0.086=00011
0.083110.004
0.08410.007
LGSSM smooth(m 15, 30, K 15)
0.085=0.012
0.084+00008
0.083=00012
LGSSM smooth(m 15, n 45, K = 10)
0.081=0.008
0.080+00009
0.079#0.003
LGSSM smooth(m 15, 45, K 15)
0.081
0.07810.007
0.077=0.011
LGSSM smooth(m 20, n 40, K = =10)
0.082+0.005
0.078+00004
0.076+00013
LGSSM smooth(m 20,
40, K =5)
0.080+0.003
0.076+00006
0.074
LGSSM smooth(m 20, n 60, K = 10)
0.076=0.008
0.073+0.002
0.070+00000
LGSSM smooth(m 20, π 60, K 15)
0.07310.013
0.071
0.06810.013
GIN filter(m 15, 30, K 10)
0.078
0.076:0.005
0.075+00004
GIN filter(m 15, 30. K 15)
0.078
0.075+00009
0.07410.012
GIN filter(m 15, π 45, K = 10)
0.014
0.073+00008
0.072
GIN filter(m 15, 45. K = 15)
0.003
0.074+10011
0.071
GIN filter(m 20, π = 40, K = 10)
0.072+0.005
0.072+0.008
0.07010.002
GIN filter(m 20. 40. K 15)
0.071=0.007
0.07140.004
0.071
GIN filter(m 20, π = 60, K = 10)
0.06710.009
0.066110.005
0.065+0.006
GIN filter(m 20, 60, K 15)
0.065+0.013
0.06410.009
0.063100010
GIN mmooth(m 15, 30,
10)
0.071
0.07010.003
0.06810.009
GIN smooth( m = 15, 30, K 15)
0.070=0.008
0.068+00011
0.068+0.007
GIN smooth( m 15, n 45,
10)
0.065
0.069+0.009
0.06410.012
GIN smooth( m = 15, n = 45, K = 15)
0.064=0.008
0.066+0.007
0.063+0.009
GIN smooth( m 20, n 40,
10)
0.064=0.005
0.065+0.003
0.062+00008
GIN smooth(1 m 20.
40, K = 15)
0.063+00004
0.064+-0001
0.063+0.007
GIN smooth( m 20, n 60, 10)
0.059+0.009
0.061+0.012
0.057+00.06
GIN mm = 20, n 60, K = 15)
0.058+00005
0.057+0.009
0.056+0.004
33

---

Table MSE for double pendulum experiment.
Model
MSE
LSTM (units 50, m 15)
0.172
LSTM (units = =50, m 20)
0.166=0.009
LSTM (units = -00, m 40)
0.167=0.011
LSTM (units = 100,
15)
0.16410.006
LSTM (units = 100, m 20)
0.162=00009
LSTM (units = 100,
40)
0.159
GRU (units 50, m 15)
0.194
GRU (units = 50, m 20)
0.189
GRU (units 50, m 40)
0.188
GRU (units = 100. m = 15)
0.173
GRU (units 100.
20)
0.169=00014
GRU (units = 100, = 40)
0.166=0.018
Model
F(Q)
MLP(Q)
GRU(Q))
LGSSM filter(m = 15, n 30, K = 10)
0.054
0.159-10.021
0.153:000009
LGSSM filter(m 15, 30, K 15)
0.152+0.008
0.153+10.00
0.152+00012
LGSSM filter(m = 15, n = 45, = 10)
0.044
0.141+00015
0.139+00013
LGSSM filter(m 15.
45,
15)
0.142+0.007
0.13810.012
0.13710.017
LGSSM filter(mn = 20. n = 40, K = 10)
0.144
0.13710.009
0.138
LGSSM filter(m 20.
40,
15)
0.041
0.13710.008
0.136
LGSSM filter(m = 20, n 60, K = 10)
0.129=0.009
0.126-1.014
0.122
LGSSM filter(m 20, n 60,
15)
0.127=0.012
0.124+0.013
0.119
LGSSM smooth(m 15, n 30, K 10)
0.147
0.14810.014
0.144
LGSSM smooth(m 15, 30, K 15)
0.146=0.014
0.146=:0.013
0.142=00017
LGSSM smooth(m 15, n 45, K 10)
0.139
0.136+0.009
0.133+0.017
LGSSM smooth(m 15, 45, K 15)
0.137
0.135+0.017
0.131100012
LGSSM smooth(m 20, n 40, K = =10)
0.136+0.014
0.131+0.022
0.132+0.011
LGSSM smooth(m 20,
40, K 15)
0.134+0.011
0.12940.011
0.129
LGSSM smooth(m 20, n 60, K = 10)
0.123+0.019
0.116+40.016
0.115+00013
LGSSM smooth(m 20, π 60, K 15)
0.120
0.11240.009
0.108
GIN filter(m 15, 30, K 10)
0.126
0.125
0.125#0.011
GIN filter(m 15, 30. K 15)
0.024
0.124
0.121
GIN filter(m 15, π 45, K = 10)
0.115
0.114
0.110
GIN filter(m 15, 45. K = 15)
0.114
0.112+0.020
0.110
GIN filter(m 20, π = 40, K = 10)
0.113
0.111
0.111#0.013
GIN filter(m 20. 40. K 15)
0.111
0.10910.018
0.108100009
GIN filter(m 20, π = 60, K = 10)
0.09910.018
0.09410.017
0.095#0.021
GIN filter(m 20, 60, K 15)
0.097
0.09340.009
0.091
GIN mmooth(m 15, 30,
10)
0.115
0.116940009
0.113
GIN smooth( m = 15, 30, K 15)
0.015
0.113
0.112+0.014
GIN smooth( m 15, n 45,
10)
0.105
0.10140.014
0.101+0.015
GIN smooth( m = 15, n = 45, K 15)
0.102+0.013
0.100#0.011
0.09810.008
GIN smooth( m 20, n 40,
10)
0.101
0.098+0.010
0.0941:0.115
GIN smooth(1 m 20.
40, K = 15)
0.098+0.017
0.095+0.014
0.092+00007
GIN smooth( m 20, n 60, 10)
0.086+0.011
0.081+0.008
0.079+00009
GIN sm = 20, n 60, K = 15)
0.083+0.009
0.079+0.006
0.076+0.013
34

---

Table 10: MSE for bouncing ball experiment.
Model
MSE
LSTM (units 50,, m 15)
3.724
LSTM (units = =50, m 20)
3.698=0.127
LSTM (units = 500, m 40)
3.684=0.152
LSTM (units = 100,
15)
3.476=1.191
LSTM (units = 100, m 20)
3.39-0.282
LSTM (units = 100,
40)
3.28
GRU (units 50, m 15)
3.941
GRU (units = 50, m 20)
3.725
GRU (units 50,, m 40)
3.65
GRU (units = 100. m = 15)
3.631
GRU (units 100.
20)
3.512=0.4666
GRU (units = 100, = 40)
3.389
Model
F(Q)
MLP(Q)
GRU(Q))
LGSSM filter(m = 15, n 30, K = 10)
0.559
0.557:10.039
0.543:00.049
LGSSM filter(m 15, 30, K 15)
0.052
0.5471:0019
0.501
LGSSM filter(m = 15, n = 45, = 10)
0.041
0.543+-0.022
0.524
LGSSM filter(m 15.
45,
15)
0.539
0.538+0.042
0.51910.039
LGSSM filter(mn = 20. n = 40, K = 10)
0.042
0.536:10:029
0.518
LGSSM filter(m 20.
40,
15)
0.524=0.049
0.53710.066
0.5221000337
LGSSM filter(m = 20, n 60, K = 10)
0.517
0.52410.029
0.51410.060
LGSSM filter(m 20, n 60,
15)
0.529=0.034
0.517#0.039
0.512+0.022
LGSSM smooth(m 15, n 30, K 10)
0.485+0.022
0.483:10.019
0.475
LGSSM smooth(m 15, 30, K 15)
0.476
0.479=000051
0.458
LGSSM smooth(m 15, n 45, K 10)
0.479
0.471#0.045
0.468+0.058
LGSSM smooth(m 15, 45, K 15)
0.462+0.039
0.473+0.041
0.471
LGSSM smooth(m 20, n 40, K = =10)
0.469+0.021
0.475+00.04
0.451+0.033
LGSSM smooth(m 20,
40, K 11)
0.469
0.465+0.037
0.455+00027
LGSSM smooth(m 20, n 60, K = 10)
0.460=0.038
0.452+0.014
0.442+0.058
LGSSM smooth(m 20, π 60, K 15)
0.44810.027
0.452+0.020
0.44410.019
GIN filter(m 15, 30, K 10)
0.267
0.264+0.027
0.258+0.034
GIN filter(m 15, 30. K 15)
0.265
0.260-00.00
0.251
GIN filter(m 15, π 45, K = 10)
0.204
0.259-
0.019
GIN filter(m 15, 45. K = 15)
0.209
0.260
GIN filter(m 20, π = 40, K = 10)
0.255
0.250+0.011
0.23910.051
GIN filter(m 20. 40. K 15)
0.256=0.029
0.24310.034
0.23710.016
GIN filter(m 20, π = 60, K = 10)
0.247
0.24010.018
0.231+0.028
GIN filter(m 20, 60, K 15)
0.244
0.24110.031
0.231
GIN mmooth(m 15, 30,
10)
0.041
0.146+00012
0.139100014
GIN smooth( m = 15, 30, K 15)
0.145=0.008
0.145+00.031
0.133
GIN smooth( m 15, n 45,
10)
0.140=0.015
0.13940.008
0.130#0.028
GIN smooth( m = 15, n = 45, K 15)
0.133+0.011
0.13440.024
0.129+00015
GIN smooth( m 20, n 40,
10)
0.121
0.118+0.008
0.1101:0016
GIN smooth(1 m 20.
40, K = 15)
0.115+0.012
0.117+0.009
0.106+0.014
GIN smooth( m 20, n 60, 10)
0.115+0.013
0.110+00018
0.099+00008
GIN sm = 20, n 60, K = 15)
0.112+0.010
0.106+00002
0.094+0.011
35

---

To demonstrate the simplicity of our proposed GIN, we include intuitive inference code with Ten- sorflow library. The code runs with Python 3.6+. The entire code to reproduce the experiments are available in Github repository.
A.11.1
Python intuitive code.
Inference with
import tensorflow keras as k
import Prediction
import Filtering
import Smoothing
import DynamicsNetworkk
import Encoder
import Decoder
10 import DataGen
11
12
class GIN_CELL (k. .
def ._ (self, initial_states):
self mu_tmi_filt, self Sigma_tmi_fiil initial_states self filtee__states [[self miltmi self Sigma filt]] def call (self, inputs):
15
16
17
w_1:T, r_1:7 inputs
for v_t, r.t in (v_1:T, r_1:T):
19
x_tml_filt sample (self moollf.muttm_fiill self self.Sigma_tm...fi... F_hat_t, H_hat_t (ppamiccNeetorrrtss_____iii
20
21
mu_t_pri, Sigma_t_pri Prediction(F_hatt.. H_hat_t,...
self mi__mmiffit,, self spl..sigmaatmm_fiil))
mu_t_filt, Sigma_t_filt Piltering(mm_t_prr .....
24
Sigma_t_pri, w_t, r_t, H_hat_t)
25
self wtt.fieer_ssatseepppooddddeee self mu mu_ cou_tm__fill =_t_fil self Sigma_tmi..ilt Sigma_t_filt ffg mu mu_1:T_smooth, Sigma_1:7
Sigma_t_filth
26
Smoothing( F_hat_1:T,
29
self.filter_states Sigma_1:T_pri. 1:
30
return mu_1:T_smooth T_mmotttT_smmothh
c class (IN mwdels. [M
33
def __init__ (self, initial_states):
34
self mu_0_f self Saama.iima____0 self. Selff.GINCEELLLC aogma_0_filtt
1_mm___fflll filt initial_states
35
= self GIN GIN_GEINELLLLL (self x_O_filt, self
36
self Encoder Encoder self. Decoder Decoder
37
38
def call(self, o_1:T): w_1:T, r_1:T self Encoder (o_ (o_1:T)
mu_1:T_smootil 1igma_1:T_
self GIN CELLLLBO r_1
o_1:T, 8_1:7 self Decoder (mu_1 1: smooth, Sigma_ : :T)
42
return o_1:f, s_1:T
45
initial_states mu_0_filt, Sigma_0_filt GIN_OB: (iniitaall_ttatess
46
47
Data DataGen ()
0_1:T, a_1:T GIN OBJ (Data)
36

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
Justification: The claims we have made in abstract and introduction elaborated and achieved the main paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations and assumptions necessary for Theorems 3 and 4. In the appendix, where we provide the proofs, we reiterate the requirements that the algorithm must meet.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The proofs the mentioned theorems and mathematical claims are all provided in appendix section. In the main text, refer the readers to the proofs after introducing each theorem-claim.
Justification:
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the required information for running the experiments are provided in the supplementary materials and anonymous github repository.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide instructions to synthesis the data and run the experiments. All details are provided in anonymous github repository.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed section in appendix argue about network architecture, hyper parameter optimization, experiment setting, etc.. This addresses the question you asked.
Guidelines:
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
37

---

Justification: We include confidence interval in our numerical experiments to show the reliability of the results. Initialization approaches explained with details in the main script.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We provide detailed configuration of our machines for running the experiments.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We are following all the ethics provided by NeurIPS.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We believe this work does not have negative impact on the social communities.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The paper does not use existing assets.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
38


Reviews:
[{"review_id": "GGvHDDuBZe", "reviewer": "Reviewer_g5tG", "summary": "This paper advances temporal reasoning in dynamic, high-dimensional, noisy environments by introducing a novel architecture for latent variable state space models. The architecture permits efficient Bayesian inference with nonlinear transitions and emissions. Experiments are performed on toy datasets and a simple real-world dataset for state estimation and missing data imputation, showing that it beats benchmarks relative to competing models like RNNs, autoregressive models, and latent variable approaches.", "questions": "Are there any experiments you could do that show a sequential modeling problem in which inference was previously intractable is now so?", "limitations": "Yes, limitations adequately addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear exposition of model architecture and inference algorithm. Theoretical analysis in Section 6.", "weaknesses": "I think one thing that could really strengthen this paper is showing an experiment on a more challenging data set / problem. The first two experiments are on toy problems.\n\nI think another thing is to explain more clearly how this architecture is differentiated from others, i.e. the technical novelty. E.g. what is the relation of your model to other SSMs incorporating RNNs like the Variational RNN (which you benchmark against in the experiments), and what is it about that change that improves inference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and the time you dedicated. We have responded to your questions individually and revised the draft to address your concerns.\n\n---\n\n**q0**: *I think one thing that could really strengthen...*\n\n**a0**: We performed our evaluations and experiments in accordance with popular benchmarks and trends in the related literature, allowing us to compare our results with existing works.\nWe created a table of the most relevant related works, including their experiment selections and the datasets they used. Additionally, we provided another table detailing the statistics of these commonly used datasets (please see them in .pdf file).\n\nIn particular, related works typically follow a sequence of categories for their experiments:\n\n 1. Simple Low-Dimensional Experiments: These involve relatively low observation dimension, with dynamics that can be either linear, switching-linear or nonlinear. (Datasets in first part of table 2 in .pdf)\n  2. Toy/Synthetic Experiments: These involve higher-dimensional data and potential challenges, such as noise distortion or missing frames. (Datasets in second part of table 2 in .pdf)\n  3. Real-World Datasets: These can be either lower-dimensional data with easier state estimation tasks or higher-dimensional data with complex visual observations. (Datasets in last part of table 2 in .pdf)\n\nIn our paper, we deliberately skipped the first category due to its simplicity. Instead, we chose three synthetic datasets from the second category (single pendulum, double pendulum, and bouncing ball) and one additional dataset from the third category (KITTI visual odometry). The KITTI dataset, with $1242\\times 375\\times3$ dimension of image observations, is, to the best of our knowledge, one of the most challenging problems to infer in this literature that most of related works skip it (visualization in Fig 1 of PDF response). \nPlease note that, in the revised version (line 274), we included additional experiment to highlight the efficiency of theorems, introduced in the paper (also added in table 1 in PDF response).\n\n---\n\n**q1**: *I think another thing is to explain more...*\n\n**a1**: Our structural novelty can be divided into three main parts:\n\n 1. New Inference Algorithm: We propose an inference algorithm that models the transition and emission distributions.\n 2. Parameterization and Approximations: We introduce parameterizations and approximations for the parameters and distributions.\n 3. Stability Scheme: We provide a stability scheme for the newly parameterized algorithm.\n\nTo provide more specific information and test the improvement corresponding to each part, we compared our model with different approaches through extensive ablation studies.\n\n 1. First, we used a simple encoder-decoder without latent parameterization to demonstrate the effectiveness of parameterizing the latent state (lines 292-293 first draft, 288-289 revised draft).\n\n 2. Second, we evaluated the effectiveness of the smoothing-filtering parameterization designed in our transition block by removing it and replacing it with benchmark RNNs such as AE-RNN. This demonstrated that the smoothing-filtering improves inference compared to benchmarks lacking this parameterization. (lines 299-302 first draft, 295-298 revised draft).\n\n 3. Third, to validate the accuracy of our distribution approximations, we compared them with state-of-the-art methods.\n    These comparisons highlight how our approach differs from other SSMs incorporating RNNs, such as the Variational RNN, and demonstrate the improvements in inference achieved through our proposed method. ( lines 303-308 first draft, 299-304 revised draft)\n \n 4. Finally, in the newly added experiment in the revised draft, we aim to demonstrate that the proposed stability scheme outperforms the traditional gradient clipping approach in preventing gradient explosion, which threatens the stability of the GRU cells used for parameterization.\n\n---\n\n**q2**: *Are there any experiments you could...*\n\n**a2**: In sequential modeling, especially with high-dimensional sensory observations, it is intractable to obtain the exact evidence likelihood, such as $p(o)$.This is because calculating $\\frac{p(o|x).p(x)}{p(x|o)}$, is challenging due to the difficulty in obtaining the distribution for ${p(x|o)}$. Common approaches involve approximating this term, as is done in related works and in our model.\n\nFor simple systems with linear dynamics, such as car movement modeled by the celebrated Kalman Filter (KF), Linear Gaussian State Space Models (LGSSMs) can be used to derive exact filtering and smoothing distributions (tractable posterior). However, we have chosen to focus on more complex systems, as the optimal solution for linear systems (i.e., the Kalman Filter) already exists and is well-established.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dwidRzlUm1", "reviewer": "Reviewer_Db5x", "summary": "The paper introduces a very well theoretically motivated State-Space Model learning approach, which is implemented by a gated inference network. The network implements a Hammerstein-Wiener model within a modularized deep learning architecture. It uses GRU cells to mimic Kalman Filtering operations. Forward as well as forward-inverse processing routines optimize the hidden state space estimations. Several theoretical components add to the paper contribution. Evaluations show superior performance on several challenging toy problems with noisy data (pendulum, double pendulum, ball bouncing in irregular polygon, as well as odometry prediction from kiti data) evaluating state estimation and imputation tasks.", "questions": "Figure 1 trajectories are not quite as smooth as I had expected. Any reason for this? The task, particularly between bounces, to predict a linear trajectory should be very simple. \n\nI do not really see how Theorems 1 and 2 are actual theorem. Don’t they just define the log likelihoods of the ground truth states / observations? \n\nTheorem 2 – is the second summand starting with factor $(1-o_t^{(k)})$ necessary? \n\nLine 260: should read \"in theorem 3\", no?", "limitations": "Further evaluations and ablations to truly identify the core components that yield the great results of the GIN system.\n\nConfirm theory components experimentally.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Paper is very well-structured. The work is also very well-motivated and well-embedded into the literature. \n\nThe theoretical motivation and system derivations are impressive and usefully embed the author’s GIN system into the Kalman Filtering background. Approximating everything in a variational inference manner via estimations of Gaussians and their Covariances is efficient.\n\nTheorems 3 and 4 offer a theoretical derivation for ensuring stability of the unfolding recurrence. \n\nThe evaluations contain sufficiently challenging problems. Performance is compared with many alternatives, showing superior performance nearly throughout. Only in Table 3 GIN was partially beaten by DeepVO.", "weaknesses": "The theorems 3 and 4 are not really experimentally evaluated. Is instability observed when the recurrent matrix is not modified as proposed? The theorem’s proposal should be verified experimentally. \n\nEven more elaborate evaluations would of course be great. Seeing the great content and the importance of the theoretical derivations, though, I consider this a very minor point, which can be tackled in subsequent work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and time. We have answered your questions one by one and revised the draft to address your concerns.\n \n---\n\n**q0**: *The theorems 3 and 4 are not really ...*\n\n**a0**: We appreciate your concern regarding the experimental evaluation of Theorems 3 and 4. Given the page limit of the conference, we could only present a portion of our experimental results. However, we conducted multiple experiments to demonstrate the contributions of these theorems.\nSpecifically, addressing your concern, we tested each experiment multiple times with various settings, including the approaches described in Theorems 3 and 4 (SVD and Gershgorin Circle Theorem) and gradient clipping. We add these details in the revised draft (line 274). We also attached this table in the PDF response file.\n\nIn summary, table 1 in response PDF listing the log likelihood and its standard deviation for three experiments: single pendulum, double pendulum, and irregular polygon. These experiments were trained under different settings for handling gradient explosion: the conventional Gradient Clipping (GC), our first solution using Singular Value Decomposition (SVD), and our second solution using the Gershgorin Circle Theorem (GCT).  In this table, $\\theta$ represents the threshold for gradient clipping, and $\\delta$ is the threshold in our method that keeps the spectral radius less than 2, i.e., $\\sigma_1(\\mathbf{U}_h) + \\delta = 2$. We also included the success rate to indicate that not only high performance but also consistent stability across multiple experiments is important. \n\nWe hope this clarifies how Theorems 3 and 4 have been empirically validated and show the effectiveness of our proposed theorems.\n\n---\n\n**q1**: *Figure 1 trajectories are not quite as smooth ...*\n\n**a1**: Thank you for your observation regarding the trajectories in Figure 1. There are two primary reasons for the observed sharpness in the movements: \n\n  1. *Initial Training Focus:* In the initial epochs, we focus the system on learning the auto-encoder and globally learned parameters, such as \n$F$ and $H$\n , while excluding the Dynamics Network parameters \n$\\alpha(x)$. This approach allows the system to initially acquire robust embedding and meaningful latent state vectors (essentially better embedding of the objects' locations and velocities within their environment) before involving the Dynamics Network. By doing this, we enable the Dynamics Network to be trained later with rich latent representations, resulting in more confident weight selection.\n\n   2. *Mode Collapse Loss Term:* We incorporate a mode collapse loss term that increases the distance between each of the $K$ transition matrices. In terms of the Frobenius norm, these matrices can be considered as being maximally separated. This separation leads to more distinct movement patterns. \n\n Subsequently, all parameters, including those of the Dynamics Network, are jointly learned. This approach ensures that the Dynamics Network assigns weights to the relatively separated \n $F$ matrices, each modeling an entirely different movement pattern. After a considerable number of epochs, the Dynamics Network selects the weights of the relevant \n $F$ matrices with high confidence, almost certainly (weights are around one for the target \n $F$ and around zero for the non-relevant \n $F$ matrices relying on the numerical results).\n\nSpecifically, the issue of soft reflections, and in more severe cases, movement along the edges of the walls after bouncing, was observed in our experiments prior to implementing the mode collapse loss term. The addition of this term has significantly improved the movement patterns, making them more distinct and reducing such anomalies.\n\nWe hope this addresses your concern and provides clarity on the observed trajectory behavior.\n\n---\n\n**q2**: *I do not really see how Theorems 1 and 2 are actual...*\n\n**a2**: Your observation about the log likelihoods of the states and observations is correct to an extent. These theorems indeed involve the log likelihoods, but it is important to note that they represent approximated log likelihoods rather than exact ones.\n\nSince our filtering and smoothing distributions are approximations of their exact counterparts, we derive a *lower bound* of the log likelihood rather than the exact log likelihood. The theorems detail the derivation process of this lower bound, illustrating where the inequalities and approximations arise. This comprehensive process includes the final descriptions for state inference (modeled as Gaussian distributions) and image inference (modeled as Bernoulli distributions).\n\nThe inclusion of these derivations as theorems, complete with proofs, is intended to provide a guarantee for our approximations proximity and to ensure the completeness of our method. \n\n---\n\n**q3**: *Theorem 2 – is the second summand...*\n\n**a3**: \nThe log likelihood in this equation arises from modeling the pixels of the original observations using a Bernoulli distribution. Specifically, the likelihood term is given by $\\prod d_k^{o_k} (1-d_k)^{1 - o_k}$. \nWhen considering all pixels of an observation with \n$D$ dimensions, across \n$T$ time steps, and over \n$N$ Monte Carlo samples, the complete likelihood expression accounts for the contributions from all these dimensions. Taking the logarithm of this likelihood function results in the equation with the second term in summand you are referring to. This summand captures the likelihood of the observations under the Bernoulli distribution, ensuring the proper representation of the data's probabilistic structure.\n\n---\n\n**q4**: *Line 260: should read \"in theorem 3\", no?*\n\n**a4**: You are correct; the reference should indeed read \"in Theorem 3.\" We appreciate your attention. We made the necessary correction in the revised version of the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K9yByiu9ja", "reviewer": "Reviewer_254u", "summary": "The paper presents a deep state-space model architecture with non-linear transitions and emissions. The model disentangles the latent representation for the dynamics and the one for the observed data at each time step - allowing therefore effective state estimation at future time steps and the ability to deal with missing data imputation.\nInference is performed with a deep Extended Kalman Filter, that relies on a RNN architecture to make a more efficient approximate computation of the Kalman Gain (KG) and smoothing gain (SG).\nThe method is tested on a number of simulated and realistic approaches, and it outperforms competing architectures.", "questions": "I am happy to increase the score as long as the comments/questions in the weaknesses section are clarified", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. Non-linear/deep state-space models are being used more and more in many applications. Parameter learning and state estimation is however challenging in this setting, and this paper provides an interesting method for this\n2. The method is more scalable than comptering KF-based methods thanks to the dynamics network approximation, but still effective despite the aproximation\n3. The method builds on some models in the literature, but provides some useful novel components\n4. The authors did extensive and well-though experiments/ablations, comparing with many SOTA models\n5. There is an extensive appendix covering many details that did not fit in the main text. I particularly appreciated \"A.11.1 Python intuitive code.\"", "weaknesses": "The paper is not straightforward to read (had to read it carefully twice), mostly because of the way the required derivations are presented.\nThe notation used is somewhat not conventional within the ML-heavy NeurIPS community, and should be improved/clarified:\n1. In Section 4 you use o^+ notation which is not common in the ML community. Can you clarify what it means and why you need it? This explanation needs to be done in section 4, not referring to a different section.\n2. Similarly, what is s in line 219 and why do you need to introduce this notation? The sentence in line 219-221 is key but unclear\n3. Why do you need to define the \"gt\" in line 229?\n4. Not sure the SI perspective helps in the ML-heavy NeurIPS community, it brings confusion. Maybe can be added in the appendix?\n\nIn terms of novelty, the final model seems to me more similar to the Kalman VAE (KVAE) model than what the authors claim. Your model can almost be seen as a modification/extension to the KVAE in which you add the RNN approximation to avoid the O(n^3) complexity, model the transition noise covariance and use a slightly different parameterization for the dynamics network.\nCan you clarify the differences between your model and the KVAE? \nIn line 96 and Table 6 in the appendix, I don't think your KVAE description is correct: it has a setup very similar to your model which allows to estimate the state dynamics, and allows for direct optimization unlike what you claim.\nI am not as familiar with the other KF-based methods mentioned, but make sure your description is correct.\n\nMinor comments:\n1. Line 53: typo \"To to\"\n2. Line 71: typo \"We\" -> \"we\"\n3. You introduce gamma in line 144, but only say what it is in line 150, making the reader wonder if it was defined above and look for it\n4. Line 283: you say \"with n=3m\" without specifying what n and m are. Even if they are defined before, being a notation-heavy paper, better to remind the reader what n and m are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review. We appreciate the time and energy you put on this work.  We reply to your questions one by one.\n\n---\n\n**q0**: *In Section 4 you use $o^+$ notation... + What is s in line 219 ...*\n\n**a0**:  Thank you for your question regarding the \n$o^+$ and $s$ notations used in the paper. We used $o^+$ notation to represent the output of the generative process contingent on the task. Specifically, in our paper, we address two tasks: \n\n 1. Direct State Estimation:\nHere, the generative process generates \n$s$, representing the real-world estimated state (e.g., the position and velocity of the ball in the third experiment).\n\n 2. Image Imputation:\nIn this case, the generative process constructs the high-dimensional sensory observation $o$.\n\nTherefore, $o^+$ can either be equivalent to \n$s$ or \n$o$, depending on the task at hand (lines 219-221 first draft ).\nThe differences in tasks lead to structural variations in the decoder \n$d(.)$ for each task, as clarified in Table 4 of the appendix. We explicitly define these differences in the decoder architecture for generating \n$s$ (state estimation) and \n$o$ (image imputation).\n\n To address your concern about the clarity of this specific notation, we have modified the text and figures in the revised manuscript. We have omitted the  $o^+$   notation and revised the figures accordingly.\n For instance, Figures 2 and 3 now explicitly produce both $s$ and $o$, with a description indicating that the output is contingent on the task ( line 146 revised draft).\n Additionally, we have moved the explanation of all notations and dimensions, including $n$ and $m$, to the beginning of Section 4 for better clarity ( lines 134-138 revised draft). \nAlso lines 219-221 (in old draft) is revised to improve readability (now lines 215-217 in revised draft). We hope these changes address your concerns and clarify the use of notation in our paper.\n\n---\n\n**q1**: *Why do you need to define the \"gt\" in line 229?*\n\n**a1**: State estimation requires supervised training, and we intended to indicate that ground truth states are integrated during the training phase. During training (fitting), when modeling $\\mathcal{L}(s_{1:T}|o_{1:T})$ in Equation 10,  the ground truth states $s_{1:T}$ are known and used to calculate gradients (by calculating the likelihood and attempting to maximize it). However, during inference, the ground truth states \n $s_{1:T}$\n  are not known and their estimation must be sampled from Equation 10. We aimed to clarify that these two procedures—training and inference—are distinct. \n\n  To avoid confusion and to align with community notation, we have decided to remove the \"gt\" notation from the text and Equation 10. Instead, in the rebuttal version, we emphasize the training scheme in the introduction (line 67; \"the objective is maximized...\") and in the experiment section, where we clarify that the estimated states are samples from modeled distributions during inference and should not be mistaken for ground truth.\n\n---\n\n**q2**: *Not sure the SI perspective helps...*\n\n**a2**:  We appreciate your suggestion and agree that it may not be as relevant to the ML-heavy NeurIPS community. To address this, we moved the discussion of System Identification and its various versions, along with a detailed explanation of the EM algorithm used in these approaches, to the appendix in the revised draft. \n\n---\n\n**q3**: *In terms of novelty, the final model seems to me...*\n\n**a3**:  KVAE is an EM-based variational model designed to maximize the observation (or evidence) likelihood. It focuses on inferring high-dimensional observations for frame generation (pixel space). The maximization of the observation likelihood in KVAE (Equation 7 in their paper) is conceptually similar to our goal of image imputation, where we aim to maximize $\\mathcal{L}(o_{1:T})$  in Equation 11. This similarity holds despite differences in parameterization, scalability, and distribution approximations.\n\nAlthough KVAE allows for sampling from posterior states (Equation 11 in their paper), it lacks a framework for direct state optimization, such as providing and maximizing the log likelihood for states like ball position-velocity in the polygon experiment. In contrast, our model addresses this by explicitly modeling $\\mathcal{L}(s_{1:T}|o_{1:T}) $.\n\nOur model parameterizes $p(s_{1:T}|x_{1:T}, w_{1:T}, o_{1:T})$  with approximated distributions that enable direct state optimization. This *conditional* likelihood modeling is conceptually similar to the objectives defined in works of \n[Shmakov et al. (2023)](https://openreview.net/forum?id=v7WWesSiOu&referrer=%5Bthe%20profile%20of%20Alexander%20Shmakov%5D(%2Fprofile%3Fid%3D~Alexander_Shmakov1)),\n[Zhang et al. (2023)](https://openreview.net/forum?id=o16sYKHk3S&referrer=%5Bthe%20profile%20of%20Akash%20Srivastava%5D(%2Fprofile%3Fid%3D~Akash_Srivastava1)), and \n[Sohn et al. (2015)](https://papers.nips.cc/paper_files/paper/2015/hash/8d55a249e6baa5c06772297520da2051-Abstract.html),\nwhere *conditional* likelihoods are modeled, albeit with different parameterizations and applications.\n\nThis dual functionality distinguishes our model from KVAE by allowing for direct optimization of state dynamics.\n\nTo address your concern, we decided to modify the word \"estimate\" to \"optimize\" in line 96, as it may imply that drawing samples of the state (considered as estimated states during inference) is not feasible in KVAE. Accordingly, we have updated the state estimation column of Table 6 in the appendix for KVAE, EKVAE and MVAE to prevent confusion for readers.\n\nWe hope this clarifies the distinctions between our model and KVAE and addresses the concerns raised.\n\n---\n\n**q4**: *Minor comments..*\n\n**a4**: We greatly appreciate your careful review and attention to detail. We fixed typos, added gamma description before using it, and added definition of $n$ and $m$ in experiment section again (line 279 revised draft)\nfor better readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers for their detailed comments and questions. Our rebuttal response is mainly organized into three sections.

1. To address the concern of one of the reviewers regarding the sufficiency and complexity of our experiments, we created a table listing the most relevant studies, detailing their experimental setups and the datasets they utilized. Additionally, we provided a separate table outlining the statistics of these commonly used datasets. We compared our experimental choices with those of the listed studies, evaluating both the complexity (e.g. observation dimension/type, sequence length)  and the number of experiments conducted. These tables are in the PDF response file.


2. We included the results from an additional experiment to show the effectiveness of the theorems presented in the paper. In brief, we compared the proposed theorem with the traditional gradient clipping method for managing gradient explosion. We reported the success rates of the experiments to reflect the stability across various seeds. These results are available in a one-page PDF included in the response (Table 1). We also add this to the revised paper.

3. We have revised the notation and figures for improved clarity and removed ambiguous terms. Specifically, to address the reviewer’s concerns, we eliminated the use of the less common $o^+$ notation. Additionally, all notations have been moved at the beginning of Section 4. We also refined Comparison Table 6 and clarified the shortcomings of related works (particularly KVAE) to prevent any confusion. Figure including revised variables (omitted $o_{1:T}^+$) for state estimation of visual odometry task is in one-page PDF response.



Detailed answers with further elaborations on the reviewers' questions are provided in separate rebuttals for each reviewer.

The 
[revised paper](https://raw.githubusercontent.com/AnonymousEzhdeha/GIN2024/main/Neurips2024_Rebuttal.pdf)
 is available after applying the reviewers comments.

Meta-review: Reviewers appreciated the novel approach to approximating the Kalman gain with a GRU for approximate filtering and smoothing. They also praised the extensive comparisons to existing methods. While I agree that the proposed approach seems interesting and the comparisons in Tables 1 and 2 go above and beyond, I still have strong reservations about this paper. I found the presentation to be below the bar of a top-tier conference like NeurIPS, and I am surprised that only Reviewer 254u raised concerns about it.  In my opinion, the sloppiness of the presentation could cause the reader to question the validity of the empirical results as well. Please address the issues below. Some will require substantial revision.

**Major Issues**
- Starting with the title, "inferencing" is not a word.
- As reviewer 254u noted, the paper uses nonstandard and inconsistent notation, making it extremely difficult to follow. The authors use "original observations $o$, task-dependent outputs $o^+$, and transferred (sometimes called lower-dimensional) observations $w$. None of these variables are ever clearly defined, and it is unclear what the difference between them is. Section 4 (line 140) incorrectly states that these terms are defined in Section 3. The terms "states" and "dynamics" are used interchangeably, when the rest of the field clearly distinguishes these concepts. Then, in Section 6, new notation $s$ and $gt$ are introduced for no apparent reason (it is simply stated that $s=o^+$). 
- lines 93: "SVAE, VAE, DVAE, EKVAE, ... use original KF equations for filtering and smoothing. However they cannot directly estimate states (dynamics) and do not model state space noise..." This is simply false. First, estimating states is not the same as estimating dynamics parameters. Second, these methods do model latent state dynamics and noise, and they offer principled approaches for state inference and parameter estimation. Existing work must be portrayed accurately.
- The graphical model in Fig 3 is not consistent with the mathematical formulae in Section 4. No equations are given for $o$ and $o^+$, so it's completely unclear how they are related to the other variables in the model. In particular, $o$ never appears in any graphical model or joint distribution. 
- The main contribution of the paper is in the approximation of the Kalman gain via a GRU (eq 6 and 8). However, this contribution is buried in an unnumbered paragraph of Section 5, and there is no context for why this formulation is chosen. For example, it is never stated that this form is meant to mimic the form of the actual Kalman gain for a linear dynamical system.
- Theorem 1 has several major problems. First, it is a definition, not a theorem. Second, it is intended to be an evidence lower bound, but it is written as a conditional distribution of task-dependent observations given original observations, which doesn't make much sense since those terms were never defined. At best, it could be interpreted as a expected log likelihood, not a lower bound on the marginal log likelihood. Third, the theorem introduces new notation $s$ and $gt$, which are unnecessary and confusing. 

**Minor issues**
- Figure 1 is not referenced in the text, and it conveys essentially zero information about the method. Figure 3 is referenced before Figure 2. 
- Several of the characterizations of existing work in Table 6 (Appendix) are incorrect
- line 47: SMM should be SSM
- line 101: Figorla should be Frigola
- The term "Hammerstein-Wiener (HW) models" is repeatedly used without definition
- periods are used where \cdot should have been 

To reiterate, the empirical results look good, and it is clear that a lot of work went into this paper. I see why the reviewers appreciated those aspects of the submission. However, the presentation has severe problems which lead me to doubt the soundness of the rest of the paper. It would be unfair to reject the paper at this point. Instead, I strongly encourage the authors to address these issues, even if they require substantial revision of the text. Finally, I expect the authors to hold themselves to a higher standard of writing quality in future submissions.
Decision: Accept (poster)