Non-Parametric State-Space Models: Identifiability, Estimation and Forecasting
Authors: Chenghao Liu
Weiran Yao
Steven Hoi
Kun Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: state-space model
time series forecasting
causal representation learning

Abstract: State-space models (SSMs) provide a standard methodology for time series analysis and prediction. While recent works utilize nonlinear functions to parameterize the transition and emission processes to enhance their expressivity, the form of additive noise still limits their applicability in real-world scenarios. In this work, we propose a general formulation of SSMs with a completely non-parametric transition model and a flexible emission model which can account for sensor distortion. Besides, to deal with more general scenarios (e.g., non-stationary time series), we add a higher level model to capture time-varying characteristics of the process.
Interestingly, we find that even though the proposed model is remarkably flexible, the latent processes are generally identifiable. Given this, we further propose the corresponding estimation procedure and make use of it for the forecasting task. Our model can recover the latent processes and their relations from observed sequential data. Accordingly, the proposed procedure can also be viewed as a method for causal representation learning. We argue that forecasting can benefit from causal representation learning, since the estimated latent variables are generally identifiable. Empirical comparisons on various datasets validate that our model could not only reliably identify the latent processes from the observed data, but also consistently outperform baselines in the forecasting task.

Under review as a conference paper at ICLR 2023
NON-PARAMETRIE STATE-SPACE MODELS: IDENTIFIA- BILITY, ESTIMATION AND FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
State-space models (SSMs) provide a standard methodology for time series analysis and prediction. While recent works utilize nonlinear functions to parameterize the transition and emission processes to enhance their expressivity, the form additive noise still limits their applicability in real-world scenarios. In this work, we propose a general formulation of SSMs with a completely non-parametric transition model and flexible emission model which can account for sensor distortion. Besides, to deal with more general scenarios (e.g., non-stationary time series), add higher-level model capture the time-varying characteristics of the process. Interestingly, we find that even though the proposed model is remarkably flexible, the latent processes are generally identifiable. Given this, we further propose the corresponding estimation procedure and make use of it for the forecasting task. Our model can recover the latent processes and their relations from observed sequential data. Accordingly, the proposed procedure can also be viewed as method for causal representation learning. We argue that forecasting can benefit from causal representation learning, since the estimated latent variables are generally identifiable. Empirical comparisons on various datasets validate that our model could not only reliably identify the latent processes from the observed data, but also consistently outperform basclines in the forecasting task.
INTRODUCTION
Time series forecasting plays crucial role in various automation and optimization of business processes (Petropoulos et al., 2022; Benidis et al., 2020; Lim & Zohren, 2021). State-space models (SSMs) (Durbin & Koopman, 2012) are among the most commonly-used generative forecasting models, providing a unified methodology to model dynamic behaviors of time series. Formally, given observations x,, they describe dynamical system with latent processes 2, as:
Z f(zz-1) + Ce.
X g(z,) + M/.
(1)
Transition
Emission
where 7/t and é denote the i.i.d. Gaussian measurement and process noise terms, and f()) and g(-) are the nonlinear transition model and the nonlinear emission model, respectively. The transition model captures the latent dynamics underlying the observed data, while the emission model learns the mapping from the latent processes to the observations. Recently, more expressive and scalable deep learning architectures were leveraged for modeling nonlinear transition and emission models effectively (Fraccaro et al., 2017; Castrejon et al., 2019; Saxena et al., 2021; Tang & Matteson, 2021).
However, these SSMs are not guaranteed to recover the underlying latent processes and their relations from observations. Furthermore, stringent assumptions of additive noise terms in both transition and emission models may not hold in practice. particular, the additive noise terms cannot capture nonlinear distortions in the observed or latent values of the variables, which might be necessarily true real-world applications (Zhang & Hyvarinen, 2012; Yao et al., 2021), like sensor distortion and motion capture. If we directly apply SSMs with this constrained additive noise form, the model misspecification can lead to biased estimations. Second, the identification of SSMs a very challenging task when both states and transition models unknown. Most work so far has focused on developing efficient estimation methods. We argue that this issue should not be ignored, and becomes more severe when nonlinear transition and emission models implemented with deep

---

Under review as a conference paper at ICLR 2023
learning techniques. As the parameter space has increased significantly. SSMs are prone to capture spurious causal relations and strengths, and thus identifiability of SSMs is vital. Furthermore, the transition model is usually assumed to be constant across the measured time period. This stationary assumption hardly holds in many real-life problems due to the changes in dynamics. For example, the unemployment rate tends to rise much faster the start of recession than it drops at the beginning of a recovery (Lubik & Matthes, 2015). In this setting, SSMs should appropriately adapt to the time-varying characteristics of the latent processes to be applicable in general -on-stationary
scenarios.
Observed Sequential Data
1. Latentt Causal Model/CCossal Representation Learning
1. Compact Predictor
shift
Robust to distribution
3. Compact way to model change factors
The proposed predictor trained latent space
Fur
PSR
X4-t
Latent Predictor
Vanilla predicters trained observation
x1-1
Time stept
Time step [+
x2
Frz-s
Model Estimation
Testing
Figure I: Left: The proposed estimation framework mainly includes the learning of latent causal model learning and prediction model. Right: Motivational examples demonstrate the benefit of latent causal model learning for forecasting. (1). It provides compact representations for forecasting, as vanilla predictors include complicated dependencies. (2). The prediction model is more robust the distribution shift (Red circles here indicate distribution change). (3). It gives compact way to model the change factors to address non-stationary forecasting issues.
In this work, in contrast to state-of-the-art approaches following the additive form of transi- tion/emission models, we propose a general formulation of SSMs, called the Non-Parametric State- Space Model (NPSSM) !. In particular, we leverage the non-parametric functional causal model (Pearl, 2009) for the transition process and the post-nonlinear model (Zhang & Hyvarinen, 2012) to capture nonlinear distortion effects in the emission model. Besides, we add a higher level model to NPSSM, called N-NPSSM, to capture the potential time-varying change property of the latent processes for more general scenarios (e.g., non-stationary time series). Interestingly, although the proposed NPSSM is remarkably flexible, the latent processes are generally identifiable. To this end, we further develop novel estimation framework built upon the structural variational autoencoder (VAE) for the proposed NPSSMs. It allows to recover latent processes and their time-delayed causal relations from observed sequential data and use them to build the latent prediction model simultaneously (illustrated in Figure (left)). Accordingly, the proposed procedure can be viewed as a method for causal representation learning or latent causal model learning from time series data.
We argue that forecasting tasks can benefit from causal representation learning, as latent processes generally identifialle in NPSSM. As shown in Figure 1(right), first, it provides a compact structure for forecasting, whereas vanilla predictors (bottom), which directly learn a mapping function in the observation space, face the issue of complicated and spurious dependencies. Second, the predictions following the correct causal factorization are expected to be more robust distribution shifts that happen to some of the modules in the system. If some local intervention exists on one mechanism. it will not affect other modules, and those modules will still contribute generously to the final prediction. Although formulating this problem and providing quantitative theoretical results seem challenging, our empirical studies illustrate this well. Third, it gives a compact way to model the distribution changes. In realistic situations, data distribution might change over time. Fortunately, given the sigh-dimensional input, the changes often occur in a relatively small space in a causally-factorizze system, which is known as the minimal change principle (Ghassami et al., 2018; Huang et al., 2020)
Here, the definition of "non--naneen parammarrmttric" is not about the general form of mapping function but indicates the functional causal model which takes the cause variables and errors as the input of a general function. Unlike the additive noise form, there is no constraint for how the noise interacts with the cause variable. Formal definition be found in line 4 below Eq. (1.40) in (Pearl, 2009)
2

---

Under review as a conference paper at ICLR 2023
or sparse mechanism shift (Schölkopf et al., 2021). We can thus capture the distribution changes with low-dimensional change factors a causal system instead of in the nigh-dimensional input space.
In summary, main contributions are follows:
We propose a general formulation of SSMs, namely, NPSSM, together with its extension to allow nonstationarity of the latent process over time, which provides flexible form for the transition and emission model that is expected to be widely applicable;
We establish the identifiability of the time-lagged latent variables and their influencing strengths for NPSSM under relatively mild conditions;
Based on our identifiability analysis, we propose a new structural VAE for model estimation and use it for forecasting tasks;
Estimation of the proposed model can be seen as a way to learn the underlying temporal causal
processes, which further facilitates forecasting of the time series;
We evaluate the proposed method on number of synthetic and real-world datasets. Experimental results demonstrate that latent causal dynamics could be reliably identified from observed data under various settings and further verify that identifying and using the latent temporal causal processes consistently improves the prediction performance.
2 PROBLEM FORMULATION
2.1 NPSSM: NON-PARAMETRIC STATE-SPACE MODEL AND IDENTIFIABILITY
To make SSMs in Eq. (I) flexible, we adopt the functional causal model (Pearl, 2009) to characterize transition process. Specifically, each latent factor Z is represented with a general form of structural causal model Z = f Pa(za)} (parents) denotes the set of time-lagged variables that directly determine the latent factor Zu. and r denotes the time lag index. In this way, noise 6x together with parents of 7it generate 7, via unknown function f(()) Formally, NPSSM can be formulated as
E Pa(zu)); cat,, where i.j denotes variable element index,
Zzz fe((2,,tr
Pa(zz)),
X1 Post nonlincar emission
g(z,, Ne) g1 (z)) + ne),
Structural causal latent transition
where E/z are mutually independent (i.e. spatially and temporally independent) random noises sampled from noise distribution p(Exr). 91 (-) is the invertible post-nonlinear distortion function, 92(() ) is the nonlinear mixing function and 7) are independent noises (detailed notations are given Appendix A2.1). To the best of our knowledge, this is the most general form of SSMs. In this transition function, the effect Z is just a smooth function (it refers to condition 3 of Theorem 1, which the core condition to guarantee the identifiability of NPSSM) of its parents Pa(z.) and noise eis, and it contains linear models, nonlinear models with additive noise, and even multiplicative noise models as special cases. The Independent Noise condition and Conditional Independent condition (Pearl, 2009) are widely satisfied in time series data. Furthermore, in the emission function, the post-nonlincar transformation 91 (-) can model sensor measurement distortion that usually happens when the underlying processes are measured with instruments (Zhang & Hyvarinen, 2012; Zhang & Hyvärinen, 2010).
Now, we define the identifiability of NPSSM in the function space. Once the latent variables Z1,-..- 27 are identifiable up componentwise transformations and permutation, latent transition (causal relationships) are also identifiable because conditional independence relations fully character- ize time delayed causal relations in a time-delayed causally sufficient system. Therefore, we can say that NPSSM is identifiable if the latent variables are identifiable.
Definition 1 (Identifiability of NPSSM), For a ground truth (f.g. and a learned (f.g 9. p(e))
models defined in Eq. (2), if the joint distribution for observed variables Pi sisple((e (x(x) and
Pi.g.a(e) (x)) are matched almost everywhere, then we can say NPSSM are identifiable if observational equivalence can always lead to identifiability of the latent variables up permutation and component-wit invertible transformation T::
Pa.f (x:) = Po.f.p. (xx) = --------- TO..
where g 11 -11 are invertible functions that maps x, 24 and 2n, respectively.
3

---

Under review as a conference paper at ICLR 2023
Here present the identifiability result of the proposed model. W.L..gg, assume the maximum time lag = in our analysis. Note that is trivial to extend our analysis for long lag L > 1. We can that, somewhat surprisingly, although NPSSM is remarkably flexible, it is actually identifiable up to relative minimum indeterminacies. Each latent process can he recovered up to its component-wiss invertible transformations. In many real-world time series applications, these indeterminacies may be inconsequential.
Theorem 1. Suppose that we observe data sumpled from generative model defined according to 2 with parameters (f. (.)).. Assume the following holds:
1. The set {xt E X Pm (xt) 0} has measure zero, where Py, is the characteristic function of the density p(T)) = Pg (x.lz)). The post nonlinear functions 91, 91 are invertible. The mixing functions 92, 92 are injective and differentiable almost everywhere.
2. The process noise terms Ea are mutually independent.
3. Let T7kt
A log p(zzt 72-1), Tkt is twice differentiable in 7ut and is differentiable in 71,--1, 1
1,2, 1,2....,h. n. For each value of Ze, Vit, Vit, V2t, Vat---., -.. Vnt, Vut as 2n vector functions in
71,1-1:722 7,,-11 2 are linearly independent, with Vkt and Vee defined below:
8
8
Vk.t
Das, Oan,sllas e-1
Vi.t
DE
then Z, must be an invertible, component-wii transformation of a permuted version of 2,-
The proof's are provided in Appendix A2.2. Theorem 1 indicates that we can find the underlying causal latent processes from the observed data. The differentiability and linear independence in condition is the core condition to assure the identifiability of latent factors z, from observed x,- It indicates that time-lagged variables must have sufficiently complex and diverse effect on the transition distributions in terms of the second- and third-order partial derivatives. From this condition, we can find that the linear Gaussian SSM is unidentifiable since the second- and third-order partial derivatives would be constant, which violates the lincar independence assumption.
2.2 N-NPSSM: NON- CON-STATIONARY NON- PARAMETRIC STATE SPACE MODEL
Considering that time series are non-stationary in many real situations, we now add a higher-level model to NPSSM to allow it to capture the vire-narying characteristics of the process. We propose the Nta-station Morrmarrect State Space Model(N-NPSSMM which is formulated
91 (92 + m.), Za = 11 E Pa(za)} ce. Cxt) fe((cc-)/aa..))) (4)
Nonlinear emission
Structural causal latent transition
Time-varying change factors
where G., similar to Est, are mutually independent (i.e., spatially and temporally independent) random noises. fc(-) is the transition function for the time-varying change factors, which is also formulated in a general form of structural causal model. It includes the vanilla SSMs in Eq. (1) as a particular case in which the time-varying change factors do not exist. It also includes the time-varying parameter vector autoregressive model (Lubik & Matthes, 2015) as special case, which allows the coefficients or the variances of noises in the linear auto-regressive model to vary over time following specified law of motion. In contrast to explicitly specifying how time-varying change factors affect the transition process, our model is quite general in that we a low-dimensional vector e, characterize the vame-narying information and use it as an input for the transition model. Establishing the theoretical identifiability of this model is technically even more challenging, and our empirical results on various simulated data sets strongly suggest that it is actually identifiable.
3 ESTIMATION FRAMEWORK
Given our identifiability results, we propose the estimation procedures of NPSSM in Eq. (2) and N-NPSSM in Eq. 4). Since NPSSM is special case of N-NPSSM, below, we consider only the estimation framework of N-NPSSM, and its properly constrained version will apply to NPSSM. The model architecture is shown in Fig. 2(a). Here x, and X, are the observed and reconstructed variables. Similarly, Z₁ and 2, denote the truth and estimated latent variables. The overall framework is structural variational auto-encoder which learns the underlying latent temporal process via the latent causal model and then build the auxiliary latent prediction model on the uncovered latent variables. The implementation details are in Appendix A4.
4

---

Under review as a conference paper at ICLR 2023
(a) VAE Architecture
(b) Transition Prior Model
Build latent causal model
dbc dey
p(Culfe-1)
p(t.
6 6:303 1.0
Encader
2
Penalty with transition prior (e(c@)))l(((((lee--a (lon(G)lllGd)ee.))
i
1.00.25.300000 0.
Encoder 9,
dn
X--1
7
p 2 12
Rx) (eex)
Build latent predictor
:
Auri
One step prediction error
:
3..00.35.0022
Encoder
Pareed (
X1-1
2t-1
Figure 2: Fig (a) demonstrates the overview of our structural VAE framework. It mainly includes the latent causal model and latent prediction model. In latent causal model, it recovers latent process via minimizing reconstruction error and the regularization between factorized posterior ( ) ((C1:)) and transition prior p(21.T) ). p(êi:7). which implicitly models the temporal dynamics. Fig (b) shows the transition prior model, representing the latent causal processes 2, and č₁-
Latent Causal Model To facilitate our implementation, we adopt the Variational Auto-Encoder (Hsu et al., 2017), which implicitly implies that the measurement noise is additive. This is a particular case of the post-nonlinear mixing procedure given in Eq. (2). is challenging to model the causal de- pendencies among observed and latent variables, especially for the design of the encoder/decoder. An alternative to follow dynamic VAE (Girin et al., 2020) to encode the latent causal relationships the encoder explicitly. To make the estimation more efficient, inspired by (Klindt et al., 2020; Yao et al., 2022), the transition prior p(a 22)
p(21) ......pp... p(2)) II- ( Cx)
and p(c1..... è = p(č1 I p(č:/2;-1 to encode latent causal relationships and approximate the joint probability of posterior on 21:r and er.r with factorized form. Specifically, the poste-
rior(encoder) for Z1:7 is defined as I[ ((, xt). and similarly the posterior(encoder) for C1.r is defined as II ge
An alternative to model the transition prior probability (transition prior model) p(2 ) 3 Gx) is to leverage the forward prediction function 2,1 = = (( 1 C2, Est). However, we argue that forward prediction with fixed loss cannot model latent processes without parametric form. For example, the latent process ZE,4 = gs ((zz-r)+=
He cannot be estimated by forward prediction function with squared loss because of the coupling effect from noise variable and cause variable. Thus, we propose to obtain this transition prior by explictly model the noise function, which can be treated as inverse latent transition function, i.e. éµ ri (2zz. Ct2, (2 Particularly, they are implemented by a set of separate MLP Networks {r:} (to satisfy the independent noise condition in Thm 1), which take the estimated latent causal variables and time-varying change factors input and output the noise terms. By applying the change of variables formula to the transformation, the transition probability can be formulated as:
Ör; θž,1
12 H/ c.) = Pes. (ri(Zse, ćt, (2e- 14
(5)
Because of the mutually independent noise assumption, the Jacobian is lower- lower-triangular We can efficiently calculate its determinant as the product of each element. By applying the independent noise assumption, the transition probability can be formulated as:
Dr:
log
Σ 1=1
log p(ex) +
Σ log
02.t
Given this, the transition probability factorized noise distribution ΣL_ log (a). To the estimated noises terms, we model each noise distribution p(εa) as transformation from the standard normal noise (0, through function
0,) can be efficiently evaluated using the
x(.), which can be formulated as p(εxt) PA((..)) (8 ((se)) ( dees
Fortunately, we do not
5

---

Under review as a conference paper at ICLR 2023
need to explicitly estimate the term compensate it. Similarly, we define the transition probability of change factors C₁ as log Pe(Cz/Ot-1) Σn= log + Σi= log
de (fx)
since inverse causal transition functions {r,} could
dest
du:
where uy denotes the inverse change transition function.
Auxiliary Latent Prediction Model While the above latent causal model could estimate latent variable žn in the non- con-parmmetric form, could not explicitly model the forward prediction relationship which is required for the forecasting task. Therefore, we propose to train the axuiliary latent prediction models. With penalization hyperparameter, one can view this module as a regularization the temporal predictability of the learned latent processes, for the purpose of time series forecasting. Formally, the axuiliary latent prediction models is defined as Ppred (2
enforce
C,, ε2), which takes the recovered latent variables { change factor é₁ and noise & as the input. Note that é is not available at time / - 1 in prediction mode. One straightforward solution is to build an extra prediction model for change factor é. Interestingly, we can skip this step, since change factor C had to be inferred from the latent variables {2,-r)f-0 as well, like the definition of cesterior(encoder (c, via ( ( Eg). Specifically, we use the LSTM network to implement this predictor. The noise & is generated from the inverse latent transition function rr é₁2 { (2c-r) )+z1) in the training phase, while it is sampled from the standard normal distribution N NN(0, 1) in the forecasting phase.
Therefore, we can directly learn the auxiliary latent predictor
This way, the prediction procedure decouples the forecasting task into three steps: (1). The encoder recovers the latent causal representation from the observed data; (2). Next-step prediction is gener- ated via the latent prediction model in the latent space; (3) prediction results are transformed into observation space by the decoder.
By taking into account the above two components, we jointly train the latent causal model and the latent prediction model with the following objective L:
Optimization
logp=(x,lz:) 3DD (00 (2:))) (1) D (
latent causal model
log Perea (Zelér, (
auxiliary latent predictor
where Ps (xx//z) and Pored 2e 22, {2g- ) denote the decoder distribution and prediction distri- bution, in which we use MSE loss for the likelihood.
4 RELATED WORK
Identifiability of State-Space Models It is well-known that the linear state space model with additive Gaussian noise unidentifiable (Arun & Kung, 1990), thus can not recover the latent process. Under specific structural constraints the transition matrix, (Xu, 2002) find it identifiable. (Zhang & Hyvärinen, 2011) 1) further consider the linear non-Gaussin Gaussian setting and prove that when the emission matrix is of full column rank and the transition matrix is of full rank, the model is fully identifiable. In the non-stationary environment, (Huang et al., 2019) prove that the time-varying linear causal model is identifiable if the additive noise is a stationary zero-mean white noise process. For the vector autoregressive model with the latent process, (Jalali & Sanghavi, 201 1) show that if the interactions between observed variables are sparse, interactions between latent variables and observed variables are sufficient, the transition matrix can be identified. (Geiger et al., 2015) find that if the additional genericity assumptions hold and the exogenous noises are independent non non-Gausiian, then the transition matrix is uniquely identifiable. In contrast, our work considers remarkably flexible state space model, which does not require constraints like linear transition or additive noise. Even so, we find that the latent process is generally identifiable.
Deep State-Space Models To leverage advances in deep learning, (Chung et al., 2015; Fraccaro al., 2016; Karl et al., 2016; Krishnan al., 2017) draw connections between the state space models

---

Under review as a conference paper at ICLR 2023
and RNN and propose the dynamic VAE framework to model temporal data. For (Chung et al., 2015). they associate the latent variables in the state space model with the deterministic hidden states of RNN. As such, the transition model is nonlinearly determined by the RNN and the observation model. These works propose different variants of deep learning architectures to parameterize transition and emission models enhance expressiveness. These models vary in how they define the generative and inference model and how they combine the latent dynamic variables with RNN to model temporal dependencies (Girin et al., 2020). Meanwhile, the training paradigm of these works is similar to the VAE methodology. Inference networks define a variational approximation to the intractable posterior distribution of the latent variables. Approximation inference is applied, which may lead to sub-optimal performance. To address it, (Fraccaro et al., 2017; Rangapuram et al., 2018; Becker et al., 2019) take advantage of Kalman filters/smoothers to estimate the exact posterior distribution. For (Fraccaro et al., 2017), they use the standard Gaussian linear dynamical system to model the latent temporal process. The hidden states of RNN are used to predict the parameters of this dynamical system to enable closed-form Bayesian inference. However, these methods require expensive matrix inversion operation and the linear transition model limits the expressiveness. An alternative (Zheng al., 2017) is to use variational sequential Monte Carlo to draw samples from the posterior directly. Recently, (Klushyn et al., 2021) propose a constraint optimization framework to obtain accurate predictions of the dynamical system. They achieve it by combining amortized variational inference with classic Bayesian filtrriggsmooohingg to model dynamics. These works present different methods infer the latent variables more accurately. Besides, some work leverage neural SDE model the transition process (Yildiz et al., 2019). While these works enhance the expressivity of the transition model with deep architectures, they are still constrained by the additive noise form, which can be treated as special cases of our work.
5
EXPERIMENTS
To show the efficacy of N-NPSSM for identifying latent processes and forecasting, we apply it to various synthetic and real-world datasets with one-step-ahead forecasting tasks.
Evaluation Metrics To evaluate the identifiability of the learned latent variables, we report Mean Correlation Coefficient (MCC), which is a standard metric in ICA literature for continuous variables. We use Spearman's 's rank correlation coefficients to measure the discrepency between the ground- truth and estimated latent factors after component-wise transformation and permutation are adjusted (details are given in Appendix A3.2). MCC reaches when latent variables are identifiable up to componentwise invertible transformation and permutation. To evaluate the forecasting performance, we report the Mean Absolute Error (MAE) and p-risk, which quantifies the accuracy of a quantile p of the predictive distribution. Formally, they are defined as:
MAE Σ Xxa - -xiel, Ro-loss
Σ Xx) (1 101-))a -D) i,
(8)
where XG, is the empirical p-quantile of the prediction distribution and I is the indicator function. For the probabilistic forecasting models, forecast distribution is estimated by 50 trials of sampling, and 82 is calculated by the predicted median value.
Baselines We compare N-NPSSM with typical deep forecasting models and deep state-space models: (1) LSTM(Hochreiter & Schmidhuber, 1997) which is baseline for the deterministic deep forecasting model; (2) (SepnR(Salinas et al., 2020) which is an peacoder-based probabilistic deep forecasting model; (3) VRNN(Chung al., 2015) and (4) KVAE(Fraccaro al., 2017) which are deep state space models. Note that KVAE implicitly considers time-varying change factors by formulating the transition matrix as weighted average of set of base matrices and using an RNN to predict the combination weights at each step.
5.1 SYNTHETIC EXPERIMENTS
We generate synthetic datasets that satisfy the identifiability conditions in the theorems. In particular, we consider four representative simulation settings to validate the identifiability and forecasting performance under fixed causal dynamics (Synthetic fixed causal dynamics with distribution shift (Synthetic2), time-var marying causal dynamics with inter-dependent change factors (Synthetic3) and time-varying causal dynamics with changing causal strengths (Synthetic4) (more details of data generation given in Appendix 13.1.1). For all the synthetic datasets, we set latent size 8, and
7

---

Under review as a conference paper at ICLR 2023
Table 1: Identifiability and forecasting performance for the four synthetic datasets (more empirical results can be found in A3.3). Note: "N/A" indicates the deterministic model LSTM is not applicable to output predictive distribution
Synthetic
Synthetic 3
Method
MCC
MAE
Roos-loss
MCC
MAE
Rlos-
LSTM
0.110=0.02
0.416:10.03
N/A
0.140:0.02
0.583=10003
N/A
KVAE
0.406=0.02 0.404+0.01
2.237=0.05
0.513
0.455=0.02 0.555:10.03
6.446+0000
VRNN
0.520=0.08
0.515=0.02
4.341=0.06
0.543=0.01
3.50
DeepAR
0.267 0.087=00.03
0.353+10.03
0.432:10.02
0.095=
0.606++0.02
N-NPSSM
0.987-0.01
0.054
0.220-10003
0.998-10.01
0.057
0.363
Synthetic
Synthetic 4
Method
MCC
MAE
Ro.9-loss
MCC
MAE
Ro.9-loss
LSTM
0.199=0.02
0.498:10.08
N/A
0.22710.02
0.64110.02
N/A
KVAE
0.407=0.02 0.479-+0.05
25.94+0.61
0,478+10.03
0.480=+0001
2.090++0.22
VRNN
0.491=0.08 0.637++0.06
28.58=0083
0.397+00.0
0.498=0.02
1.167+0.01
DeepAR
0.297=0.01 0.133:10.03
3.28410.04
0.087=0.01 0.35110.03
0.179:00.01
N-NPSSM
0.995=0.01
0.069110001
1.86610.02
0.992110001
0.081=1.01
0.169
the maximum latent process lag is set to L = 2. For time-varying settings, the dimension of change variables set to 4. The emission function g(-) a random three-layer MLP with LeakyReLU units.
As shown in Table 1, N-NPSSM can successfully recover the latent processes under different settings, as indicated by the highest MCC (close to 1). In contrast, the baseline models, including the deep forecasting model and deep state-space models, cannot recover the latent processes. Besides, our method gives the best forecasting accuracy, as indicated by the lowest MAE and Ru.9-loss. In Figure 4, each left sub-figure shows the MCC correlation matrix of each factor, while each right sub-figure shows the scatter plot of recovered factors and truth factors. We can find that the time-delayed causal relationships are successfully recovered, as indicated by high MCC for the causally-relate factors. Besides, the latent causal variables are estimated up to permutation and componentwise invertible transformation (more empirical results are given in A3.3).
To investigate the consequence of the viola- tion of the critical assumptions. We create an- other two datasets: (1) with dependent process noise terms, and (2) with additive Gaussian noise terms, in which (1) violates the mutually independent noise condition. and (2) violates the linear independence condition. From Fig- ure 3, we can find that violating the indepen- dent noise condition deteriorates the identifia- bility results significantly. Additionally, when the latent processes follow linear, additive Gaussian temporal model (thus, the linear independence condition is violated), the identifiability results are also distorted. However, if the noise terms are slightly non-Gaussian (we change the shape parameter of the generalized Gaussian noise distribu- tion from 3 = 2.0 to 1.5 or 3 = 2.5, we can observe the final MCC scores increase significantly and the underlying latent processes become identifiable in both non-Gaussian noise scenarios.
Violation of assumptions
1.00
Dependent Generalized Gaussian noise (beta=1.5)
0.75
Gaussian noise (beta-2.0) Generalized Gaussian noise (beta-2.5)
0.00
25K
50K
100K
Sleps
Figure MCC trajectories of NPSSM for temporal data with clear assumption violations.
5.2 REAL DATA EXPERIMENTS
We evaluate N-NPSSM three real-world datasets: Economics, Bitcoin and FRED. Economics and Fred contain a set of macroeconomic indicators, while Bitcoin includes the potential influencers of the bitcoin price (The detailed data descriptions and preprocess given in Appendix A3.1.2). As shown Table 2, N-NPSSM outperforms all competitors in terms of both MAE and llos- which verifies the effectiveness of N-NPSSM (more qualitative experiments are given in Appendix A3.3).
8

---

Under review as a conference paper at ICLR 2023
NCC+6..80 2886
MCC
Estimated latents
Starks
latents
latents
(a) Synthetic (Fixed causal dynamics)
(b) Synthetic (Fixed causal dynamics with distribu- tion shift)
MCCw0.995
PCC+O
9--000
0.28
Esliratte liferts
Eximater iatents
True
The Intents
(c) Synthetic (Time-varying causal dynamics with (d) Synthetic 4 (Time-varying causal dynamics with inter-dependent change factors)
changing causal strength)
Figure 4: MCC for causally-related factors and scatter plots between estimated factors and true factors on four synthetic datasets.
Table 2: Forecasting performance on three real-world datasets
Economics
Bitcoin
FRED
Method
MAE
Ro.s-loss
MAE
Ro.g-loss
MAE
Ro.g-loss
LSTM KVAE
0.007
N/A
0.747+0.04
N/A
0.632
N/A
0.618
0.11
0.551+00.01
0.290+0003
0.619++003
0.88
VRNN
0.786+0011
1.534+0.00
0.759+0006
0.728+0000 0.222+0001
1.045+0008
DeepAR
0.741=0.008 1.288=0.12
1.465:10001
0.317=0.05
0.752+0.05
0.65410.04
N-NPSSM
0.603.10.05
1.19010.11 0.40311.01
0.14310.01
0.48411.03
0.580
CONCLUSION AND FUTURE WORK
In this work, we propose general formulation of state-space models called NPSSM, which includes a completely non-parametric transition model and a flexible emission model. We prove that even though it is flexible, it generally identifiable. Moreover, we further propose N-NPSSM to capture the possible time-varying change property of the latent processes. We further develop the estimation procedure based on VAE and make use of it for forecasting tasks. Empirical studies on both synthetic and real-world datasets validate that our model could not only identify the latent process but also outperform baselines in the forecasting task. While we do not establish theories with time-varying change factors, we have demonstrated through experiments the possibilities of generalizing identifiability results to this setting. Extending our theories to address the issue of a completely con-parametric emission model will also be one line of our future work. Another interesting direction to apply this framework to other time series analysis intelligence tasks, like anomaly detection and change point detection, which is also interesting directions.
REPRODUCIBILITY STATEMENT
Our code for NPSSM is attached supplementary material. The implementation details can be found in A4. For theoretical results, the assumptions and complete proof of the claims are A2.2. For synthetic experiments, the data generation process is described in A

---

Under review as a conference paper at ICLR 2023
REFERENCES
KS Arun and SY Kung. Balanced approximation of stochastic systems. SIAM journal on matrix analysis and applications, 11(1):42-68, 1990.
Philipp Becker, Harit Pandya, Gregor Gebhardt, Cheng Zhao, C James Taylor, and Gerhard Neumann. Recurrent kalman networks: Factorized inference in nigh-dimensional deep feature spaces. In International Conference on Machine Learning, pp. 544-552. PMLR, 2019.
Adi Ben-Israel. The change-of-variablle formula using matrix volume. SIAM Journal on Matrix Analysis and Applications, 21(1):300-312, 1999.
Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Bernie Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Dohlke-Schneider David Salinas, Lorenzo Stella, et al. Neural forecasting: Introduction and literature overview. arXiv preprint wxXv:2000..12444 2020.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional VRNNS for video prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7608-7617, 2019.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. recurrent latent variable model for sequential data. Advances in neural information processing systems, 28, 2015.
James Durbin and Siem Jan Koopman. Time series analysis by state space methods, volume 38. OUP Oxford, 2012.
Marco Fraccaro, Søren Kaae Sønderby, UIrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. Advances neural information processing systems, 29, 2016.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. Advances in neural information processing systems, 30, 2017.
Philipp Geiger, Kun Zhang, Bernhard Schoelkopf, Mingming Gong, and Dominik Janzing. Causal inference by identification of vector autoregressive processes with hidden components. Interna- tional Conference on Machine Learning, pp. 1917-1925. PMLR, 2015.
Zoubin Ghahramani and Geoffrey E Hinton. Switching state-space models. Technical report, Citeseer, 1996.
Zoubin Ghahramani and Geoffrey E Hinton. Variational learning for switching state-space models. Neural computation, 12(4):331-864 2000.
AmirEmad Ghassami, Negar Kiyavash, Biwei Huang, and Kun Zhang. Multi-domain causal structure learning in linear systems. Advances in neural information processing systems, 31, 2018.
Laurent Girin, Simon Leglaive, Xiaoyu Bie, Julien Diard, Thomas Hueber, and Xavier Alameda- Pineda. Dynamical variational autoencoders: A comprehensive review. arXiv preprint arXiv:2008. 12595, 2020.
Joshua Glaser, Matthew Whiteway, John Cunningham, Liam Paninski, and Scott Linderman. Re- current switching dynamical systems models for multiple interacting neural populations. Advances in neural information processing systems, 33:14867-14878. 2020.
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey Webb, Rob J Hyndman, and Pablo Montero- Manso. Monash time series forecasting archive. arXiv preprint 2:X1v:2105.06643, 2021.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()):
1735-1780, 1997.
10

---

Under review as a conference paper at ICLR 2023
Elizabeth E Holmes, Eric J Ward, and Kellie Wills. MARSS: multivariate autoregressive spate-space models for analyzing time-series data. R journal, 2012.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in neural information processing systems, 30, 2017.
Biwei Huang, Kun Zhang, Mingming Gong, and Clark Glymour. Causal discovery and forecasting in nonstationary environments with state-space models. In International conference on machine learning, pp. 2901-2910,, 20. PMLR, 2019.
Biwei Huang, Kun Zhang. Jiji Zhang, Joseph D Ramsey, Ruben Sanchez-Romero, Clark Glymour, and Bernhard Schölkopf. Causal discovery from beterogeneous/noosttiionaay data. J. Mach. Learn. Res., 21(89): 1-53, 2020.
Ali Jalali and Sujay Sanghavi. Learning the dependence graph of time series with latent factors. arXiv preprint arXiv: 1887, 2011.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick Van der Smagt. Deep varia- tional bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint a60iv: 1605.06633 2016.
Yitzhak Katznelson. An introduction to harmonic analysis. Cambridge University Press, 2004.
llyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR, 2020.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement natural data with temporal sparse coding. arXiv preprint arXiv:2007. 10930, 2020.
Alexej Klushyn, Richard Kurle, Maximilian Soelch, Botond Cseke, and Patrick van der Smagt. Latent matters: Learning deep state-space models. Advances in Neural Information Processing Systems, 34, 2021.
Rahul Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Rahul G Krishnan, Uri Shalit, and David Sontag. arXiv:1511.05121, 2015.
Deep kalman filters. arXiv preprint
Simon Leglaive, Xavier Alameda-Pinedd Laurent Girin, and Radu Horaud. A recurrent variational autoencoder for speech enhancement. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 371-375, IEEE, 2020.
Ismael Lemhadri, Feng Ruan, and Rob Tibshirani. Lassonet: Neural networks with feature sparsity. In International Conference on Artificial Intelligence and Statistics, pp. 10-18. PMLR, 2021.
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: survey. Philosophical Transactions of the Royal Society A, 379(2194):22200200) 2021.
Thomas A Lubik and Christian Matthes. Time-varying parameter vector autoregressions: Specifica- tion, estimation, and an application. Estimation, and an Application, 2015.
Xiaolin Luo. lime-varying autoregressive modeling of nonstationary signals. 2005.
Jaakko Luttinen, Tapani Raiko, and Alexander Ilin. Linear state-space model with time-varying dynamics. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 338-353. Springer, 2014.
Judea Pearl. Causality. Cambridge university press, 2009.

---

Under review as a conference paper at ICLR 2023
Fotios Petropoulos, Daniele Apiletti, Vassilios Assimakopoulos, Mohamed Zied Babai, Devon K Barrow, Souhaib Ben Taieb, Christoph Bergmeir, Ricardo Bessa, Jakub Bijak, John Boylan, et al. Forecasting: theory and practice. International Journal Forecasting, 2022.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances in neural information processing systems, 31, 2018.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork variational autoencoders. Advances in Neural Information Processing Systems, 34, 2021.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner. Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109-53662--634 2021.
Chukiet Sodsri. Time-varying autoregressive modelling for nonstationary acoustic signal and its frequency analysis. The Pennsylvania State University, 2003.
Binh Tang and David S Matteson. Probabilistic transformer for time series analysis. Advances in Neural Information Processing Systems, 34:23592-23608, 2021.
Lei Xu. Temporal Factor Analysis (TFA): stable-identifiable family, orthogonal flow learning, and automated model selection. In Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No. 02CH37290), volume 1, pp. 472-476. IEEE, 2002.
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data, arXiv preprint 10.0422) 2021.
Weiran Yao, Guangyi Chen, and Kun Zhang. Learning latent causal dynamics. arXiv preprint arXiv:2202.04828, 2022.
Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. Ode2vae: Deep generative second order odes with bayesian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, pp. 5670-5679. PMLR, 2018.
Kun Zhang and Aapo Hyvärinen. Distinguishing causes from effects using nonlinear acyclic causal models. In Causality: Objectives and Assessment, pp. 157-164. PMLR, 2010.
Kun Zhang and Aapo Hyvärinen. A general linear non- gaussian state-space model: Identifiability. identification, and applications. Asian Conference on Machine Learning, pp. 113-128, 128. PMLR. 2011.
Kun Zhang and Aapo Hyvarinen. On the identifiability of the post-nonlinear causal model. arXiv preprint arXiv: 2599, 2012.
Xun Zheng, Manzil Zaheer, Amr Ahmed, Yuan Wang, Eric Xing, and Alexander J Smola. State space LSTM models with particle MCMC inference. arXiv preprint arxiv: 2017.
12

---

Under review as a conference paper at ICLR 2023
Appendix for
"Non-Parametric State-Space Models: Forecasting"
Identifiability,
Estimation and
Appendix organization:
A1 Extended Related Work
13
A2 Identifiability Theory
14
A2.1 Notations
14
A2.2 Proof of Identifiability Theory
14
A3 Experiment Details
16
A3.1 Datasets
16
A3.1.1 Synthetic Dataset Generation
16
A3.1.2 Real-world Dataset
16
A3.2 Evaluation Metric
17
A3.3 Additional Experimental Results
17
A3.3.1 Additional Time Series Forecasting Results
17
A3.3.2 Ablation Studies
18
A3.3.3 Empirical Studies for Violation Assumptions
18
A3.3.4 Model Complexity
19
A3.3.5 Interpretability Analysis
19
A4 Implementation Details
20
A4.1 Network Architecture
20
A4.2 Training Details
20
A1 EXTENDED RELATED WORK
Time-Varying State-Space Models In many real situations, the temporal process may vary over time. This inspired the early efforts to allow the parameters of vector autoregressive models change over time (Sodsri, 2003; Luo, 2005), which consider the effect of time variation in coefficients and the variance of noises. These works be treated as special cases of the state space models, which directly learn the transition in observation space. Time-varying linear state space models (Luttinen al.. 2014: Holmes al.. 2012) make one step further, vector autoregressive models. A similar research topic is the switching-regin (Ghahramani & Hinton, 1996; 2000; Glaser et al., 2020), which assumes the transition lies in a set of linear dynamical models and model the transition process with hidden Markov models. Thus, these models cannot capture the continuous change over time. Recently, some deep state space models have implicitly considered the time-varying characteristic of data. Both of these works (Rangapuram et al., 2018; Fraccaro et al., 2017) consider the Gaussian linear dynamical systems the latent space. In (Rangapuram et al., 2018), the transition/emission matrices and two noise covariance matrices are predicted by RNN at each step. In (Fraccaro et al., 2017), they assume the transition/emission matrices are weighted average of set of base matrices, where the RNN model predicts the weights
is more powerful and general than
state space models
13

---

Under review as a conference paper at ICLR 2023
each step. Note that all these existing works require specifying how time- varying change factors affect the transition process, which may not be applicable in practice without prior knowledge. In contrast, our model is flexible since consider general transition model, and the time- varying change factors are treated as the input for the transition process.
A2
IDENTIFIABILITY THEORY
A2.1 NOTATIONS
We summarize the notations used throughout the paper in Table
Table Al: Notations.
Index
t
Time index
i, i, k
Variable element (channel) index Time lag index
L
Maximum time lag for latent variable
Le
Maximum time lag for time-varying change factors
Variable
Xx
Observation data Reconstructed observation latent variable estimated latent variable time-varying change variable estimated time-varying change variable Set of direct cause nodes/parents of node 2/ measurement noise term Process noise term noise term for time-varying change factor
8,
Z1
21
Ct.
Pa(z (, {2(-T}
6st
Çt
Function and Hyperparameter
Distribution function (e.g., Pea, is the distribution of eax-)
g
Arbitrary nonlinear and invertible mixing function Nonlinear transition function for 2 Nonlinear transition function for e₁ Post nonlinear distortion function
f:
So
h
Ti
Learned inverse transition function for residual é; Learned inverse change transition function for residual & Weights in the augmented ELBO objective
ug
8,7,O
Latent size
Permutation operation
T
Component-wise invertible nonlincarities
A2.2 PROOF OF IDENTIFIABILITE THEORY
Before the proof, we first produce Lemma 1, which presents the identifiability of latent variables in fixed latent dynamics. This result will used in the proof of Theorem
Lemma 1. (Theorem in (Yao et al., 2022)) The fixed latent causal dynamics takes on the following form:
X1 = g(Z)) Za = f((12,1-11 Pa(zz)),,Ex).
Let Tht e log plzx 2,-1), (t)(t) is twice differentiable in Zke and is differentiable in Zt 1, 1. 2, ...... Suppose there exists invertible function & that maps x2 24, i.e., 2, g(x:), such
14

---

Under review as a conference paper at ICLR 2023
that the components of in are mutually independent conditional on 21-1- Let
Pipee
8rjt
Tjstt
Vk,t -
02 022,
Tett
Tktt
8 Tktt
dzk dz1,c-1 02 022.111 dz 02n.t-1
for each value of Zr V1,t: V1,2, V2,3, V2,t, Vn,t, Va.. as 2n vector functions in 21,1-1, 22,t-1 2n,t-1, are linearly independent, then Zy must be an invertible, component-wise transformation of permuted version of 2;-
Second, we consider the additive noise model, in which Φ1 the identity mapping. To identify the noise-free free distribution g(z,) from noisy data with assumption 1, we follow the idea of using convolution theorem to decouple measurement error (Khemakhem al., 2020). Since the volume of a matrix vol A is defined as the product of the singular values of A. We could obtain that vol det when is invertible. We use A the change of variables formula to replace the absolute determinant of the Jacobian (Ben-Israch, 1999). Suppose the joint distribution for observed variables Pf...p(c) (x, 11) and f.g.p(e) [x (2-2)_1) are matched almost everywhere. Then:
[ Pfiple) (zi Z (2 dz = / Pf.pte) ( (x )z):
(10)
[ Prip(e) (2 (allz-1))pe (x: g(2+))
I Pf.ste) (2/ 22 Pm (x+ g 21 )
(11)
According to the Jacobian matrix the mapping from x, = ((()) and 8, = ((zz). we have
[ (r (xx) 2 (x))) (X:) (x+ x,)
[ P Pl.pte (9 (
(2x) (xi Xx, )dx,
(12)
Let us assume Pfo())a (xi) -Ps (g (x,) 1z- - Ix (xt), and then we have
J P..tee..oa. (Re)) (x+ d Pf (xe) Pm, (x, x,)),
(13)
According to the convolution theorem (Katznelson, 2004) that the convolution in one domain (e.g.. time domain) equals point-wise multiplication in the other domain (e.g., frequency domain). We could obtain that,
(Pirlc)am (x.) = (Ffstc)-a Pre (x:),
(14)
11 (w) FlDj, (w),
(15)
where denotes the convolution operator and F|- denotes the Fourier transform. We can find Pur F [Pnm by the definition of characteristic function in Assumption 1. Then, we can remove Pn. (w) the term from both sides, as it is non-zero almost everywhere. We have,
F FfPr.atc..0.. = FlP F1D
(16)
(x) = -f.stc)....,,,x11 (x.).
(17)
Thus, we can conclude that if the distributions are the same with additive noise, the noise-free distributions are still the same. Combined with the results from Lemma that the latent variables are identifiable up to permutation and component-wise invertible transformation.
Lastly, we consider the effect of post non-linear function 91 (-). Let us denote =2 + +1,, then (91 o (91))1 o ooggggl (x+) We can further assume that 91 = 91 ((gi) -1 o 91) = 91 o 9a. in which Øa represents the indeterminacy on the space of X, Following the proof of Theorem I of (Klindt et al., 2020), we have that gs can only be a bijection if both 92-91 91 are injective functions. Thus, we can treat it as adding a component-we wise invertible nonlinear function 93 on x,, which does not affect the identifiability of z, up to permutation and component-wis wise invertible transformation. Therefore, NPSSM in is identifiable.
the learned post non-linear function x, = Ô (X,) can be written as
15

---

Under review as a conference paper at ICLR 2023
A3
EXPERIMENT DETAILS
A3.1
DATASETS
A3.1.1
SYNTHETIC DATASET GENERATION
To evaluate the identifiability and forecasting capability of our model under different scenarios, we generate the synthetic data with 1) fixed causal dynamics; 2) fixed causal dynamics with distribution shift; 3) time-varying causal dynamics with changing noise variances, and 4) time-varying causal dynamics with changing causal strengths. We use the first 80% data for training and the rest 20% for evaluation.
Stationary Causal Dynamics For the fixed causal dynamics, we generate 100,000 data points based on the following equation:
Fk,t = qro (x)((()))) + by (((()=))) Ek,
(18)
Here, Ce,t is the process noise, which are sampled from i.i.d. Gaussian distribution (a 0.1). 61,1 C2.1. Ca,t are mutually independent and independent of Zz-1. The process noise terms are coupled with the history information through multiplication with the average value of all the time- lagged latent variables. We set the latent size 8 and the lag number of the process L = 2. We apply a 2-layer MLP with LeakyReLUU as the state transition function. The emission function is random three-layer MLP with LeakyReLU units.
Stationary Causal Dynamics with Distribution Shift We follow the same way as the setting of fixed causal dynamics and generate 80,000 data points for the training set. To simulate distribution shift in the test phase, we vary the values of the first layer of the MLP in the test set and generate 20,000 samples. The entries of the kernel matrix of the first layer are uniformly distributed between [-1,1].
Time- Varying Causal Dynamics with Changing Causal Strengths For the time-varying causal dynamics with changing causal strengths, generate 100,000 data points based the following equation:
Ck = Ck,t-1 + SN.t
Zk,t (00((=((0000 (121-)), + bu((zc-+)) Cket
(19)
where the noises Ske are sampled from i.i.d. Laplace distribution (o 1). We take the change factor C₁ as an input for the latent transition function for Z
Varying Causal Dynamics with Inter-Dependent Change Factors For the time-varying causal dynamics with inter-dependent change factors, instead of considering the independent sources using temporal dependencies, here we consider the inter-dependence across the different variable index. Formally we generate 100,000 data points based on the following equation:
C = Cc Sk.t
Zk = gu () by be((zi-7),c) c.) Ehito
(20)
where C' is the transition matrix for change The The noises Çat sampled from i.i.d. Laplace distribution (a = 1). In the latent transition process for 2,, noise terms are coupled with the history information and change factors through multiplication with the average value of all the time-lagged latent variables 2:-7 and current time-varying change factor c,,
13.1.2 REAL- DATASET
Three roal-world datasets are used to evaluate the forecasting performance of the proposed model. We use the first 80% data for training and the rest 20% for evaluation.
16

---

Under review as a conference paper at ICLR 2023
Table A2: Comparison with additional baselines on real-world datasets.
Economics
Bitcoin
FRED
Methods
MAE
Ro.s-Loss
MAE
Ro.s-Loss
MAE
Ro.9-Looe
DKF
0.784
1.877
0.516
0.148
0.722
0.901
RVAE
0.798
1.775
0.524
0.183
0.666
1.128
SRNN
0.710
1.266
0.530
0.202
0.787
1.289
DSAE 0.736
1.594
0.493
0.227
0.650
1.061
N-NPSSM
0.603
1.190
0.403
0.143
0.484
0.580
Economics The economics dataset was used in (Huang et al., 2019). We investigate the time-lagged causal relationships among 10 macroeconomic variables ranging from CPI, inflation unemployment rate with monthly data from 1965 to 2017 in the USA The data are normalized by subtracting the mean and dividing them by the standard deviation.
The bitcoin dataset was used in (Godahewa et al., 2021). We investigate the time-lagged causal relationships about 16 daily time series which have potential influences on the bitcoin price. Specifically, it includes hash rate, block size, mining difficulty, public opinion, etc. The data are normalized by subtracting the mean and dividing them by the standard deviation.
Bitcoin
The FRED dataset was used in (Godahewa et al., 2021). We investigate the time-lagged causal relationships about 107 monthly time series 4. It contains set of macroeconomic indicators from the Federal Reserve bank. The data are normalized by subtracting the mean and dividing them by the standard deviation.
FRED
A3.2 EVALUATION METRIC
Mean Correlation Coefficient (MCC) MCC is a standard metric for evaluating the recovery of latent factors in ICA literature. We first apply a nonlinear regression the recovered factors, aiming to get rid of the component-wise transformation indeterminacy, for each possible pair of the estimated factor and the true one. Then, we calculate all pairs of correlation coefficients (the absolute values of the Spearman's ' rank correlation coefficients) between ground-truth latent factors and the estimated latent factors (after the component-win transformation). We further solve a linear sum assignment problem assign each latent component to the ground-truth component that best correlates with it, thus finding the correspondence between the estimated factors and the true ones in the latent space. A high MCC means one successfully recovered the true latent factors, up to invertible, component transformation and permutation.
A3.3
ADDITIONAL EXPERIMENTAL RESULTS
A3.3.1
ADDITIONAL TIME SERIES FORECASTING RESULTS
We list additional comparison with more baselines from typical deep state space models, in- cluding DKF(Krishnan et al., 2015), SRNN(Fraccaro al., 2016), RVAE(Leglaive et al., 2020), DSAE( Yingzhenn & Mandt, 2018). As shown in Table A2, our method still consistently outperforms the additional baselines.
In Table A3, we evaluate the proposed model N-NPSSM and basclines on more real-world datasets NN5 OIKOLAB and Pedestrain". We can find that N-NPSSM still outperforms the baselines.
DDwnlooaded from http://wwww.ieegobblcccooommmmm
'Dwnlooaded from https///zeenaoooog//cccc//1/1222 2ttp:/22000.042maa2aeeeeaoo YzPm7exB200
'Downloaded from http:://sanndoooogg////o/////4/5 Bzh
https://eeadooorg//cco//////
http:://zennoooorr/rccoo/1f11111 httt:://eeonddooorryccc/4446/6/1
17

---

Under review as a conference paper at ICLR 2023
Table A3: Comparison with baselines on additional real-world datasets.
NN5
OIKOLAB
Pedestrain
Methods
MAE
Ro.9-Loss
MAE
Ro.9-Loss
MAE
Ro.9-
LSTM
0.890
N/A
0.422
N/A
0.739
N/A
KVAE
0,573
8.858
0,424
1.190
0.362
6.475
VRNN
0.869
9.259
0.842
1.310
0.769
9.794
DeepAR
0.567
6.405
0.191
0.325
0.387
4.573
N-NPSSM
0.560
5.969
0.152
0.323
0.293
4.191
Table A4: Comparison with baselines on multi-step (3-step) time series forecasting setting,
Economics
Bitcoin
Fred
Methods
MAE
Loss-Loss
MAE
Loss-Loss
MAE
Ro.9-Loss
LSTM
0.724
N/A
0.634
N/A
0.694
N/A
DeepAR
0.895
1,665
1.483
0.658
0.922
0.677
KVAE
0.612
1.365
0.551
0.299
0.534
0.795
VRNN 0.797
1.592
0.685
0.248
0.728
1.009
N-NPSSM
0.588
1.177
0.430
0.166
0.595
0.602
In Table A4, we compare N-NPSSM with the baselines multi-step (3-step) forecasting setting. We can find that our proposed model N-NPSSM consistently outperforms the baselines.
A3.3.2 ABLATION STUDIES
In table A5, show the performance of N-NPSSM and NPSSM synthetic datasets. We find that N-NPSSM achieves comparable performance with NPSSM on fixed causal dynamics settings, while N-NPSSM has a higher MCC score on vareing causal dynamics settings.
In table A6, we compare the identifiability results of NPSSM with/without transition prior network. We can find this module is critical to the identifiability of NPSSM.
A3.3.3
EMPIRICAL STUDIES FOR VIOLATION ASSUMPTIONS
NP Processes w/ Violations
1.00
0.75
MCC
0.50
0.25
Noninvertible Noise
0.00
OK
20K
40K
60K
80K
100K
Steps
Figure A1: MCC trajectories NPSSM for temporal data with non-inveritble noise.
To verify the effectiveness of NPSSM when noise function is non-inveritble, we consider the genera- tion process of replacing the stationary causal dynamics in Eq. (eq:heteo) with the squared noise, i.e.
Zk.t gu (1Zz-r)) F7 Ek,t.. We show the results of MCC trajectories of NPSSM in Figure
A1. We can find that NPSSM still achieve high MCC around 0.9.
18

---

Under review as a conference paper at ICLR 2023
Table A5: Comparison between N-NPSSM and NPSSM for MCC scores and forecasting performance on synthetic datasets
Synthetic
Synthetic 3
Method
MCC
MAE
Ro.s-loss
MCC
MAE
Ro.g-loss
NPSSM
0.984
0.073
0.288
0.933
0.061 0.057
0.384
N-NPSSM
0.987
0.054
0.220
0.998
0.363
Synthetic 2
Synthetic 4
Method
MCC
MAE
Ro.y-loss
MCC
MAE Ro.y-lss
NPSSM 0.996
0.080
2.178
0.946
0.095
0.196 0.169
N-NPSSM 0.995
0.069
1.866
0.992
0.081
Table A6: Ablation study for the effectiveness of the transition prior network
Method
Synthetic 1
Synthetic 2
Synthetic 3
Synthetic
NPSSM w/t transition prior NPSSM w/o transition prior
0.984
0.996
0.933
0.946
0.641
0.700
0.729
0.634
A3.3.4 MODEL COMPLEXITY
In Table A7, report the total number of parameters of different methods in our synthetic exper- iments. Compared to baseline models, the proposed NPSSM and N-NPSSM requires more extra parameters. This is because these two methods have extra transition prior models. Compared to NPSSM, N-NPSSM has more parameters since it needs to explicitly model the encoder for č1:7 conditioned on (2
A3.3.5 INTERPRETABILITY ANALYSIS
In Figure A2, we show the recovered causal relationships from NPSSM and KVAE in terms of MCC and causal-related factors in Synthetic3 dataset. Compared KVAE, we can find that NPSSM has better latent causal variables recovery, which is estimated up to permutation and component-wiss invertible transformation.
MCCwd
MCC+6.
Esiimated ed laterts
eddimatellaeeet laberts
True latents
True latents
(a) NPSSM
(b) KVAE
Figure A2: MCC for causally-related factors and scatter plots between estimated factors and true factors on two synthetic datasets for NPSSM.
Figure A3 present some showcases for different models in Economics dataset for qualitative evaluation. We observe that N-NPSSM can predict well under various temporal data characteristics.
To visualize nonlinear relations, we use LassoNet (Lemhadri et al., 2021) as a post-processe post-processing tool to remove weak edges and generate the sparse causal relation graph from the results the economics dataset. This method prunes input nodes by jointly feeding the first hidden layer and the residual layer through hierarchical threshold-based optimizer. We first fit the LassoNet the emission
19

---

Under review as a conference paper at ICLR 2023
Table A7: Model size (Total parameters) of different methods synthetic experiments.
LSTM
KVAE
VRNN
DeepAR
NPSSM
N-NPSSM
Model Size
72.3k
56.2k
46.3k
78.2k
117k
KWAE
KYAE TV NPSSS
1.5
TV karesan
Truth
oround Truth
Ieoonnd Touth
1.0
1.6
-1.0
1.0
0.0
2.2
0.25
Direpoon
Despont
Groound
1.6
-0.75
1.2
M
-0.50
-0.73
3.0
20+
90
20
4.0
50
1.0
20
2.0
1.0
Kunk
4.7
Groutal Truth
TV-NESSM
2.6
2.8
Ormuan Truth
DeepAR
0.5
2.4
-0.5
1.5
1.3
1.9
-.0
TV-NISSI
Figure A3: The observations each model on economics dataset
model, which receives latent variables and outputs the observation variables at the same time step. As shown in Figure A4, we can find that industrial production and business confidence survey are simultaneously correlated, both of them are affected by latent factor '1'. Additionally, foreign exchange reserves, CPI and money supply are simultaneously correlated, as all of them effected by latent factor '4'.
In Figure A5, we use LassoNet again to extract the sparse time-lagged causal relation in latent space. We can observe that most of the latent factors are affected by their time-lagged parents' nodes. Meanwhile, our model can also recover the cross relations between latent variables.
A4 IMPLEMENTATION DETAILS
A4.1
NETWORK ARCINTEECTURE
We summarize our network architecture in Table A8.
A4.2
TRAINING DETAILS
The models were implemented by PyTorch 1.9.0. The VAE network is trained using AdamW optimizer and early stops ELBO loss does not decrease. The maximum epoch is 200 for synthetic datasets and 700 for real-world datasets. A mini-batch size of 64 is used. We used three random seeds in each experiment and reported the mean performance with standard deviation averaged across random seeds.
20

---

Under review as a conference paper at ICLR 2023
Industr
reserves
Busnegg Co
erke survey
Comanee
rdet ICPO
Uamp enttroaa
nones!
omity
Figure A4: The causal relation between latent variables and observed variables. The blue circles with number indicate latent factors, while the green circles indicate the observed variables. Note that latent factors '0', '2' and '5' has been removed by the pruning step when constructing this relation graph. means these factors do demonstrate strong causal strengths.
The hyperparameters of N-NPSSM include [3,y,a] which represent the weights for transition prior for latent variable z, change factor z, and auxiliary predictor. Since the objective of transition prior does not consider the initial time-lagged variables, follow the conventional VAE and use the standard normal distribution (0, 1) as the prior distribution for these initial latent variables instead. Therefore, we augment the hyperparameters select these hyperparameters, which are Ir E [le 3, 5e 3, 2e 2]. 3 E [8 3, le 2, 2e 2) Benit E [5e 4, and σ E [0.1, 0.5. 1]. To facilitate comparison, the training parameters of baselines, e.g., optimizer, batch size, as well as the encoder and decoder architecture are identical to N-NPSSM. Similarly, we performed a grid search to select learning rate, Ir E [5e 15e-4 4, le 3, 5e leeee-2 5 leele-1. 1). and the hyperparameter of KL divergence term, a E [le - 4, 5e 4, 1e - 3, 5e 3, le - 2, 5e 2, 2, 1]. For all experiments, we E I and E 2 and set the maximum time lag L by the rule of thumb. For the initialization of VAE, we follow the instruction of 3-VAE (Higgins et al., 2016) and adopt the He initialization. For the rest of the modules/networks, adopt uniform initialization.
[8, Sinst Y, Tinit; o.. We performed grid search to
3). y E [le 4, 5e 3, le 2, 2e 2), Kimit E [3e 3, 5e 3,
2].
Training Stability We have used several standard tricks to improve training stability: (1) we use AdamW optimizer as a regularizer to prevent training from being interrupted by overflow or underflow of variance terms of VAE; (2) For the experiments learning procedure into two phases. We focus on the reconstruction task first and uncover the latent process, then we learn the latent predictor. This allows the model to first find the identifiable latent representations and then learn how to utilize them for the forecasting task. For the real-world datasets, we jointly learn these two components.
synthetic datasets, we separate the
Computation Hardware
We use Nvidia 100 GPU to run our experiments.
21

---

Under review as a conference paper at ICLR 2023
7
5.
0.t-1
0,1-1
0
0.L-3
2
6.t.2
4.t.2
6.t
4.t-3
161
0.t-1
4.1
5.L-2
7
3,52
0,1-
6.t.3
6,1-1
6,1-3
7.t
1.33
0.t
Figure A5: The time-lagged causal relations graph for latent variables. The blue circles indicate the lime-lagged source latent factors, while the green circles indicate the target latent factors.
22

---

Under review as a conference paper at ICLR 2023
Table A8: Architecture details. BS: batch size, length of time series, i_dim: input dimension, z_dim: latent dimension, c_dim: time-varying change factor dimension, LeakyReLU: Leaky Rectified Linear Unit.
Configuration
Description
Output
MLP-Encoder
Encoder Network
Input: X1:2
Observed time series
BS x T x x_dim
Dense Dense
128 neurons, LeakyReL.. ReL.
B$ 128
128 neurons, LeakyRela ReLU
BS x 128
Dense
128 neurons, LeakyReLU
BS X T 128
Dense
Temporal embeddings
BS z.dim
2. MLP-Decoder
Decoder Network
Input: 21.7
Sampled latent variables
BS x T 7_dim
Dense
128 neurons, LeakyReLU 128 neurons, L.akkyReL.U
BS 128
Dense
BS x x 128
Dense
idim neurons, reconstructed &1r
BS idim
3. Inference Network for 21.T
Inference Network
Input
Temporal embeddings
BS x T x z.dim
Bottleneck
Compute mean and variance of posterior
Pi.,, Rir 2
Reparameterization
Sampling
4. Inference Network for C1.7
Inference Network
Input
Temporal embeddings
BS T x c.dim Pirroff.
Bottleneck
Compute mean and variance of posterior Sampling
Reparameterization
C1.7
5. Transition Prior for Zi.T
Nonlinear Transition Prior Network
Input
Sampled latent variable sequence 21:r and č1:r Compute estimated residuals ést
BS x T z,dim BS T z.dim
InverseTrannee Transitiionn
JacobianCompute
Compute log (ldet (J)))
BS
6. Transition Prior for c1:2*
Nonlincar Transition Prior Network
Input
Sampled latent variable sequence ê1.r
BS c.dim BS c.dim
TuvesssTTannsttoon
Compute estimated residuals &o.
JacobianCompute
Compute log (|det (J)))
BS
7. Auxiliary Predictor
Prediction Network
Input
Sampled latent variable sequence ē1. Use past {2.-r} to predict à;
BS T z.dim BS x z.dim
LSTMInference
23


Reviews:
[{"review_id": "n1bLxyrvNC", "reviewer": "Reviewer_AS1z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a general formulation of SSMs, called the Non-Parametric State-Space Model (NPSSM). This framework is able to depict the non-stationarity of the latent process over time. Based on the identifiability of time-lagged latent variables, they proposed a new structural VAE for model estimation and forecasting tasks. They optimize the reconstruction of the observation and the prediction of the latent from the latent at the previous time. ", "summary_of_the_review": "The work is interesting, but the clarity of the methods needs work.", "strength_and_weaknesses": "1. It is not clear how the ‘time-varying change factors’ c_t enter into the transition equations. Specifically, they are not present in the latent predictor model, nor in the loss function (apart from the KL). It is unclear if they are being used at all, and if so, how.\n2. The metrics of comparison for the true and estimated latent variables can be much improved. The authors can learn an optimal permutation to compare the latent variables.\n3. The invertibility of the transmission plays an important role in the proposed model: the model seems to be under an assumption that the transition is invertible. Please discuss the situation where the transition and change transition functions are not invertible. This would seemingly lead to errors in the inference.\n\nMinor\n1. In Figure 4, the readers are not able to know the order of the latents in scatter plots, in particular, which one is the latent 2 in the MCC matrix?\n2. Some notations in equations need clarifying, e.g., ‘i’ and ‘j’ in Eq 2 need to be clearer, the difference between z_t and \\hat z_t in Eq 7 should be clarified, the notations in A.3.2, and so on.\n3. Grammar: page2, paragraph 2, line 3,  ‘, In particular’.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QLfhKW6807", "reviewer": "Reviewer_Uowm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a SSM adopting functional causal model for transition and post nonlinear model for emission, inferred in a VAE framework. The method was motivated to overcome several limitations of existing SSM formulations including the use of additive noises, and handling non-stationary time series. Experiments were conducted on synthetic and real-world datasets with a small number of selected baselines.", "summary_of_the_review": "This paper discusses two interesting ideas for improving deep SSMs. However, the lack of clarity, the insufficient consideration of comparison models, and the lack of ablation study made it difficult to assess the actual impact of these two ideas on improving SSM-based time-series modeling and forecasting. \n", "strength_and_weaknesses": "Strengths:\n\n\nThe use of functional causal modeling in SSM is interesting and promising. The issues of dealing with non-additive noises and non-stationary transition functions are also important.\n\n\n\nWeakness:\n\nThe writing of the papers is generally difficult to follow. It was not clear how some of the stated goals were achieved. \n\n- The paper emphasizes \"non-parametric\" models, referring presumably the use of casual function modeling for the transition process. How is this model effectually reflected in the estimated SSM is quite unclear. As described in page 5-6, the latent prediction model p_pred of z essentially is modeled as a LSTM, and the noise is generated from the inverse latent transition function that is modeled as MLPs. It is not clear where is the latent causal model in the final ELBO loss in Equation (7). Is it in p(z_1:T) and p(c_1:T)? If yes, what are their relations to the \"latent prediction model\"? Why do we need both the latent causal model (equation 2) and latent prediction models at the same time? All of these questions made it difficult to assess the contribution of the presented causal transition functions.\n\n- How is the 2nd term in the ELBO loss (7) supervised at training time? What's the significance of this term?\n\n- The baselines used are limited. As revised in the related works, there is a large set of SSMs with complicated modeling of q(z) and p(z)'s, such as those below. These should be included in the comparisons in order to truly understand the contribution of the presented work over state of the arts.\n\nKrishnan et al, Deep Kalman Filters, 2015\n\nKlushyn et al, Latent matters: Learning deep state-space models, NeurIPS 2021\n\n\n- Please elaborate on the definition and calculation of MCC in the main text. In the Appendix, it is stated that 'It first computes the absolute values of the Spearman’s rank\ncorrelation coefficients between every ground-truth factor against every estimated latent factor\". It is not clear how is this done in the temporal setting. What are ground-truth factors in the synthetic experiments and ho are the \"estimated latent factor\" determined? How are all the permutation done in a time series setting for all z_t's over time? How are these done for the baseline models?\n\n- Given that the method introduced two rather complicated components to state-of-the-art SSMs, an ablation study is needed to understand the importance of each of the components (causal model, and non-additive noises)\n\n\n- As the authors discuss identifiability, it seems that it mainly refers to the identifiability of the latent variables z_t's. Does the identifiability of z_t's directly translates to the identifiability of the underlying transition function? In traditional SSM, the identifiability of the system states and system parameters (that define the transition functions and others) are not necessarily the same?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "24nNiIzN0XW", "reviewer": "Reviewer_qCMD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed a new State Space Model (SSM) for forecasting. The proposed work relies on 1) non-parametric functional causal model (Pearl, 2009) for the transition process and 2) post-nonlinear model (Zhang & Hyvarinen, 2012) to capture nonlinear distortion effects in the emission model. The resulting model is named as NPSSM. They also extend it with time-varying change factors, which serves as input to the transition process, to model the non-stationary aspect (N-NPSSM).\n\nThe authors proved, under certain conditions, their proposed NPSSM is identifiable. The proof extends the theorem in Yao et al., 2022 to additive noise case with post non-linear function (their formulation of NPSSM). An estimation framework based on structural VAE is proposed and experiments on 4 synthetic datasets and 3 real-world datasets demonstrated superior performance over 4 deep-forecasting and deep state-space models.\n", "summary_of_the_review": "The paper has balanced contributions in theoretical, methodological and empirical aspects. It is a small step towards a more general and flexible SSM for forecasting tasks. But the theoretical and methodological contributions are extensions of previous works, limiting the significance. More importantly, the experiments section need to be improved so that the empirical results are more convincing. Please refer to Strength and Weakness section for more detail.\n\n=== After authors' response ===\nThe authors addressed my questions on the empirical side and I increased my score to 6.\n", "strength_and_weaknesses": "The strength of the paper is the balanced contributions in theoretical, methodological and empirical aspects. It is a small step towards a more general and flexible SSM for forecasting tasks.\n\nThe weakness of the paper mostly is the experiment. Even though it seems strong empirically, the setting is not very convincing for me. Below is a list of comments on possible way to improve experiments.\n* There are no results for NPSSM and include it may serve a good ablation study. \n* The author mentioned many deep learning and deep state space models in the related work but 4 are chosen including a naive LSTM. Including more baselines for sure could help.\n* Nowadays, 3 datasets are very limited from my perspective. \n* Looking one step ahead is fine but would be better if multi-step ahead is also considered. \n* If using one step ahead for evaluation, one simple baseline can be the last seen value.\n* Why R_{0.9} loss instead of considering the average of all the quantiles? \n* The hyperparameters and training detail are missing for the baselines.\n* Regarding to the MCC in Table 1: Is it fair to compare latent space with other algorithms when the data generation largely follows the proposed architecture?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject