Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement
Authors: Yan Li
Xinjiang Lu
Yaqing Wang
Dejing Dou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Time Series Forecasting
Generative Modeling
Diffusion Probabilistic Model

Abstract: Time series forecasting has been a widely explored task of great importance in many applications. However, it is common that real-world time series data are recorded in a short time period, which results in a big gap between the deep model and the limited and noisy time series. In this work, we propose to address the time series forecasting problem with generative modeling and propose a bidirectional variational auto-encoder (BVAE) equipped with diffusion, denoise, and disentanglement, namely D3VAE. Specifically, a coupled diffusion probabilistic model is proposed to augment the time series data without increasing the aleatoric uncertainty and implement a more tractable inference process with BVAE. To ensure the generated series move toward the true target, we further propose to adapt and integrate the multiscale denoising score matching into the diffusion process for time series forecasting. In addition, to enhance the interpretability and stability of the prediction, we treat the latent variable in a multivariate manner and disentangle them on top of minimizing total correlation. Extensive experiments on synthetic and real-world data show that D3VAE outperforms competitive algorithms with remarkable margins. Our implementation is available at https://github.com/PaddlePaddle/PaddleSpatial/tree/main/research/D3VAE.

try to model the Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement
Yan List", Xinjiang Lu & Yaqing Wang', Dejing Dout
Business Intelligence Lab, Baidu Research
$Zhejiang University, China
wangyaqing01, doude iuaaaddduuuuuuuuuuuu.....uu..... com Ly21121@zju.edi edu. en, (
Abstract
Time series forecasting has been a widely explored task of great importance many applications. However, it is common that real-world time series data recorded in a short time period, which results a big gap between the deep model and the limited and noisy time series. In this work, propose address the time series forecasting problem with generative modeling and propose a bidirectional variational auto-encoder (BVAE) equipped with diffusion, denoise, and disentan- glement, namely DAVAE Specifically, coupled diffusion probabilistic model is proposed to augment the time series data without increasing the aleatoric uncer- tainty and implement more tractable inference process with BVAE. To ensure the generated series move toward the true target, we further propose to adapt and integrate the multiscale denoising score matching into the diffusion process for time series forecasting. In addition, enhance the interpretability and stability of the prediction, we treat the latent variable in multivariate manner and disentangle them on top of minimizing total correlation. Extensive experiments on synthetic and real- real-world data show that VAE outperforms competitive algorithms with remarkable margins. Our implementation available at g com/ Baadduaaadd..aadalsssseeeeeeerrr
Introduction
Time series forecasting is of great importance for risk-averse and decision-makinn Traditional RNN- based methods capture temporal dependencies of the time series to predict the future. Long short-term memories (LSTMs) and gated recurrent units (GRUs) [6 15 40) introduce the gate functions into the cell structure to handle long-term dependencies effectively. The models based on convolutional neural networks (CNNs) capture complex inner patterns of the time series through convolutional 3 Recently, the Transformer-be -based models have shown great performance in time with the help of multi-head self-attention. However, one big issue of neural networks in time series forecasting is the uncertainty [44 I resulting from the properties of the deep structure. The models based vector autoregression (VAR) 5 10 23
operations 28
series forecasting [54]
56
25 29
distribution of time series from hidden states, which could provide more reliability to the prediction,
while the performance is not satisfactory (27
Interpretable representation Icarning is another merit of time series forecasting. Variational auto- encoders (VAEs) have shown only the superiority in modeling latent distributions of the data and reducing the gradient noise 36 24 80 45 but also the interpretability of time series forecasting [IT [2]. However, the interpretability of VAES might be inferior due the entangled latent variables.
"This work was done when the first author was an intern at Baidu Research under the supervision of the second author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

There have been efforts to learn representation disentangling (22 2 18], which show that the well- disentangled representation can improve the performance and robustness of the algorithm.
Moreover, real-world time series are often noisy and recorded in a short time period, which may result in overfitting and generalization issues [33 49
To this end, we address the time series forecasting problem with generative modeling. Specifically, we propose a bidirectional variational auto-encoder VAE) equipped with diffusion, denoise, and disentanglement, namely D More specifically, we first propose a coupled diffusion probabilistic model to remedy the limitation of time series data by augmenting the input time series, as well as the output time series, inspired by the forward process of the diffusion model 42 19 34 35], Besides, we adapt the Nouveau VAE [45] to the time series forecasting task and develop BAE diffusion model. In this way, the expressiveness of the diffusion model plus the tractability of the VAE can be leveraged together for generative time series forecasting. Though the merit of generalizability is helpful, the diffused samples might be corrupted, which results in a generative model moving toward the noisy target. Therefore, we further develop a scaled denoising acore-matching network for cleaning diffused target time series. In addition, we disentangle the latent variables of the time series by assuming that different disentangled dimensions the latent variables correspond to different temporal patterns (such as trend, seasonality, etc.). Our contributions can be summarized as follows:
57 41
substitute for the reverse process of the
We propose a coupled diffusion probabilistic model aiming to reduce the aleatoric uncertainty of the time series and improve the generalization capability of the generative model.
We integrate the multiscale denoising score matching into the coupled diffusion process to improve the accuracy of generated results.
We disentangle the latent variables of the generative model to improve the interpretability for time series forecasting.
Extensive experiments on synthetic and real-world datasets demonstrate that D& outper- forms competitive baselines with satisfactory margins.
2 Methodology
2.1 Generative Time Series Forecasting
Problem Formulation. Given an input multivariate time series X = {21, 02, In x4 E R+ and the corresponding target time series Y = {Un+1, Un+22 "* -nttm yj E Rd' (d < d). We assume that Y can be generated from latent variables Z E fz that can be drawn from the Gaussian distribution Z ~ X). The latent distribution can be further formulated as Po. (o(2 = go (X) where ge denotes nonlinear function. Then, the data density of the target series is given by:
/ ps (o()) -o(())dZ,
PY
Jaz
where fa denotes a parameterized function. The target time series can be obtained directly by sampling from (Y).
In our problem setting, time series forecasting is to learn the representation Z that captures useful signals of X, and map the low dimensional X to the latent space with high expressiveness. The framework overview of D VAE is demonstrated in Fig. Before diving into the detailed techniques, we first introduce preliminary proposition.
Proposition 1. Given a time series X and its inherent noise ex, we have the decomposition: X x= ex), where X, is the ideal time series data without noise. X, and ex are independent of each other. Let po X) )==--)))- = po( Z (x), (x) the estimated target series Y can be generated with the distribution po ( Po (Y, po po(ey)Z) where Y, is the ideal part of Y and Eq. is the estimation noise. Without loss of generality, Y, can be fully captured by the model. That is, ||Y,, Y, + where Y, is the ideal part of ground truth target series Y. In addition, Y can be decomposed as Y = =(Y,, Ey) (ey denotes the noise of Y). Therefore, the error between ground truth and prediction. i.e., | YIl ||ey - -Eyll 0, can be deemed as the combination of aleatoric uncertainty and epistemic uncertainty.
'The detailed literature review be found in Appendix A.

---

Coupled Diffusion Probabilittc Model
X9
yd
BAB
Encoder
Dilitsal
Recidual
Residuul
Recidual
DEL
x
Roidaal
Residual
Residaal
Couplel Dilfcaion Process
Less
Disentanglement
T
...
TC&
TC
Scure Matchinng
Denoising
I = DRL Lusse" + L(G.e)
Lyc
Optimization
clean
The framework overview of V&E. First, the input and output series are augmented simul- taneously with the coupled diffusion process. Then the diffused input series are fed into a proposed BVAE model for inference, which can be deemed reverse process. A denoising score-matching mechanism is applied to make the estimated target move toward the true target series. Meanwhile, the latent states in BVAE are leveraged for disentangling such that the model interpretability and reliability can be improved.
Figure I:
2.2 Coupled Diffusion Probabilistic Model
The diffusion probabilistic model (diffusion model for brevity) is a family of latent variable models aiming to generate high-quality samples. To equip the generative time series forecasting model with high expressiveness, coupled forward process is developed to augment the input series and target series synchronously. Besides, in the forecasting task, more tractable and accurate prediction is expected. To achieve this, we propose a bidirectional variational auto-encoder (B VAE) to take the place of the reverse process in the diffusion model. We present the technical details in the following two parts, respectively.
2.2.1 Coupled Diffusion Process
The forward diffusion process is fixed to Markov chain that gradually adds Gaussian noise the data 42 19.. To diffuse the input and output series, we propose coupled diffusion process, which is demonstrated Fig. 2 Specifically, given the input X ( x(x())), the approximate posterior q (x)) can obtained as
(1(() (0)
[X(0 x(x1)) =N(x0), v1 (1x (,II,, (2)
where uniformly increasing variance schedule {31, .r BB control the level of noise be added. Then, let a₃ - and α -I Os, have (0) -(x10) (x -o.
[0, 1)} is employed to
7( (x10)x
Furthermore, according to Proposition I we decompose X as (0) (Xr, Then, with Eq. ( the diffused (+) can be decomposed follows:
x (1) v (1 ūx)6x = VarXn, Varex + (1 ax)dx)
idcal
muisy part
where 6x denotes the standard Gaussian noise of X. As be determined when the variance schedule ( known, the ideal part is also determined in the diffusion process. Let X!!") V and 8 Vāsex + ā,)6x, then, according to Proposition and Eq. (4) we have
Po (2((0))(1)) pz() x( 800) p6
po (Y."" )pe (0 10011(1)) (5)
3

---

8y
6y
x(1 -B1
x(1 B22
x(1 B4)
xB
xB2
xB
Y())
p())
y(r-1)
Y
8x
8x
6x
x(1 - (.)
x(1 - B2)
x
BT)
(
XB
X
xB2
X
xBr
X
Figure 2: An illustration of the coupled diffusion process. The input X and the corresponding target (0) diffused simultaneously with different variance schedules. -, .8 the variance schedule for the input and 3 = [31, [ } is for the target.
where 8 denotes the generated noise of (t), To relieve the effect of alcatoric uncertainty resulting
from time series data, we further apply the diffusion process to the target series Y = y ~ (Y) In particular, scale parameter
(0, is adopted, such that 3 w ag 1 3 and
ae IT' a'. Then, according Proposition we can obtain the following decomposition
(similar to Eq. (4:::
y = (o + +(10++111100 (1-a1))) ā.)or = VayY + + d')o8) (1- (Y,3) (2,,04)
(6)
ideal part
noisy part
Consequently, we have q( (Y))) q(Y 19(5 Afterward, we can draw the following conclusions
with Proposition| I and Eqs. (5) and (6). The proof's can be found in Appendix B.
Lemma 1. Ve 0, there exists a probabilistic model f6.8 = (p4,Pe) to guarantee that DKL(g(Y, (Y(1)) (lp
E, where Y (x())
Lemma 2. With the coupled diffusion process, the difference between diffiusion noise and generation [2()) < DKL(g(ey) lppo (cp))
noise will be reduced, iee. lim4- DKL (
Therefore, the uncertainty raised by the generative model and the inherent data noise can be reduced through the coupled diffusion process. In addition, the diffusion process simultaneously augments the input series and the target series, which can improve the generalization capability for (esp. short) time series forecasting.
2.2.2 Bidirectional Variational Auto-Encodes
Traditionally, in the diffusion model, reverse process is adopted generate high-quality samples 19). However, for the generative time series forecasting problem, not only the expressiveness but also the supervision of the ground truths should considered. In this work, we employ a more efficient generative model, i.e., bidirectional variational auto-encoder (B (AV) 45). to take the place of the reverse process the diffusion model. The architecture of BVAE is described in Fig. where 7 is treated in multivariate fashion 7 {=1, 2, ) (zj E R 24+1 ~ p(2++1/ 12, X). Then, n is determined accordance with the number of residual blocks in the encoder, well as the decoder. Another merit of BVAE is that it opens interface to integrate the disentanglement for improving model interpretability (refer to Section 2.4
42
......... 2i,m) and
2.3 Scaled Denoising Score Matching for Diffused Time Series Cleaning
Although the time series data can be augmented with the aforementioned coupled diffusion proba- bilistic model, the generative distribution pv({((e) tends to move toward the diffused target series (()) which has been corrupted Denoising Score Matching (DSM) to accelerate the de-uncertainty process without sacrificing the
further "clean" the generated target series, we employ the

---

model flexibility. DSM 46 32 was proposed to link Denoising Auto-Encoder (DAE) 47 to Score Matching (SM) 20 Let Y denote the generated target series, then have the objective
LDsM(() = p.o (P.r) V log(so. (VI)) + E(E
where Paol (Y. Y) the joint density of pairs of corrupted and clean samples (Y,Y). Vp log( (V1)) is derivative of the log density of a single noise kernel, which dedicated replacing the Parzen density estimator: Pro (Y) f you (Y))) in score matching. and C)
() is the energy function. In the particular case of Gaussian noise, log(2e. (Y|)) =
- 18 - /20/22002 + C. Thus, we have
(C) E Pog (P,r) 1
- &
(8)
Then, for the diffused target series at step we can obtain
LDSM(C, t) = E
Pro (P(O, rr) || 4 + + p:o (()(())))))
(9)
To scale the noise of different levels [32]. monotonically decreasing series of fixed a values
fo OT o or = 1 - ā (refer to the aforementioned variance schedule 33 Section 2.2) is adopted. Therefore, the objective of the multi-scaled DSM is
L(S, = g, (o)))) 9 V Pi E(P())))
(10)
where a E (o or) and ((x+) oz. With Eq. 10)). we can ensure that the gradient has the right magnitude by setting do-
In the generative time series forecasting setting, the generated samples will be tested without applying the diffusion process. To further denoise the generated target series Y, we apply single-step gradient denoising jump (39
cleam ý - F(P
(11)
The generated results tend to possess larger distribution space than the true target, and the noisy term in Eq. (TT) approximates the noise between generated target series and the "cleaned" target series. Therefore, Vg C): can be treated as the estimated uncertainty of the prediction.
2.4 Disentangling Latent Variables for Interpretation
The interpretabilityy of the time series forecasting model is of great importance for many downstream tasks [7 Through disentangling the latent variables of the generative model, not only the interpretability but also the reliability of the prediction can be further enhanced [3I].
To disentangle the latent variables Z (.ı,... , } we attempt to minimize the Total Correlation (TC) 550 22], which is popular metric to measure dependencies among multiple random variables,
TC(2))
Po(z))
II po(sss)
(12)
where m denotes the number of factors of z; that need to be disentangled. Lower TC generally means better disentanglement if the latent variables preserve useful information. However, very low TC can still be obtained when the latent variables carry no meaningful signals. Through the bidirectional structure of BVAE, such issues can be tackled without too much effort. As shown in Fig. [ the signals are disseminated in both the encoder and decoder, such that rich semantics are aggregated into the latent variables. Furthermore, to alleviate the effect of potential irregular values, we average the total correlations of 21z0 then the loss w.r.t. the TC score of BVAE can be obtained:
1 n
Lrc
TC(x.)
(13)
n
1=1
5

---

Algorithm 1 Training Procedure.
1: repeat
2:
X ~ )(x:())) y(u) ?(Y(0))) 5x NíO, Ta), br N(0, Ta)
3: Randomly choose (1.................. . and with Eqs. 4 and
4:
x VaiX (0) (1- au)ox, y( v Va,Y
afjoy
5:
Generate the latent variable Z with BVAE, Z (2
6:
Sample F ( (Z) and calculate (x)((( (1p0 (P(1))
7:
Calculate DSM loss with Eq. (IO)
8:
Calculate total correlation of Z with Eq.
9: Construct the total loss with Eq.
10:
0, 0 + argmin(C)
11:
until Convergence
Algorithm Forecasting Procedure.
1: Input: X ~ ((X)
2: Sample Z ~ p. (Z(x)
3: Generate y ~ po (
4: Output: Yokms and the estimated uncertainty with Eq.
11
2.5 Training and Forecasting
Training Objective. To reduce the effect of uncertainty, the coupled diffusion equipped with the denoising network proposed without sacrificing generalizability. Then we disentangle the latent variables of the generative model by minimizing the TC of the latent variables. Finally, we reconstruct the loss with several trade-off parameters, and with Eqs. 100. and (13 we have
L = 1/ (xY [lpe (P)) + À L(S, + Lrc + Lmnse (P Y()),
(14)
where Luse calculates the mean square error (MSE) between (1))) and (t),, We minimize the above objective to learn the generative model accordingly.
Algorithms. Algorithm | displays the complete training procedure of VAE with the loss function Eq. (14). For inference, as described in Algorithm given the input series X, the target series can be generated directly from the distribution Pe which is conditioned on the latent states drawn from the distribution Ps-
Experiments
3.1 Experiment Settings
Datasets. We generate two synthetic datasets suggested by 9.
W/ W4-1 tanh(b w)-2) + sin(w, + +.((0,0557)
= [W1, W2, wN] F + 0.500.
where We E I22 and 0 < Wt,1; W1, < 1 (1 = 1,2,3). F E B2xk ~ 1[-1, 1]. k denotes the dimensionality and N is the number of time points, a, are two constants. We set 0.9, b 0.2, k = 20 to generate D1, and a = 0.5, b = 0.5, k = 40 for D2, and N = 800 for both D2 and D2.
Six real-world datasets with diverse spatiotemppral d dynamics Electricity W Wind (Wind Power) [ and ETTS (56] (ETTml and ETTh1). To highlight the uncertainty in short time series scenarios, for each dataset, slice subset from the starting point to make sure that each sliced dataset contains at most 1000 time points. Subsequently, we
selected, including Traffic 27
uci http:///ww....t.aeeee./////e//ta
https://www hww.::w.wc-jjna..gg. /eetet//ageeeeeeeeeer//////////
This dataset is published at http:////hhhh.oooooo//////////ad aadiieppatial//tttsse////laaa

---

Table Performance comparisons on synthetic data in terms of MSE and CRPS. The best results are boldfaced.
Model
D"VAE
NVAE
3-TCVAE
[-VE
DeepAR
TimeGrad
GP-copula
VAE
2.33
1.201
0.631
0.8544.00m
1.1
0.966=.102 102
1.2022.10s
0.912
8
0.585+.uz 0.905+.u11
0.658g.oo2
0.7454.036
0.758+.0as
0.698
0.773 ..03
0.7864.003
Di
0.571 025
1.184
0.758
1.046
270
0.911+.046
0.945
0.915 059
0.908
16
0.625
0.897
0.747
0.835
0.6994.014
0.709
0.7044..22
0.7655..07
0.599
1.9664.047
3.096
3.353 430
0.977
0.963
1.037
.o022
3.079
0.628 027
1.255
1.680 062
1.6-10
154
0.727
0.7062.125
0.753 026
1.5044.098
Da
16
0.786 1.01
1.955
3.067 443
3.
3.109
428
0.972
0.850 .011
.0611
1.082 .01
3. 3.132 1.0
0.728
1.251
1.643
1.558 0.720
0.762 o0s 1.560
Table The performance comparisons on real-world datasets in terms of MSE and CRPS, and the best results are in boldface.
Model
VAEAE
NVAE
3-TCVAE
[-VAE
DeepAR
TimeGrad
GP-copula
VAE
0.081 1.003
1.300
1.003 008
0.9824.059
3.8951.306
3.695
4.299
0.794
Traffic
8
0.207
0.593+.004
0.894 one
0.666+.aa2
1.3914.071
1.410
1.408
0.759
0.081+.000
1.271
0.997= 004
0.998+.042
4.141
3.495#.362
4.575+.141
0.6322.057
16
0.200+.014
0.5894.001
0.893_ o02
0.692+.026
1.3384.043
1.3294.057
1.5064.025
0.671
Electricity
0.251
1.134+.029
0.901
0.893
2.9344.173
2.7034.0s7
2.924
0.853
8
0.3984.011
0.542
0.831
004 0.809+.024
1.2444 037
1.208 .24
1.2494.044
048
0. 0.795=2.01
0.308
1.150+.032
0.850=
0.807+.034
2.803 199
2.770
3.065 186 0.8462.062
16
0.437
0.531 003
0.814 0.7821.024
1.2204.048
1.2401.044
1.307
0.793
0.169 4.02
0.801+.024
0.234=.uaz
0.591+.19s
2.317
2.715+.189
2.412+.783
0.560
Weather
8
0.24
0.757+.013
0.404_.040
0.565+
0.8584.078
0.9204.013
0.8974.115
0.572
0.187
1.077
0.811
0.212 012
0.530
1 1.2694.189
1.110
1.357 1.15
0.4244..141
16
0.361 oo00
0.759 .009
0.388 .14
0.547
0.783
0.733
.016
0.811
0.503
2.73
0.921+.026
1.538 254
2.326+.445
2.2044.420
1.877
2.024
2.375
ETTml
0.557
0.760+.026
1.015
112
1.260
0.9844.074
0.9084.03s
0.961 2.027
1.258
0.9684.104
1.100
1.744_
2.3394.270
2.350
2.0322..24
2.486 207
2.321 +.69
16
1.82
0.822+.026 026
1.104_.o41
1.2494.08s
0.9744.016
0.9194.03
0.9844.o1 016 1.2594.132
0.292
0.4831.017
0.703=.054
0.870
3.451
4.25911.15
4.278
1.006
ETThl
8
0.4244.u83
0.161
0.6411..zs 0.730+.o00
1.1944.034
1.092
1.169+.055
0. 0.76224.115
0.374 4.061
0.488+.010
0.681=.01s
0.983+.139
1.929
1.332+.125
1.701 4.08
0.6814.104
16
0.488 14888_oss .39
0.463
0.640=.oos
0.760+.om2
1.0294.030
0.879
0.999
0.641#.055
0.681
1.854+.037
1.321
1.942+.101
12.53+2.22
12.67+1.76
11.35+6.61
2.0064.145
8
Wind
0.5964.u02
1.223+.014
0.863
143
1.0674.086
1.370+.107
1.4404.059
1.305
1.103
1.033_ .022
1.955+.016
0.8944.035
1.262+.178
13.96
1.53
12.8642.60
13.79
1.1384.205
16
0.7571.053 1.2471.011
0.785 037
0.8431.066
1.3171.060
1.240+.070
1.261 1.11 0.862
obtained 5%,-raffic, 3% Weeethher, 2%-Wind, 1%-ETTml, and 5%-ETThl. The statistical descriptions of the real-world datasets can be found in Appendix C.I. All datasets are split chronologically and adopt the same rrain/validationtess ratios, i.c.. 7:1:2.
We compare VAE with one GP (Gaussian Process) based method (GP-copula B7). two auto-regressive methods (DeepAR (8) and TimeGrad (35)), and four VAE-based methods, i.e.,
Baselines.
vanilla VAE, NVAE [45), factor- VAE (f- VAE for short) 22 and 3-TCVAE (
Implementation Details. An inpet-l.--redicc-ll window is applied to roll the train, validation, and test sets with stride one time-step, respectively, and this setting is adopted for all datasets. Hereinafter, the last dimension of the multivariate time series is selected as the target variable by default.
7

---

We use the Adam optimizer with an initial learning rate of 5e 4. The batch size is 16, and the training set to 20 epochs most equipped with early stopping. The number of disentanglement factors is chosen from (4, 8). and 8r E /3 is set to range from 0.01 to 0.1 with different diffusion steps 7 E [100, 1000), then set to 0.1. The trade-off hyperparameters are set as 1 = 0.05, À = 0.1,y 0.001 for ETTs, and t/ 0.5, À = 1.0, 0.01 for others. All the experiments were carried out on a Linux machine with a single NVIDIA P40 GPU. The experiments are repeated five times, and the average and variance of the predictions are reported. We use the Continuous Ranked Probability Score (CRPS) 33 and Mean Squared Error (MSE) as the evaluation metrics. For both metrics, the lower, the better. In partieular, CRPS is used evaluate the similarity of two distributions and is equivalent to Mean Absolute Error (MAE) when two distributions are discrete.
3.2
Main Results
Two different prediction lengths, i.e., ly E 16] (12 = ly), are evaluated. The results of longer prediction lengths are available in Appendix D.
Toy Datasets. In Table 1 we can observe that D achieves SOTA performance most of the time, and achieves competitive CRPS in D2 for prediction length 16. Besides, VAEs outperform VARs and GP on D,, but VARs achieve better performance on D2, which demonstrates the advantage of VARs in learning complex temporal dependencies.
Real- World Datasets. As for the experiments on real-world data, V achieves consistent SOTA performance except for the prediction length 16 on the Wind dataset (Table 2) Particularly, under the input-8-predict-8 setting. D3 VAE can provide remarkable improvements in Traffic, Electricity, Wind, ETTm1, E and Weather w.r.t MSE reduction (90%, 71%, 48%, 43%, 40% and 28%). Regarding the CRPS reduction, V achieves 73% reduction in Traffic, 31% in Wind, and 27% in Electricity under the input-8 8- setting, and a 70% reduction in Traffic, 18% in Electricity, and 7% Weather under the input- 1-----6dde------- 16 setting. Overall, V gains averaged 43% MSE reduction and 23% CRPS reduction among the above settings. More results under longer prediction-length settings and on full datasets can found Appendix D.I.
Uncertainty Estimation. The uncertainty can be assessed by estimating the noise of the outcome series when doing the prediction (see Section 2.3)). Through scale parameter w, the generated distribution space can be adjusted accordingly (results on the effect of w can be found in Appendix D.3). The showcases in Fig. demonstrate the uncertainty estimation of the yielded series in the Traffic dataset, where the last six dimensions are treated as target variables. We can find that noise estimation can quantify the uncertainty effectively. For example, the estimated uncertainty grows rapidly when extreme values are encountered.
Sensor 862
Sensor 861
Sensor 860
Vallue
Vallu
-1.
-12-
-15
-1.5
Prediction Lengtth
Prediction Lengthh
Prediction Lengtth
Sensor 858
Volue
-25
-12
-1.
Fredett Twreimmaeieeii
-15
Prediction Length
Prediction Length
Prediction Lengthh
Figure 3: Uncertainty estimation of the prediction of the last six dimensions in the Traffic dataset and the colored envelope denotes the estimated uncertainty.

---

Table 3: Ablation study of the coupled diffusion probabilistic model w.r.t MSE and CSPR.
M1 -221
Traffic
Electricity
Dataset
16
32
16
32
DOVAE_ y
0.122 vos 0. 0.1264.01x 0.3
0.422 012
0.250= 0o8 0.2611..0 .17 0.480 032 0.551 012
DOVAE_ y DSM
0.096 0oe 0.0922..00....00 oos 0.331 023 0.502 079
Prediction Lengtt
0.217 0. 0.220+.013 0.450 021 0.584_.053
323
D'VAE_x
0. 003 0.117 0.35
0.4 0.420
0.256 .008 0.253 0113 0.481 1: 0.0 0.540
0. 0.123
0.118+.ons
0.365
025 0,439=
D'VAE-CDM
0.255 nor 0.252 .11 0.498 0088 0.561
0.123 003 0.119 4003 0.338 0.448
D'VAE-CDML ----CCDM-DSSS
0.255 + 003 0.253 + .uos 0.467 +..20 0.555
Prediction Langth
Figure 4: Comparisons of predic-
DOVAE
0.081 + 009 0.091 + .00 0.308 2. .300 0.410
tions with different By and varying
0.200 ..1 0.216 0.437 # .22 0.53440008
T on the Electricity dataset.
Disentanglement Evaluation. For time series forecasting, it is difficult to label disentangled factors by hand, thus we take different dimensions of Z as the factors be disentangled: 2; 12 3..m] (24 E 2). We build classifier to discriminate whether an instance 21,j belongs to class j such that the disentanglement quality can be assessed by evaluating the classification performance. Besides, we adopt the Mutual Information Gap (MIG) [6] as a metric to evaluate the disentanglemen more straightforwardly. Due to the space limit, the evaluation of disentanglement with different factors can be found in Appendix E.
3.3 Model Analysis
Ablation Study of the Coupled Diffusion and Denoising Network. To evaluate the effectiveness of the coupled diffusion model (CDM), we compare the full versioned D VAE with its three variants: i) D*VAE_ i.e. D without diffused Y, ii) D&VAE_ i.e. D without diffused X, and iii) DOVAE- i.e. D'VAE without any diffusion. Besides, the performance of DDVAE without denoising score matching (DSM) is also reported when the target series is not diffused, which are denoted as D -DMM and DDVAE--CMMDDSM The ablation study carried out on Traffic and Electricity datasets under input- 16 preddctedeteeee 16 and --ret---rred In Table 3 we can find that the diffusion process can effectively augment the input or the target. Moreover, when the target not diffused, the denoising network would deficient since the noise level of the target cannot be estimated by then.
Variance Schedule 33 and The Number of Diffusion Steps T. To reduce the effect of the uncertainty while preserving the informative temporal patterns, the extent of the diffusion should be configured properly. Too small a variance schedule or inadequate diffusion steps will lead meaningless diffusion process. Otherwise, the diffusion could be out of control [ Here we analyze the effect of the variance schedule 3 and the number of diffusion steps T. We set 31 and change the value of 3r in the range of [0.01, 0.1], and T ranges from 100 to 4000. As shown in Fig. 4 we can see that the prediction performance can be improved proper ,3 and 7 are employed.
4 Discussion
Sampling for Generative Time Series Forecasting.
The Langevin dynamics has been widely applied to the sampling of energy-based models (EBMs) 51
g 53
Y2 Y - Eo N
(15)
"An illustrative showcase be found in Appendix F.

---

where & E {0, ,, K denotes the number of sampling steps, and is constant. With K and being properly configured, high-quality samples can be generated. The Langevin dynamics has been successfully applied to applications in computer vision 26 52 and natural language processing 7
We employ single-step gradient denoising jump in this work generate the target series. The experiments that were carried out demonstrate the effectiveness of such single-step sampling, We conduct an extra empirical study investigate whether further performance improvement of time series forecasting, We showcase the prediction results under different sampling strategies in Fig. By omitting the additive noise Langevin dynamics, we employ the multi-step denoising for VAE to generate the target series and plot the generated results Fig. 5al Then, with the standard Langevin dynamics, we can implement generative procedure instead of denoising and compare the generated target series with different p (see Figs. 55 to [d]. We can observe that more sampling steps might not be helpful in improving prediction performance for generative time series forecasting (Fig. 5a). Besides, larger sampling steps would lead to high computational complexity. On the other hand, different configurations of Langevin dynamics (with varying p) cannot bring indispensable benefits for time series forecasting (Figs. 5b to 5d
worth taking more sampling steps for
0.005
F-
Longth
Prodiciio Length
Length
Prodiction Length
(a) Multi-step denoising.
(b) p 0.003.
(c) 0.005.
(d) P 0.007.
Figure 5: The prediction showcases in the Electricity dataset with different sampling strategies.
Limitations.
With the coupled diffusion probabilistic model, although the alcatoric uncertainty of the time series can be reduced, a new bias brought into the series to mimic the distribution of the input and target. However, as a common issue in VAEs that any introduced bias in the input will result in bias in the generated output 48), the diffusion steps and variance schedule need to be chosen cautiously, such that this model can be applied to different time series tasks smoothly. The proposed model is devised for general time series forecasting, should be used properly to avoid the potential negative societal impacts, such as illegal applications.
In time series predictive analysis. disentanglemmnt of the latent variables has been very important for interpreting the prediction to provide more reliance. Due to the lack of prior knowledge of the entangled factors generative time series forecasting, only unsupervised disentanglement learning can be done, which has been proven theoretically feasible for time series [3I]. Despite this, for boarder applications of disentanglement and better performance, it is still worth exploring how to
label the factors of time series in the future. Moreover, because of the uniqueness of time series data, it is also a promising direction explore more generative and sampling methods for the time series generation task.
5 Conclusion
In this work, we propose a generative model with the bidirectional VAE as the backbone. To further improve the generalizability, we devise coupled diffusion probabilistic model for time series forecasting. Then a scaled denoising network developed to guarantee the prediction accuracy. Afterward, the latent variables further disentangled for better model interpretability. Extensive experiments on synthetic data and real-world data validate that our proposed generative model achieves SOTA performance compared to existing competitive generative models.
Acknowledgement
We thank Longyuan Power Group Corp. Ltd. for supporting this work.
10

---

References
[1] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavanzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243 76:243-297, 2021.
[2] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives, IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8))1798- 1828, 2013.
[3] Mikolaj Binkowski, Gautier Marti, and Philippe Donnat. Autoregressive convolutional neural networks for asynchronous time series. In International Conference Machine Learning, pages 580-589. PMLR, 2018.
[4] Anastasia Borovykh, Sander Bohte, and Cornelis W Oosterlee. Conditional time series forecast- ing with convolutional neural networks. STAT, 1050:16, 2017.
[5] Li-Juan Cao and Francis Eng Hock Tay. Support vector machine with adaptive parameters in financial time series forecasting. IEEE Transactions on Neural Networks, (M6):1506-1518 2003.
[6] Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders, Advances in Neural Information Processing Systems, 31, 2018.
[7] Yuntian Deng, Anton Bakhtin, Myle Out, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy-based models for text generation. In International Conference on Learning Representa- lions, 2019.
[8] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances Neural Information Processing Systems, 32, 2019.
[9] Amirreza Farnoosh, Bahar Azari, and Sarah Ostadabbas. Deep switching auto-regressive factorization: Application to time series forecasting. arXiv preprint arXiv:2009.05135, 2020.
[10] Konstantinos Fokianos, Anders Rahbek, and Dag Tjøstheim. Poisson autoregression. Journal of the American Statistical Association, 104(488): (00(488):1430-111 1430-1439, 2009.
[11] Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. GP-VAE: Deep probabilistic time series imputation. In International Conference on Artificial Intelligence and Statistics, pages 1651-1661. PMLR, 2020.
[12] Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. COM-VAE: Interpretable discrete representation learning on time series. International Conference Learning Representations, 2019.
[13] John Cristian Borges Gamboa. Deep learning for time-series analysis. arXiv preprint 17110:1701.01887 2017.
[14] Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, al. A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.03342. 2021.
[15] Felix A Gers, Nicol N Schraudolph, and Jürgen Schmidhuber. Learning precise timing with LSTM recurrent networks. Journal of Machine Learning Research, 31 2002.
[16] Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. LSTM: A search space odyssey, IEEE Transactions on Neural Networks and Learning Systems, 28(10):2222-2233 2016.
[17] Michaela Hardt, Alvin Rajkomar, Gerardo Flores, Andrew Dai, Michael Howell, Greg Corrado, Claire Cui, and Moritz Hardt. Explaining an increase in predicted risk for clinical alerts. In ACM Conference Health, Inference, and Learning, pages 80-89, 2020.
11

---

[18] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. VAt: Learning basic visual concepts with constrained variational framework. 2016.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-688 6851, 2020.
[20] Aapo Hyvärinen, Jarmo Hurri, and Patrik O Hoyer. Estimation of non-normalized statistical models. In Natural Image Statistics, pages 419-426. Springer, 2009.
[21] Aya Abdelsalam Ismail, Mohamed Gunady, Hector Corrada Bravo, and Soheil Feizi. Bench- marking deep learning interpretability in time series predictions. Advances in Neural Information Processing Systems, 33:6441 16452, 2020.
[22] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, pages 2649-2633 PMLR, 2018.
[23] Kyoung-jae Kim. Financial time series forecasting using support vector machines. Neurocom- puting, 55(1-2):307331, 2003.
[24] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in Neural Information Processing Systems, 29, 2016.
[25] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001 04451,, 2020.
[26] Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv: 085508 2019.
[27] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 95-104, 2018.
[28] Colin Lea, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks: A unified approach action segmentation. In European Conference on Computer Vision, pages 47-54. Springer, 2016.
[29] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, Advances in Neural Information Processing Systems, 32, 2019.
[30] Yingzhen Li and Richard E Turner. Rényi divergence variational inference. Advances in Neural Information Processing Systems, 29, 2016.
[31] Yuening Li, Zhengzhang Chen, Daochen Zha, Mengnan Du, Denghui Zhang, Haifeng Chen, and Xia Hu. Learning disentangled representations for time series. arXiv preprint 175i::2105.08179, 2021.
[32] Zengyi Li, Yubei Chen, and Friedrich T Sommer. Learning energy-based models in high-dimensional spaces with multi-scale denoising score matching. arXiv preprint 11X.v:1910007762. 2019.
[33] James E Matheson and Robert L Winkler. Scoring rules for continuous probability distributions. Management Science, 22(10):0877-106, 1976.
[34] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[35] Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denois- ing diffusion models for multivariate probabilistic time series forecasting. In International Conference Machine Learning, pages 8857-8868. PMLR, 2021.
12

---

[36] Geoffrey Roeder, Yuhuai Wu, and David K Duvenaud. Sticking the landing: Simple, lower- variance gradient estimators for variational inference. Advances Neural Information Process- ing Systems, 30, 2017.
[37] David Salinas, Michael Bohlke-Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank Gaussian copula processes. Advances Neural Information Processing Systems, 32, 2019.
[38] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181-1199 2020.
[39] Saced Saremi, Aapo Hyvärinen, et al. Neural empirical Bayes. Journal of Machine Learning Research, 2019.
[40] Alex Sherstinsky. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Physica D: Nonlinear Phenomena, 404: 132306, 2020.
[41] Qiquan Shi, Jiaming Yin, Jiajun Cai, Andrzej Cichocki, Tatsuya Yokota, Lei Chen, Mingxuan Yuan, and Jia Zeng. Block hankel tensor ARIMA for multiple short time series forecasting, In AAAI Conference on Artificial Intelligence, volume 34, pages 5758-5766, 2020.
[42] Jascha vised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper-
PMLR, 2015.
[43] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[44] Sana Tonekaboni, Shalmali Joshi, Kieran Camphell, David K Duvenaud, and Anna Goldenberg. What went wrong and when? instance-wis feature importance for time-series black-box models. Advances in Neural Information Processing Systems, 33:799-809, 2020.
[45] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33:19667-19679, 2020.
[46] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7): 1661-1674,,11 2011.
[47] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of Machine Learning Research, 11(12), 2010.
[48] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in Neural Information Processing Systems, 34, 2021.
[49] Juntao Wang, Wun Kwan Yam, Kin Long Fong, Siew Ann Cheong, and KY Wong. Gaussian process kemnels for noisy time series: Application to housing price prediction. In International Conference on Neural Information Processing, pages 78-89. Springer, 2018.
[50] Satosi Watanabe. Information theoretical analysis of multivariate correlation. Ibm Journal of Research and Development, 4(1 (1):66-822 1960.
[51] Jianwen Xie, Yang Lu, Song- Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pages 2635-2644, PMLR, 2016.
[52] Jianwen Xie, Yifei Xu, Zilong Zheng, Song-Chun Zhu, and Ying Nian Wu. Generative pointnet: Deep energy- based learning unordered point sets for 3d generation, reconstruction and classification. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 14976-14985, 2021.
13

---

[53] Jianwen Xie, Song-Chun Chun Zhu, and Ying Nian Wu. Learning energy avergy-based based spatial- temporal generative convnets for dynamic patterss, IEEE transactions on pattern analysis and machine intelligence, 43(2):516-331, 2019.
[54] Jiehui Xu, Jianmin Wang, Mingsheng Long, et al. Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 2021.
[55] Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. A review of recurrent neural networks: LSTM cells and network architectures. Neural Computation, 31(7):125--1270, 2019.
[56] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In AAAI Conference on Artificial Intelligence. 2021.
[57] Yong Zou, Reik Donner, Norbert Marwan, Jonathan Donges, and Jürgen Kurths. Complex network approaches to nonlinear time series analysis. Physics Reports, 787:1-97, 2019.
14


Reviews:
[{"review_id": "f4im7ay7lpC", "reviewer": "Reviewer_GcMT", "summary": "In this submission, the authors propose to use coupled diffusion model with denoising mechanism to augment the time series. The forecasting can be performed directly with by the generative model. In particular, the diffusion model aims to deal to aleatoric uncertainty. Also, the authors propose to perform latent variable disentangling to enhance interpretation. ", "questions": "1. What is the actual relationship between the aleatoric uncertainty and the diffusion model?\n2. What is the advance of the diffusion model in this work wrt the application to time series?\n2. What is the connection between the diffusion model and disentanglement?", "limitations": "The authors have mentioned several limitations in the main text. I agree with the authors that the process is sensitive to hyperparameters such as beta and time steps. One missing part is the computation performance, which is a common concern when using diffusion model.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality\nThis method is built on several original insights, especially on the connection between noise and diffusion model. On the other hand, the model components each as BVAE and denoising disentangling are extended from previous works. \n\nQuality:\n1.\tThe conductance of this study is of high quality in general. The experiments are solid with several ablations.\n2.\tHowever, one missing part in the discussion of mythological advance (not difference) with previous generative or diffusion model. For example, TimeGrad also relies on the diffusion model. Why this method outperforms a lot in the experiments.\n3.\tThe overall presentation is satisfactory. However, it would be more reader-friendly for the audience to follow if there is a global picture in the main text. Also, several examples can be moved from the appendix to the main text.\n\nClarity：\n1.\tThe motivation of the method is clearly addressed. The author emphasizes the noise challenges in short time forecasting setting.\n2.\tAs the uncertainty itself is analyzed by separating aleatoric and epistemic uncertainty, the authors state that their proposed method can reduce the former. However, the reviewer finds it hard to follow the derivation of this conclusion (i.e., Lemma 1-2). It seems no bounds of this uncertainty are given.\n3.\tThe connection between the diffusion model and disentanglement is missing? Why are these two components combined in one work?\n\nSignificance\nThis paper tackles an important issue and highlights a feasible augmentation strategy for noisy input.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8FZHMqkHdKj", "reviewer": "Reviewer_4VLY", "summary": "The author proposed D3VAE, a coupled diffusion probabilistic model for time series forecasting problems. A diffusion process is introduced on top of the BVAE (Bidirectional Variational Auto-Encoder) to mitigate the effect of the random noise. Total correlation is used to disentangle the different latent variables Z to better extract dependencies from the input data hence improving the overall performance. Last, a denoising score matching (DSM) module is used to improve the accuracy of the generated results. Experimental results show that D3VAE perform better than other baseline models in most of the synthetic and real-world datasets.", "questions": "1. Which evidence can prove the learned latent variables Z are actually disentangled?\n2. Which part of the results shows that the interpretability is improved compared to other baseline methods?\n3. What is the actual architecture of D3VAE?", "limitations": "Improvement:\n1. Add experiments for evaluating the disentanglement of the learned representations.\n2. Add graphs to illustrate the increase in interpretability.\n3. Refine the writing.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The design of this model is quite ingenious. The introduction of the DRL (disentangled representation learning) could cause noises in the reconstruction; therefore, a denoising module is used for better prediction accuracy. \n2. An extensive amount of experimental results are provided regarding the effectiveness of each module over a large scale of synthetic and real-world datasets. \n3. Experimental results show that D3VAE perform better than other baseline models in most of the synthetic and real-world dataset.\n\n\nCons:\n1. There are not enough results for the disentanglement section. The author claim that using the TC can force the model to disentangle the latent variables {z1, z2, ..., zn}. However, there is not enough evaluation or results to prove that the learned latent variables are actually disentangled. The supplementary document only includes a graph that shows the difference between the ground truth and the prediction with different numbers of z used. That does not prove the claim that the author makes. The actual level of disentanglement for each zi or the single variables in zi are not investigated properly.\n2. The overall structure and procedures of the proposed D3VAE are vague since there is only an illustration of the architecture of BVAE. It is hard for me to picture each module and its locations in the whole pipeline.\n3. The claim of interpretability is not addressed in the paper. In section 2.4, the author states that the introduction of disentanglement can lead to better interpretability. However, there is not enough evidence regarding this topic. Although there are case studies in the supplementary document which might shed some insights, the relationship between those insights and the usage of disentangled representation learning is missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3vbWT8-KmKT", "reviewer": "Reviewer_ee78", "summary": "The authors present a time-series forecasting model that can handle short examples. This is done using a generative model based on a BVAE. The experiments prove an excellent performance relative to SOTA techniques.", "questions": "1. How would the performance of the system change if the time series were shortened?\n2. Why is the estimated noise discontinuous, when the ground truth is not.\n", "limitations": "I think this work is very useful for all those using time series forecasting for running their businesses. The beneficial impact that this technique could have on, for example, optimizing inventory levels, is very important.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The presented method is developed using powerful theoretical ideas. This makes it easy to justify and understand the underlying reasons that explain the design.\n2. The experiments show an important improvement relative to the specified benchmark.  \nWeaknesses:\n1. Fig. 2 shows important discontinuities in the estimated noises. These sudden breaks of continuity are not reflected or explained by the ground truth. Hence, there is a high probability that they are caused by numerical instabilities in the numerical algorithms.\n2. There is no hyperparameter search, which strongly hints that there might be versions of the same pipeline capable of reaching even better performances.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept