Learning Latent Seasonal-Trend Representations for Time Series Forecasting
Authors: Zhiyuan Wang
Xovee Xu
Weifeng Zhang
Goce Trajcevski
Ting Zhong
Fan Zhou
Venue: NeurIPS 2022 Accept (2022)

Abstract: Forecasting complex time series is ubiquitous and vital in a range of applications but challenging. Recent advances endeavor to achieve progress by incorporating various deep learning techniques (e.g., RNN and Transformer) into sequential models. However, clear patterns are still hard to extract since time series are often composed of several intricately entangled components. Motivated by the success of disentangled variational autoencoder in computer vision and classical time series decomposition, we plan to infer a couple of representations that depict seasonal and trend components of time series. To achieve this goal, we propose LaST, which, based on variational inference, aims to disentangle the seasonal-trend representations in the latent space. Furthermore, LaST supervises and disassociates representations from the perspectives of themselves and input reconstruction, and introduces a series of auxiliary objectives. Extensive experiments prove that LaST achieves state-of-the-art performance on time series forecasting task against the most advanced representation learning and end-to-end forecasting models. For reproducibility, our implementation is publicly available on Github.

Learning Latent Seasonal-Trend Representations for Time Series Forecasting
Zhiyuan Wang', Xovee Xu', Weifeng Zhang', Goce Trajcevski', Ting Zhong', Fan Zhou' !+
University of Electronic Science and Technology of China
2 Iowa State University
zhy. wangcs@gmail com, xovee@live.c com, weifzh@outlook. com, gocet25Giastate. edu, {tingzhong, fan.
edu. cn
Abstract
Forecasting complex time series is ubiquitous and vital in range of applications but challenging. Recent advances endeavor to achieve progress by incorporating various deep learning techniques (c.g., RNN and Transformer) into sequential models. However, clear patterns are still hard to extract since time series are often composed of several intricately entangled components. Motivated by the success of disentangled variational autoencoder in computer vision and classical time series decomposition, we plan to infer couple of representations that depict seasonal and trend components of time series. To achieve this goal, we propose LaST, which, based on variational inference, aims disentangle the seasonal-trend representations in the latent space. Furthermore, LaST supervises and disassociates representations from the perspectives of themselves and input reconstruction, and introduces a series of auxiliary objectives. Extensive experiments prove that LaST achieves state-of-the-art performance on time series forecasting task against the most advanced representation learning and end-to-end forecasting models. For reprodueibility, our implementation is publicly available on Githut
Introduction
Time series forecasting plays a significant role plethora of modern applications, ranging from climate analysis [1], energy production (2), traffic flows [3] to financial markets and various industrial systems 4. The ubiquity and importance of time series data have recently attracted researcher efforts resulting in myriad of deep learning forecasting models 5 6 ameliorating the time series forecasting. Based on advanced techniques such as RNN and Transformer 77),, these methods usually learn latent representations to epitomize every instant of the signals, and then derive forecasting results by predictor, achieving great progress on forecasting tasks.
However, these models have difficulties to extract exact/clear information related to temporal patterns (e.g., seasonality, trend, and level), especially in supervised end-to-end architecture without any constraint representations [ As consequence, efforts have been made to apply the variational inference into time series modeling with probabilistic form has been proved beneficial to downstream time series tasks [12). However, when various intricately co-evolving constituents exist time series data, analyzing with a single representation will result in superficial variable and models' non-reusability and lack of interpretabil- ity, due to the highly entangled nature of neural networks [13 14 Thus, while providing efficiency and effectiveness, existing approaches with a single high-dimensional representation sacrifice the
II] where improved guidance for latent representations
"orresponnding author:
http:::///iihuuu.co//hhhhh LLST
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

information utilization and explainability, which may further lead to overfitting and degenerated performance.
To address the above limitations and seek a new disentangled time series learning framework, we leverage the ideas from the decomposition strategy [1] 16] to split time series data into several components, each of which captures an underlying pattern category. The decomposition assists the analysis process and reveals underlying insights more consistently with human intuition. This insight motivates us to produce couple of latent representations that respond to different time series characteristics (in our case the seasonal and trend), from which we predict the results by formulating sequence as the sum of these characteristics. The representations should be as independent as possible avoid a model prone to feature entanglement, while also having sufficient information to the input sequence.
Towards that, we propose novel framework LaST to learn the Latent Seasonal- -rend representations for time series forecasting. LaST exploits an encoder-decode architecture and follows variational inference theory [17] to learn a couple of disentangled latent representations that describe seasonal and trend of time series. To achieve this goal, LaST enforces the representations into disentanglement dissect intrinsic seasonal-trend patterns
under two constraints: (I) from the input reconstruction, which can be readily obtained from raw time series and off-the-shelf measurement methods, and accordingly design a series of auxiliary objectives: (2) from representations themselves, we minimize the mutual information (MI) between seasonal and trend representations on the premise that grantee the consistency between input data and each of them. Our main contributions are threefold:
We start with the variational inference and information theory to design the seasonal-tres -rend representations learning and disentanglement mechanisms, and practically demonstrate their effectiveness and superiority (over the existing baselines) on time series forecasting task.
We propose LaST, a novel latent seasonal- -eeeendd--endd representations learning framework. which encodes input as disentangled seasonal -tennd representations and provides a practicable approach that reconstructs seasonal and trend separately to avoid chaos.
We introduce MI terms penalty and present a novel tractable lower bound and an upper bound for their optimizations. The lower bound ameliorates the biased gradient issue in prevalent MINE approach and ensures informative representations. The upper bound provides the feasibility to further reduce the overlapping of seasonal-trent representations.
2 Related work
Most of the deep learning methods for time series forecasting are designed as an end-to-end archi- tecture. Various basic techniques (e.g., residual structure [1]] 19, autoregressive network [20] 21), and convolutions [2 (23) are exploited to produce expressive non-linear hidden states and embed- dings that reflect the temporal dependencies and patterns. There is also body of works that apply Transformer 77] structure into time series forecasting tasks [24 25 26 6 aiming to discover the relationships across the sequence and focus on the important time points. Deep learning methods have achieved superior performance in comparison to the classical algorithms such as ARIMA 27 and VAR [28] and have become prevalent in multiple applications.
Learning flexible representations has been demonstrated be beneficial for downstream tasks by numerous researches [12 29 In time series representations domain, early methods, employing the variational inference, jointly train an encoder and corresponding decoder that reconstructs raw signals to learn approximate latent representations [T0] 30 31 Recent efforts have improve these variational methods 32 33] by establishing more complex and ficxible distributions using techniques such copula (32) and normalizing flow [34 (35. Another group of works exploited the burgeoning contrastive learning to obtain invariant representations from augmented time series [36, 37 (38]
which avoids the reconstruction process and improves representations without additional supervision.
Time series decomposition [5 is classical method that splits complex time series into several components to obtain temporal patterns and interpretability. Recent works have applied machine learning and deep learning approaches [39 40 41 to robustly and efficiently achieve the decomposi- tion on large-scale datasets. There are also research results that tackle forecasting with the assistance of decomposition. For example, Autoformer [26] decomposes time series into seasonal-trend parts by average pooling and introduces autocorrelation mechanism to empower Transformer [7] for better

---

relations discovery. CoST 38] encodes signals into seasonal and trend representations in frequeney and temporal domains, respectively, and introduces contrastive learning supervise their learning. Different from our work, these methods exploit simple average pooling decomposition mechanism which may provide incompatible periodical assumptions, or intuitively disentangle the representations by processing in different domains. Meanwhile, LaST' adaptively epitomizes the seasonality and trend by disentangled representations and boosts their disassociation from a probabilistic perspective in the latent space.
Latent seasonal-trend representations learning framework
We now formalize the problem definition and introduce the proposed LaST framework. We note that LaST we use seasonal and trend characteristics for disentanglement learning but our framework can be easily extended to adapt to situations that have more than two components to dissociate.
Problem definition.
Consider a time series dataset consisting of i.id. sequences denoted (9) TTg's j. where i E {1, 21...... N). and each (t) is univariate or multivariate value representing the current observation(s) time instant / (e.g., price and rainfall).
as x
1:7 = {51
We aim to derive model that outputs expressive representations Zi.,, suitable for predicting future sequence(s) Y X Hereafter, when there is no ambiguity we omit the superscripts and the subscript 1:7' model that infers the likelihood between observation and future Y with latent representation Z can be formulated as follows:
P(X,Y) P(Y()))() / P(YIZ)((ZIIYIL / P(XIZ)P(Z)dZ.
(1)
From the perspective of variational inference (cf. (17). the likelihood P(X|Z) is calculated by posterior distribution (Z()) and maximized by the following evidence lower bound (ELBO):
log Pe(X,Y) log / Po (C((11))0((11))) + R (X(|2))
-KL(Q(Z(X))I(((2) LELBO,
(2)
where è is composed of vb o. and denotes learned parameters.
Reconstruction Estimation
MI Estimation
Autooorrelttin Distance
Z
2
Trend x
Seasonal Encoder Q2.
Seasonal Decoder
8
DFT
Input Time Series X
www0m
1(X,2)f 1
117
244
Z34
iDFT
FFNI
%
Trend Encoder Q
Trend Decoder Po
X
2
FFN2
FFN
Scasonal x
CORT Coetficient
(a) Seasonal-trend representations learning
(b) Predictor
Figure Overview of proposed LaST framework.
However, as pointed out in Sec. | this faces an entanglement problem and cannot clearly extract complicated temporal patterns. To ameliorate this limitation, we incorporate the decomposition strategy into our LaST that learns couple of disentangled representations to depict seasonal and trend dynamics. Specifically, we formulate the temporal signals X and as the sum of seasonal and trend components, i.e., X = X X'. Accordingly, the latent representation Z is factorized into Z and Z assumed to independent illustrates the two parts of the LaST framework: (a) representations learning, producing disentangles seasonal-trend representations (separate reconstructions and MI constraints); and (b) prediction based on learned representations.
each other i.e. P(Z) P(Z)) Figure

---

Theorem 1. With the decomposition strategy, Eq. (i.e., the ELBO) naturally has the following factorized
LELBO log ( . Pu(Y|Z",Z')Q4
Z'I' dZ'
(predictor) (3)
3 22
+ Ee (Z flog Pe. [2)] + Re (2 Dog Po. (X*12") (reconstruction)
(4)
KL(Q0 (Z (2 ( (2*)) KL(Q0= (Z ('))))(()))
(KL divergence)
(5)
The detailed inference process of the above formula is provided in Appendix 1. The ELBO is split into three main units, i.e., Eqs. 4 and (5) The predictor makes forecasting and measures the accuracy (e.g.. .. LI or L2 losses), reconstruction and KL divergence are served as regularization terms aiming improve the learned representations. The three units are described in the following.
The predictor (cf. Eq. (3) and Figure Ib)) can be regarded as the sum of two indepen- dent parts: log Iz. Py. (Y*)Z=*)) (Z dZ and log J2. P& (Y* Z²)Q (Z [X)''.. Here introduce two specialized approaches to harness the seasonal-trend representations combining their own characteristics. Given the seasonal latent representation Zd Txx the seasonal predictor first employs the discrete Fourier transform (DFT) algorithm to detect the seasonal frequencies, i.e., 7 (2*) E CFxd where F -+ due the Nyquist theorem (42.. Then, inverse the frequencies back to the temporal domain to extend the representation to the future part, i.e.,
Predictor:
2 iDFT (+)) E RF More details of the DFT and iDFT functions can be found in Appendix B. Given Z trend predictor provides feed forward network (FFN) ʃ T - T to produce predictable representation ž' RTxd We end the predictor with two FFNs map 7* and 7¹ into Y and Y respectively, and obtain the forecasting result Y by their sum.
Reconstruction and KL divergence: Among these two terms, the KL divergence estimated by Monte Carlo sampling with prior assumptions. Here we take a widely used setting that priors both follow (0, I) for efficiency, more discussions of our priors can be found in Appendix C. As for reconstruction term, cannot be directly measured owing to the unknown X and X'. Besides, merging these two terms into Eq,, a= (Z*, z x) loo Po-.ce (x Zt] will result in chaos since the decoder is prone to reconstruct the intricate time series from every representation. Theorem 2.
be easily
With the Gaussian distribution assumption, the reconstruction loss Cree can be estimated without leveraging X* and X', and Eq. (4) be replaced with the following formula:
T-1
Lree Σ
[|)xx(x)= (x)l + CORT(X, 82) -|| -||x'+
X + x||
(6)
CORT(X, 81) E AXAAR
VE EE
X) is the autocorrelation coefficient with lagged value (we employ an efficient implementation in frequency domain (43), details are in Appendix B.2), CORT(X, X*) is the temporal correlation coefficient, and AX; X; X;-1 is the first difference.
where -X,
The proof is provided in Appendix A.2. According to Eq. 6). the reconstruction loss now can be estimated and, conversely, used to supervise disentangled representation learning. However, we find that this framework still holds certain drawbacks: (1) The KL divergence tends to narrow down the distance between posterior and prior. The modeling choice tends sacrifice the variational inference vs. data fit when modeling capacity is not sufficient to achieve both [44). The posterior may become almost non informative for the inputs, which causes the forecastings irrelevant to the observations. (2) The disentanglement of the seasonal-tremt representations is boosted indirectly by the separate reconstruction, where need impose direct constraint the representations themselves. We alleviate these limitations by introducing additional mutual information regularization terms. Specifically, we increase the mutual information between Za, Zt and X to alleviate the divergence narrowing problem [44] 45), while decreasing mutual information between 2 and Zt to further dissociate their representations. The maximizing objective of LaST becomes
(8) where -(.,)) denotes the mutual information between two representations. However, the two mutual information terms are untraceable [46| 47 48). We address this problem in the next section.
LLaST = LELBO + I(X, 2*) T(X, T(Z*.2")

---

4 Mutual information bounds for optimization
We now address the traceable mutual information bounds, maximizing I(X, 2*) and T(X, 2"), and minimizing I [Z*, Z²) in Eq. (8) and provide lower and upper bounds for the model optimization.
Lower bound for T(X, 2*) or [(X, Z²). We omit the superscript & or / when analyzing lower bound. Among the prior approaches exploring the lower bounds for MI 49 50 [5], MINE (5T), for example, employs KL divergence between the joint distribution and marginals and defines an energy- based variational family to achieve flexible and scalable lower bound. This can be formulated
as [XX, Z) E (2) (2) (X, Z)] log (Q()))((s)) [X.2( IIINEE where Yoo is a learned normalized critic with parameters a. However, this bound suffers from the biased gradient owing to the parametric logarithmic term (see Appendix A.3 for proof). Inspired by (47), we substitute the
logarithmic function by its tangent family to ameliorate the above biased bound:
IMIEE (x,(xx22
EQ(=) (1 log 1)
7 - (e))
2 2 (X,
where 7) denotes the different tangent points. The first inequality relies on the concave negative logarithmic function - the values on the curve are upper bounds for that on the tangent line, and is tight when the tangent point overlaps the independent variable, i.e., the true value of [ Le 22)]. The closer the distance between tangent point and independent variable, the greater the lower bound. Therefore, we set v) as the variational term (=)=)) (X) that estimates the independent variable to obtain as great lower bound as possible. In the second inequality, Yra (I. =) a critic function activated by Sigmoid is limited within [0, 1] and thus - -log 7 - 1) Z 0. This inequality is tight only if [Q(=)Q,(=)) [Tx 2)] = 1, which means Ta can discriminate whether a pair of variables (X, Z) is sampled from the joint distribution or marginals. Similarly to MINE, this consistency problem can be addressed by the universal approximation theorem for neural networks [52]. Thus, Eq. ( provides a flexible and scalable lower bound for [Xx Z) with an unbiased gradient.
For the evaluation, we exploit a traceable manner [53 51 that joint samples (X(6), (0) by (()) (x))))). As for the marginal Qa(Z), (Z), we randomly select a datapoint j and then sample it from Q (Z) ( ( Px(X))). Details of the optimization are shown in Algorithm 1
Upper bound for 7(2 2²). Few efforts have been made that explore the traceable upper bound for mutual information 544 47 (5). Existing upper bounds (listed in Appendix D.1) are traceable with Q(Z12*) or Q(Z", Z²). However, these distributions lack interpretability and can hardly be directly modeled, which leads to untraceable estimations of the above upper bounds.
known probabilistic density of joint conditional distributions here being
To avoid the direct estimation of unknown probabilistic densities, we introduce an energy-based variational family for Q(Z", Zz) that uses a normalized critic 7,5 (2" Z²) like Eq. g to establish a traceable upper bound. Specifically, we incorporate the critic 7/8 into the upper bound LCLUB [55) to obtain a traceable Seasonal- -Trend Upper Bound (STUB) for I(Z", Z"), which is defined as:
1(Z", σ²) < EQ(2' .2) llog Q(Z)) (²) - Eg Q(2") | = ICLUB
(10)
= Eeq,.. am (Z. (7 z+)] - Eqg. (29) (2)0, (Z, (Z", Z = TSTUB-
(11)
The derivation details of this formula are provided in Appendix D.2. The inequality in Eq. (1) is tight only if 2 and Z' are a pair of independent variables (55). This is exactly sufficient condition for ISTUB, since MI and Eq. (IT) are both zeros on the independent situation, which is our seasonal- trend disentanglement optimal objective. The critic 7/9, similar to Ta. takes on the discriminating responsibility but provides converse scores, constraining the MI to minimum. However, Eq. (11 may get negative values during the learning of parameter f3 resulting an invalid upper bound for MI. To alleviate this problem, we additionally introduce a penalty term | optimization, which is an L.2 loss of the negative parts in ISTUR-
to assist the model
For the evaluation, we take the same sampling manner as the one in the lower bound and optimization details are also shown in Algorithm]] [
5

---

Algorithm 1 An epoch of the optimization of LaST.
Initialize the parameters of LaST: θ {w² the Φ-, Ø², 0¹ =
2: for mini-batch with size B consisting of (x(6, y hEB in training set
3: Get samples of the latent representations (2 JE and (Z JEB from distributions
(Z*| and {Qoi (2)) respectively;
4: Shuffle the (2 and {2 hiB and form (2-()) JEB and ) respectively;
5: Compute the 17,, n²: 17p B و (X (x0), 2001) nr a/2 D 6
c (x 2110)
6: Update parameters A: + + G(Ve)[CELBO +
(aa (x(1),201))
20(1))
of (e0a Tae (X(1, ) - feT: 220)))
B - (731 (70(1), 2((1))
70(20 ()61)) + average (T) Z)(0)) 70 (2 2) Jay []:
7: Update parameters r: P t G(Vr)( -i = Yau (x(1) z0((0) - 11 - (x00))20)))
Val (X()) (00)) 9 1 ((())
B -i=j= (79 (Ze(i, z0(i) 70((0()) z(61)
average([] 2400 - 70 ((000),))))))a }]:
8: end for
Experiments
We now present the results of our extensive experimental evaluations comparing LaST with state-of- the-art baselines and report a series of empirical results, along with ablation study and visualizations of seasonal-trend representations. Further details and results are provided in Appendix F.
5.1 Settings
Datasets and Baselines. We conducted our experiments seven real-world benchmark datasets from four categories of mainstream time scries forecasting applications: (1) ETT 25]: Electricity Transformer Temperature consists of the target value "oil temperature" and six "power load" fea- tures, recorded hourly (i.e., ETTh1 and ETTh2) and every 15 minutes (i.e., ETTml and ETTm2)
over two years. (2) Electricity, from the UCI Machine Learning Repository [ and preprocessed by (56), is composed of the hourly electricity consumption of 321 clients in kWh from 2012 to 2014. (3) Exchange (56) with daily exchange rates of eight countries from 1990 to 2016. (4) Weather contains 21 meteorological indicators (e.g., temperature and humidity) and is recorded every 10 minutes in 2020. We compare our LaST with the latest of-te-of-the-aee methods on time series modeling and forecasting tasks from two categories: (1) representation learning techniques, including COST 38 TS2Vec [37]. and TNC (6); (2) end-to-end forecasting models, including
VAE-GRU [TO. Autoformer 26). Informer 225 and TCN (22) Further descriptions and settings of
these baselines are provided in appendix F.I.
Evaluation setup. Following the prior work, we run our model on both univariate and multivariate forecasting settings. In multivariate forecasting, LaST accepts and forecasts all variables in datasets. In univariate forecasting, LaST only considers a specific feature in each dataset. We employ the standard normalization and set input length T = 201 for all datasets. For the dataset split, follow a standard protocol that categorizes all datasets into training, validation, and test set in chronological order by the ratio of 6:2:2 for all datasets. We report the evaluation results on the test set while the model achieves the best performance on the validation set.
Implementation details. As for the network structure of LaST, use single-layer fully connected network as the feed forward network (FFN), which is applied in the modeling of posterior, reconstruc- tion, and predictor. Besides, we employ the 2-layer MLP for the critic y in MI bound estimations. Dimensions of seasonal and trend representations are consistent. We set them as 32 in univariate forecasting and as 128 in multivariate forecasting other datasets. MAE loss used to measure the
htttaa///gihhhbb.ccohooooooa///p uci http:///ww...teeeeeeec/////////t
https://www.brr bgc-jeng.-j...... mpg. de/

---

Table 1: Univariate forecasting comparisons. Complete results on ETT benchmark are shown in appendix F.4. Best performance is highlighted in bold font and the second best results are underlined.
LaST
CoST
TS2Vec
TNC
VAF-GRU MAE
Autoformer MSE
Informer MSE
TCN
Method
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAB
MSB
MAE
MAE
MSE
MAE
24
0.030
0.131
0.040
0.152
0.039
0.151
0.057
0.184
0.042
0.155
0.057
0.189
0.088
0.247
0.104
0.254
ETTbl
0,051
0.169
0.060
0.186
0.062
0.189
0.094
0.239
0.077
0.218
0.070
0.207
0.158
0.319
0.206
0,366
168
0.078
0.211
0,097
0.236
0.142
0.291
0.171
0.329
0.172
0.344
0.108
0.260
0.183
0,346
0.462
0.586
336
0.100
0.160
0.316
0.140
0.301
0.119
0.222
0.387
0.422
720
0.8
0.298
0.148
0.306
0.179
0.345
0.235
0.408
0.204
0.381
0.109
0.264
0.269
0,435
0.438
0.578
24
0.011
0.077
0.015
0.088
0.016
0.093
0.019
0.103
0.013
0.082
0.022
0.115
0.030
0.137
0.027
0.127
E77ml
0.021
0.108
0.028
0.162
0.026
0.120
0.032
0.069
0.203
0.040
0.154
96
0.033
0.134
0.038
0.147
0.045
0.162
0.054
0.178
0.046
0.164
0.045
0.168
0.194
0,372
0.097
0.246
0.069
0.197
0.095
0.235
0.290
0.127
0.294
0.071
0.554
0.305
672
0.100
0.239
0.113
0.257
0.142
0.290
0.136
0.290
0.217
0.399
1.102
0.254
0,512
0,644
0.445
0.576
24
0.151
0.277
0.243
0,264
0,260
0,288
0.252
0.278
0.330
0.406
0.290
0.411
0.251
0,275
0.243
0.367
Electiiiity
0.186
0.307
0.292
0,300
0.313
0.321
0.300
0.308
0.437
0.481
0.310
0,408
0.346
0.339
0.283
0.397
168
0.243
0,346
0.405
0.375
0.429
0.392
0.412
0.384
0.433
0.476
0.435
0.490
0.544
0.424
0.357
0.449
0.286
0.379
0,560
0,473
0.565
0.478
0.548
0,466
0.472
0.504
0.646
0,713
0.512
0.355
0,446
720
0.322
0.422
0.889
0.645
0,863
0.651
0.859
0.651
0.543
0.563
0.609
0.587
1.182
0,806
0.387
0.477
Average
0.121
0.236
0.208
0.268
0.222
0.289
0.234
0.328
0.219
0.326
0.201
0.306
0.345
0,400
0.278
0.403
forecasting derived from the predictor. For the training strategy, we use the Adam [57 optimizer, and
training process is early stopped within 10 epochs. We initialize the learning rate with 10-3 and decay it with 0.95 weight every epoch.
5.2 Performance comparisons and model analysis
Effectiveness. Tables D and 2 summarize the results of univariate and multivariate forecastings respectively. LaST achieves state-of-the-art performance against the advanced representation base- lines on five real-world datasets. The relative improvements on MSE and MAE are 25.6% and 22.1% against the best representation learning method CoST and are 22.0% and 18.9% against the best end-to-end models Autoformer. We note that Autoformer achieves better performance on long horizons forecasting on hourly ETT datasets and think there are two reasons: (1) Transformer-bast models intrinsically establish long-range dependencies, which plays a crucial role in long sequence forecasting: (2) employs a simple decomposition by average pooling with a fixed kernel size, which is more suitable for strongly periodic datasets like hourly ETT. This phenomenon is beneficial to terg-term forecasting but limits the sensitivity to local context, and the bonus does not have significant impact on other datasets. Compared with baselines, LaST extracts the seasonal and trend patterns with disentangled representations adaptively and thus can be applied to intricate time series.
Ablation study. We investigated the performance benefits brought by each mechanism of LaST on a synthetic dataset (generation details are provided in appendix F.3) and ETTh1. The results are shown in Table 3, consisting of two groups: MI validates the mechanisms of seasonal-trend representations learning framework. In it, "w/o seasonal" and "w/o trend" denote LaST without the seasonal and trend components respectively; "w/o coe" denotes LaST without autocorrelation and CORT coefficients while estimating the reconstruction loss. M2 judges the introduction and estimations of MI, where"w/o lower" and "wwo upper" indicate the removal of the lower and upper bounds for MI in regularization terms respectively: "with MINE" denotes that we replace our lower bound with MINE. The results show that all mechanisms improve the performance on the forecasting task. We notice that the quality drops a lot when removing the trend component. The reason is that seasonal forecasting derives from the iDFT algorithm, which is essentially a periodical repetition of historical observations. However, it captures the seasonal patterns and assists the trend component in complete LaST to bring the superiority, especially in the long-term settings and strongly periodical synthetic dataset. Besides, we observe that with biased regularization term MINE, the performance becomes unstable and sometimes even worse than LaST without MI lower bound, while our unbiased bound (cf. Eq. continuously outperforms it.
Representation disentanglement. We visualize the seasonal-trend representations with the t- SNE (58) technique in Figure?? 2 We also visualize the embeddings last layer of Autoformer decoder as a comparison. The points with same color have clearer and closer clustering in LaST, while they mix together without decomposition mechanisms ("w/o dec" indicates removal of the two decomposition mechanisms (autocorrelation and CORT coefficients, and the upper bound to MI).
7

---

Table 2: Multivariate forecasting comparisons, Complete results on ETT benchmark are shown in appendix F.4, Best performance is highlighted in bold font and the second best results are underlined.
LaST
CiST
TS2
TNC
VAF-GRU MAE
Autoformer MSE
Informer MSE
TCN
Method
MSB
MAE
MSE
MAE
MSB
MAE
MSE
MAB
MSE
MAE
MAE
MSE
MAB
0.324
0.368
0.386
0.429
0.590
0.531
0.708
0.592
0.529
0.534
0.384
0.428
0.577
0.549
0.583
0.547
0.351
0.380
0.437
0.464
0.624
0.555
0.749
0.619
0.612
0.593
0.392
0.419
0.685
0.625
0.670
0.606
168
0.468
0,453
0.643
0.582
0.762
0.639
0.884
0.699
0.758
0.647
0.490
0.481
0.931
0.752
0.811
0.680
0.566
0.512
0.679
0.931
0.728
0.768
0.844
0.692
0.505
1.128
0.873
1.132
0.815
720
0.740
0.650
0.970
0.771
1.063
0.799
1.157
0.830
1.045
0.816
0.498
0.500
1.215
0.896
1,165
0.813
24
0.218
0.289
0.246
0.329
0,453
0.444
0.522
0.472
0.509
0.452
0.383
0.403
0.453
0.444
0.522
0.472
E17ml
0.280
0.329
0.331
0.386
0.592
0.521
0.567
0,642
0.543
0,454
0.453
0,494
0.503
0,508
96
0.323
0,360
0.378
0.419
0.635
0.554
0.731
0.595
0,600
0,540
0.481
0.463
0.678
0.614
U.666
0,578
0.392
0.403
0.472
0.456
0.693
0.597
0.649
0.769
0.678
0.634
0.528
1.056
0.786
0,991
0.735
672
0.491
0.466
0.620
0.574
0.782
0.653
0.932
0.712
0.799
0.673
0.606
0.542
1.192
0.926
1.032
0.756
Electriityy
24
0.125
0.222
0.136
0.242
0.287
0.375
0.354
0.423
0.190
0.250
0.165
0.286
0.312
0.387
0.235
0.346
0.146
0.245
0.153
0.258
0.309
0.391
0.376
0.438
0.228
0.280
0.178
0.295
0.392
0.431
0.253
0.359
168
0.170
0.265
0.175
0.275
0.335
0.410
0.402
0.456
0,240
0.297
0.215
0.327
0,515
0.509
0,278
0.372
0.188
0.280
0.196
0.296
0.351
0.422
D.417
0,466
0.262
0.318
0.218
0.329
0.759
0.625
0.287
0.382
720
0.223
0.309
0.232
0.327
0.378
0.440
0,442
0,483
0.296
0.347
0.252
0.356
0.969
0.788
0,287
0.381
0.033
0.122
0.033
0.127
0.108
0.252
0.105
0.236
0.064
0.178
0.060
0.178
0.611
0.626
2.483
1.327
Exchange
0
0.162
0.058
0.166
0.200
0.341
0.162
0.270
0.133
0.262
0.091
0.222
0.680
0.644
2.328
1,256
168
0.190
0,320
0.198
0.327
0.412
0.492
0.397
0,480
0.334
0.432
0.405
0.473
1.097
0.825
2.372
1.279
0.430
0.482
1.339
0.901
1.008
0.866
0.614
0.606
0.509
0.524
1.036
3.113
1,459
720
1.521
0.898
1.855
0.998
2.114
1.125
1,989
1.063
2.285
1.117
1,447
0,941
2.478
1,310
3,150
1,458
24
0.105
0.134
0.293
0.369
0.170
0.309
0.200
0.312
0.117
0.147
0.180
0.263
0.162
0,235
0.170
0.287
Weatee
0.131
0.174
0.558
0.515
0.231
0.375
0.284
0.367
0.227
0.270
0.241
0.310
0.348
0.400
0.327
0.365
168
0.197
0.238
0.812
0.671
0.470
0.532
0.475
0.502
0.234
0.280
0.295
0.355
0.444
0.463
0.517
0.569
0.257
0.285
1.196
0.832
1.360
0.875
1.405
0.309
0.339
0.359
0.395
0.578
0.523
0.639
0.608
720
0.315
0.327
1.600
1.002
2.173
1 1.120
2.034
1.018
0.444
0.410
0.419
0.428
1.059
0.741
0.639
0.610
Average
0.330
0.347
0.533
0.482
0.654
0.575
0.731
0.591
0.487
0.468
0.394
0.415
0.819
0.661
1.008
0.663
Table 3: Ablation study results on the two parts: M1 and M2.
Original
MI
M2
Method
LaST
seasonal MSE
trend MSE
wia coe
wlo lower
w/o upper
with MINE
Metric
MSE
MAE
MAE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
Synthetic
24
0.129
0.250
0.179
0.298
2.870
1.089
0.135
0.257
0.133
0,256
0.146
0.266
0,131
0,252
168
0.631
0.617
0.830
0.644
2.868
1.032
0.635
0.619
0.632
0.618
0.634
0.620
0.633
0.620
336
1.054
0.797
1.505
0.925
2,864
1.016
1.061
0.805
1.056
0.801
1.055
0.798
1.055
0.801
24
0.324
0.368
0.328
0.373
0.631
0.548
0.334
0.380
0.339
0,383
0.332
0.376
0,332
0.375
48
0.351
0.380
0.360
0.389
0.973
0.695
0.358
0.386
0.360
0.389
0,361
0.390
0.358
0.387
168
0.468
0.453
0.482
0.463
1.106
0.788
0.497
0.474
0.508
0.482
0.477
0.459
0.502
0.484
336
0,566
0.512
0.579
0.531
1.206
0.843
0.598
0.546
0.604
0.543
0.582
0.534
0.603
0.545
720
0.740
0.788
0.672
1.240
0.803
0.689
0.766
0.665
0.768
0.670
0.669
Average
0.533
0.503
0.631
0.537
1,700
0.858
0.553
0.520
0.550
0.517
0.544
0.514
0,549
0,517
Notably, though Autofomer with the simple moving average block achieves satisfying decomposition from the time series perspective, their representations are still prone to entanglement. These results suggest that (1) learning disentangled seasonal-trend representations is not trivial, and (2) the proposed decomposition mechanisms successfully disentangle the reasonal-trend representations in latent space, each paying attention to a specific temporal pattern.
Input settings. We further investigate the influence of hyperparameter input length to validate the sensitivity and Table | shows the results. Long look-back window improves the performance especially in long term forecasting, while others even have performance degradation. This verifies that LaST can effectively utilize past information to understand patterns and make predictions.
Table 4: Multivariate forecasting performance with different input lengths on ETTml datasets.
Input length
96
168
201
672
Average
Ouiput length
48
288
672
48
258
672
48
288
672
48
288
672
LaST
MSE
0,321
0.427
0.532
0.286
0.401
0,516
0,280
0.392
0.491
0.279
0.373
0.463
0.397
MAE
0.359
0.421
0.493
0.338
0.413
0.493
0.403
0.466
0.334
0.391
0.448
0.407
COST
MSE
0.390
0.514
0.644
0.335
0.467
0.611
0,331
0.472
0.620
0,349
0.474
0.612
0.485
MAE
0.422
0.507
0.585
0.392
0.484
0.569
0.386
0.486
0.574
0.399
0.486
0.567
0.488
AutoPormer
MSF
0.454
0.634
0.460
0.578
0.529
0.463
0.545
0.562
0.562
0.656
0.695
0.562
MAE
0.453
0.528
0,542
0.456
0.514
0.504
0,452
0,512
0,516
0,503
0,571
0,586
0.511
8

---

w/o dec
Autoformer
LaST
Figure 2: Visualizations of seasonal (red) and trend (blue) representations on ETThl dataset.
ETTh1
(b) ETTml
Exchange
Figure 3: Top: learned seasonality visualizations (autocorrelation statistics reconstructed seasonal sequences). Bottom: seasonal (red) and trend (blue) reconstructions the ground truths (black).
Observations from a case-study. We further validate LaST by by visualizing the extracted seasonality and trend in specific cases. As shown in Figure LaST can capture the seasonal patterns on real- hourly and 15-minutes ETT
world datasets. For example, strong daily period is indicated datasets. Even though the period on Exchange dataset is not obvious, LaST still provides some long-term periods on the daily data. Besides, trend and seasonal components jointly accurately restore the original sequence with their own perspective, which supports that LaST can produce workable disentangled representations for intricate time series.
6 Conclusion
We presented LaST, disentangled variational inference framework with mutual information straints to disassociate a couple seasonal- trend representations in latent space, for effective fore- casting of time series. Our extensive experiments demonstrated that LaST successfully disentangles the seasonal-trend representations and achieves state-of-the-an performance. Our future work will focus tackling other challenging downstream tasks in the time series domain, e.g., generation and imputation. In addition, we plan to model stochastic factors explicitly decomposition strategies, which will better understand the real-world time series.
Acknowledgments and Disclosure of Funding
This work was supported in part by National Natural Science Foundation of China (Grant No.62072077 and N6.62176043), Natural Science Foundation of Sichuan Province (Grant No. 2022NSFSC0505), and National Science Foundation SWIFT (Grant No.2030.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proof' of all theoretical results? [Yes] See Appendix A.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] Our source code for reproducibility is publicly available at GitHub.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See "Datasets" and "Evaluation setup" in Sec. [.1]
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See Appendix F.5.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix F.2.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] See URLs in Sec. [
(b) Did you mention the license of the assets? [Yes] See folder "license" in supplementary
materials.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Our source code for reproducibility is included in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See folder "license" in supplementary materials. (e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] They do not contain personally identifiable information offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
References
[1] Suman Ravuri, Karel Lene, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski. Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, et al. Skilful precipitation nowcasting using deep generative models of radar. Nature, 597(7878):672 677, 2021.
[2] Zhiyuan Wang, Xovee Xu, Goce Trajcevski, Kunpeng Zhang, Ting Zhong, and Fan Zhou. PrEF: Proba- bilistic electricity forecasting via copula-augmented state space model. In AAAI, 2022.
[3] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusson convolutional recurrent neural network: Data-driven traffic forecasting, In International Conference on Learning Representations, 2018.
[4] Amadou Ba, Mathieu Sinn, Yannig Goude, and Pascal Pompey. Adaptive learning of smoothing functions: Application to electricity load forecasting. In Advances in Neural Information Processing Systems. volume 25, 2012.
10

---

[5] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: survey. Philosophicul Transactions the Royal Society A, 379(21944;202202209 2021.
Qingsong Wen, Tian Zhou, Chaoli Zhang. Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transform- ers in time series: A survey. arXiv preprint arXiv:2202.07/2 2022.
[7] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[8] Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learning. 109(2):373 440, 2020.
[9] Matthew Johnson and Alan Willsky. Stochastic variational inference for bayesian time series models. In International Conference on Machine Learning, pages 1854-1862. PMI.R 2014.
[10] Otto Fabius, Joost R van Amersfoort, and Diederik P Kingma. Variational recurrent auto-encoders. In ICLR (Workshop), 2015.
[11] Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. Gp-vae: Deep probabilistic time series imputation. In International conference on artificial intelligence and statistics, pages 1651-1661. PMLR, 2020.
[12] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American stutistical Association, (2/518):859-877,201
[13] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798 1828, 2013.
[14] William Whitney. Disentangled representations in neural models. arXiv preprint arXiv: 0233822....... 2016.
[15] Cleveland Robert, C William, and Terpenning Irma. Stl: A scasonal-trend decomposition procedure based on loess. Journal of official statistics, 6(1):3-73, 1990.
[16] Mike West. Time series decomposition. Biometrika, 84(2):489-493 494, 1997.
[17] Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[18] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2019.
[19] Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu. Jiang Bian, and Tie- Yan Liu. Depts: Deep expan- sion learning for periodic time series forecasting. In International Conference Learning Representations, 2022.
[20] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181-119) 2020.
[21] Syama Sundar Rangapuram, Matthias W Seeger. Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Jamuschowski. Deep state space models for time series forecasting. Advances in neural information processing systems, 31, 2018.
[22] Shaojie Bai, Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, arXiv preprint 103iv:1803.0127,, 2018.
[23] Minhao Liu, Ailing Zeng, Zhijian Xu, Qiuxia Lai, and Qiang Xu. Time series is special sequence: Forecasting with sample convolution and interaction. arXiv preprint /00.0:20000093055, 2021.
[24] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in Neural Information Processing Systems, 32, 2019.
[25] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI, 2021.
11

---

[26] Jiehui Xu, Jianmin Wang, Mingsheng Long, et al. Autoformer: Decomposition transformers with auto- correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 2021.
[27] G Peter Zhang. Time series forecasting using hybrid arima and neural network model. Neurocopputing 50:159-175 1759 2003.
[28] George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
[29] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang. Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021.
[30] Pankaj Malhotra, Vishnu TV, Lovekesh Vig. Puneet Agarwal, and Gautam Shroff. Timenet: Pre-trained deep recurrent neural network for time series classification. arXiv preprint 1..3v:17766088355 2017.
[31] Xinrui Lyu, Matthias Hueser, Stephanie I. Hyland, George Zerveas, and Gunnar Ractsch. Improving elinical predictions through unsupervised time series representation learning. arXiv preprint arXiv: /812.00490 2018.
[32] David Salinas, Michael dimensional multivariate forecasting with low-rank gaussian copula processes. Advances neural information proeesing systems, 32, 2019.
Laurent Callot, Roberto Medico, and Jan Gasthaus. High-
[33] Emmanuel de B' B'ezenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bchlke-Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. In Advances in Neural Information Processing Systems, volume 33, pages 2995-3007, 2020.
[34] Danilo Rezende and Shakir Mohamed, Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
[35] Kashif Rasul. Sabuor Sheikh, Ingmar Schuster. Urs M Bergmann, and Roland Vollgraf. Multivariate probabilistic time series forecasting via conditioned normalizing flows. In International Conference on Learning Representations, 2020.
[36] Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Representations. 2021.
[37] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings AAAI, 2022.
[38] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In International Conference on Learning Representations, 2022.
[39] Qingsong Wen, Jingkun Gao, Xiaomin Song, Liang Sun, Huan Xu, and Shenghuo Zhu. Robuststl: robust scasonal-trend decomposition algorithm for long time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5409-5416, 2019.
[40] Qingsong Wen, Zhe Zhang, Yan Li, and Liang Sun. Fast robuststl: Efficient and robust seasonal-trend de- composition for time series with complex patterns. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2203-2213, 2020.
[41] Luke Godfrey and Michael S Gashler. Neural decomposition of time series data for effective generaliza- tion. IEEE transactions on neural networks and learning systems, 29(7):9973-29 2917773-99592110 2017.
[42] Michail Vlachos, Philip Yu, and Vittorio Castelli. On periodicity detection and structural periodic similarity. In Proceedings of the 2005 SIAM international conference on data mining, pages 449 460. SIAM, 2005.
[43] Autocorrelation: Efficient computation. https://en. wttp///ia.iippiia..aaa silkipedia..rg//vii///atttee wwtp://aa//uoooorrlllaaaaaaaaaaa Accessed: 2022-05-19.
Pefficient_computaaiee
[44] Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational autocn-
coders. arxiv preprint arXiv: 2017.
[45] Ali Lotfi Rezaabad and Sriram Vishwanath. Learning representations by maximizing mutual information in variational autoencoders. In 2020 IEEE International Symposium on Information Theory (ISIT), pages 7729-2734, IEEE, 2020.
12

---

[46] Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6):11111255:
2003.
[47] Ben Pnole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference Machine Learning, pages 5171-5180. PMLR, 2019.
[48] David McAllester and Karl Stratos. Formal limitations the measurement of mutual information. In International Conference Artificial Intelligence and Statistics, pages 875-884. PMLR, 2020.
[49] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markow process expectations for large time. Communications pure and applied mathematics, 36(2):833-11, 1983.
[50] David Barber and Felix V Agakov. The im algorithm: A variational approach to information maximization. In Advances in Neural Information Processing Systems, 2003.
[51] Mohamed Ishmacl Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pages 531-540. PMLR, 2018
[52] Kurt Hornik, Maxwell Stinchcombe, and Halhent White. Multilayer feedforward networks universal approximators. Neural networks, 2(5):359-366 1989.
[53] Miguel A Arcones and Evarist Gine. On the bootstrap of and statistics. The Annals Statistics, pages 655-674, 1992.
[54] Alexander Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, ICLR, 2017.
[55] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: contrastive log-ratio upper bound of mutual information. International conference on machine learning. pages 7789-1788. PMLR, 2020.
[56] Guokun Lai, Wei-Cheng Chang, Yiming Yang, Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 95-104, 2018.
[57] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations. ICLR, 2015.
[58] Laurens Van der Maaten and Geoffrey Hinton, Visualizing data using t-sne. Journal of machine learning research, 9(1 1), 2008.
13


Reviews:
[{"review_id": "zO7QCIwaOvP", "reviewer": "Reviewer_JNkP", "summary": "This paper focuses on the time series forecasting task and presents a decomposition method LaST to learn latent representations. The decomposition method is derived from variational inference and mutual information. Along with a predictor, LaST can achieve competitive performance in many benchmarks.", "questions": "1. What is the difference between the autocorrelation calculation used in Equ 6 and the autocorrelation calculation method in Autoformer? Maybe giving a citation is better.\n2. The visualization in Figure 2 is not surprising. You can conduct the same visualization to Autoformer. I think the simple moving average block can achieve the representation disentanglement well. You can compare these two decomposition methods.\n3. I am a little confused about the predictor (Figure 1 (b)) without the supplementary material. Especially in lines 127-128, the inverse DFT cannot change the sequence length. You should give more details about the “extend” in the main text.\n4. It seems the prediction results show the over-smoothing problem (Figure 3). This problem is fatal for details predicting of time series. This can be caused by the design of the predictor. More showcases are required especially the non-stationary time series.\n", "limitations": "No, the author has not discussed any limitations of this work. I think the convergence property can be further analyzed. Also, the stochasticity of time series is neglected in the decomposition, which should also be noticed by the authors.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The loss function for time series decomposition is reasonable and novel.\n\n2. This paper is well-organized and clear.\n\n3. The model performance is competitive and is with detailed analysis.\n\n### Weaknesses\n1. The ‘proof’ of the ‘Theorem’ 2 is too intuitive.\n- The minimization between autocorrelations of $X$ and $\\hat{X}^s$ is not convincing, which will bring the noise to the optimization. And this process also relies on an underlying assumption, that the raw time series is periodic by removing the trend, which is contradictory to the statement in line 89.\n- The same problem is also in the design of CORT($X$,$\\hat{X}^t$).\n2. Comparing baselines:\n- I think the N-BEATS is a necessary baseline since you both adopt the decomposition and a similar predictor for forecasting (linear model for trend and sine/cosine functions for seasonal).\n- The most important thing of this paper is to explain the necessity of learning ‘latent’ representations. What about using the moving average of Autoformer for decomposition and adopting the same predictor for prediction? I think this vanilla baseline is also necessary.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "ovVwsDvcg6", "reviewer": "Reviewer_hjLn", "summary": "The paper presents an approach for learning latent season-trend based representations for time-series forecasting, based on variational inference. A sesonal encoder and a trend encoder are used to obtain latent representations. The latent representations are then combined together to produce a forecast. The forecast is done via the predictor module, which makes use of Fourier Transform for the seaosnal representation and a simple MLP for a the trend representation, and then sums them up to obtain the forecast. Reconstruction error cannot be minimized directly, as the true trend and season components are not known, as a result autocorrelation distance is used for seaosnality reconstruction, and the temporal correlation is used for reconstructing the trend reconstruction. For the purpose of disentanglement, the MI between the latent components is minimized. The overall loss function has an ELBO loss, and MI maximization between the original time-series and latent component, and MI minimization between the season and trend latent components. Since the optimization is untractable, lower and upper bounds for MI are derived to help with the optimization. Experiments are performed on standard univariate and multivariate timeseries benchmarks, showing LaST achieves state of the art performance. Visualizations show effective disentanglement.  ", "questions": "- how complex was performing this training and getting it to work? To me, it seemed like a daunting task", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- good motivation to expand on decomposition of trend seasonality\n- novel way to do so compared to existing approaches (e.g. variational inference vs CoST using contrastive learning)\n- meaningful loss function (elbo, reconstruction, predictor)\n- identified technical challenges in reconstruction, and proposed a way to address it\n- identified challenges in MI optimisation and developed bounds to address it \n- good coverage and comprehensive experiments - baselines + datasets\n- good ablation / visualizations\n- a generally comprehensive paper which is well written\n\nWeakness\n- superposition of a lot of complex components to make it work, possibly creating uncertainty on the robustness of the model", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FqjOEGdKLx7", "reviewer": "Reviewer_GaRG", "summary": "The author proposed VAE-based method, LaST, to disentangle the seasonal-trend representations of sequential data. The LaST uses the trend and seasonal data as the reconstructed targets, which enforce the model to learn representations dedicated to its target hence achieving better disentanglement. Finally, a predictor is introduced to guarantee its performance on the downstream tasks. Results show that LaST can achieve better predicting performance compared to other forecasting baselines.", "questions": "1. Does these trend data Xt and seasonal data Xs are extracted from X? If so, how do you extract them?\n2. Can you compare your results with other disentangled representation learning methods for sequential data?\n", "limitations": "1. The generalisability of this LaST might be limited since it requires that the input data are a triplet: Time series data X, trend data Xt and seasonal data Xs. Many sequential datasets, especially the real-world ones, often don't have the complete triplet data or with a lot of missing/noising data. So its generalisability should be further investigated.\n\n2. The author might need to add more baselines (other disentangled representation learning methods for sequential data).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The usage of trend and seasonal input Xt and Xs force the model to learn dedicated representations and make us easier to evaluate the disentanglement.\n2. The author provides rigorous mathematical proof, including the decomposition of ELBO, and the lower and upper bounds for MI optimization.\n3. The author provides extensive results on both the prediction performance for downstream tasks and the disentanglement of the trend and seasonal features.\n\nCons:\n1. The author stated that \"existing approaches with a single high-dimensional representation sacrifice the information utilization and explainability\". Models like CoST uses different modules to extract trend/seasonal dependencies. The author fails to address the difference between those models that do not use a high-dimensional representation.\n\n2. Other disentangled representation learning methods for sequential data are not referenced as baselines i", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6u_gBx4gZKF", "reviewer": "Reviewer_r48g", "summary": "In this submission, the authors propose to learn disentangled seasonal and trend representations of seasonal time series. In particular, in contrast to previous methods that use average pooling for the trend feature, the major contribution here is to directly optimize the two representations by variational method. Also, a detailed analysis of the loss function and the optimization bound is provided. Empirical results show strong effectiveness and outperforms several strong baselines. The following analysis also supports the motivation.", "questions": "1. When discussing the connection to previous papers, the writing should both include the difference and the inheritance. Which components are extended from the previous studies? Also, what are techniques that can potentially be combined (such as contrastive learning from CoST)?\n\n2. What is the boundary of the method? The drawback of feature degeneration should be discussed in more detail.\n\n3. What is the training and inference time? How it can be compared with the baselines?", "limitations": "As discussed above, the reviewer encourages the authors to face the general limitation of VI methods and discuss the implications in this setting.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: \nThe proposed method of representation disentanglement is an extension of other fields such as computer vision. However, given the nature of seasonal time series, the application of this method is non-trivial, as well as the design of a proper predictor. The bound proof is original in this setting.\n\nQuality: \n1. The conduct of empirical analysis is of high quality. The experiments are sufficient to support the claims (great performance boost). One weakness, as also noticed by the author, is the potential degeneration of features caused by narrowing down prior and posterior, and the effect is not discussed in detail. \n2. As the reviewer is not an expert in theoretical analysis, I would leave this part to other reviewers.\n\nClarity:\n1. The overall presentation is clear, including a good schematic pipeline description and results tables.\n2. The development of Theorem 2 is a bit confusing. The authors first claim that L rec can be estimated without leveraging Xs and Xt, while eq 6 still has the two variables. Line 143, what is \"first difference?\"\n\nSignificance:\nThe reviewer thinks this method provides significant advances in this subfield.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept