VARIATIONAL ADAPTIVE GRAPH TRANSFORMER FOR MULTIVARIATE TIME SERIES MODELING
Authors: Long Tian
Wenchao Chen
Bo Chen
Muyao Wang
Liang Dai
BaoLin Sun
Mingyuan Zhou
Venue: Submitted to ICLR 2023 (2023)

Abstract: Multivariate time series (MTS) are widely collected by large-scale complex systems, such as internet services, IT infrastructures, and wearable devices. The modeling of MTS has long been an important but challenging task. To capture complex long-range dynamics, Transformers have been utilized in MTS modeling and achieved attractive performance. However, Transformers in general do not well capture the diverse relationships between different channels within MTS and have difficulty in modeling MTS with complex distributions due to the lack of stochasticity. In this paper, we first incorporate relational modeling into Transformer to develop an adaptive Graph Transformer (G-Trans) module for MTS. Then, we further consider stochastity by introducing a powerful embedding guided probabilistic generative module for G-Trans to construct Variational adaptive Graph Transformer (VG-Trans), which is a well-defined variational generative dynamic model. VG-Trans is utilized to learn expressive representations of MTS, being an plug-and-play framework that can be applied to forecasting and anomaly detection tasks of MTS. For efficient inference, we develop an autoencoding variational inference scheme with a combined prediction and reconstruction loss. Extensive experiments on diverse datasets show the efficient of VG-Trans on MTS modeling and improving the existing methods on VG-Trans outperforms state-of-the-art methods on a variety of MTS modeling tasks.

Under review as a conference paper at ICLR 2023
VARIATIONAL ADAPTIVE GRAPH TRANSFORMER FOR MULTIVARIATE TIME SERIES MODELING
Anonymous authors Paper under double-hlind review
ABSTRACT
Multivariate time series (MTS) are widely collected by large-scale complex systems, such as internet services, IT infrastructures, and wearable devices. The modeling of MTS has long been important but challenging task. To capture complex long- range dynamics, Transformers have been utilized in MTS modeling and achieved attractive performance. However, Transformers in general do not well capture the diverse relationships between different channels within MTS and have difficulty in modeling MTS with complex distributions due to the lack of stochasticity. In this paper, we first incorporate relational modeling into Transformer to develop an adaptive Graph Transformer (G-Trans) module for MTS. Then, we further consider stochastity by introducing powerful embedding guided probabilistic generative module for G- Trans to construct Variational adaptive Graph Transformer (VG- Trans), which is well-defined variational generative dynamic model. VG-Trans is utilized to learn expressive representations of MTS, being an plug-and-play framework that be applied to forecasting and anomaly detection tasks of MTS. For efficient inference, we develop an autoencoding variational inference scheme with combined prediction and reconstruction loss. Extensive experiments on diverse datasets show the efficient of VG-Trans on MTS modeling and improving the existing methods on a variety of MTS modeling tasks.
INTRODUCTION
Multivariate time series (MTS) is important type of data that arises from wide variety of domains, including internet services (Dai et al., (202T: 2022). industrial devices (Finn et al. 2016 6 Oh et al. 2015). health care (Choi et al./ 2016bb a), and finance (Maeda al. [019] Gu et al. 2020). to name a few. However, the modeling of MTS has always been a challenging problem as there exist not only complex temporal dependencies, as shown in the red box in Fig. D but also diverse cross- channel dependencies, as shown in the blue box in Fig. [ Moreover, there exist inherently stochastic components, as shown in the green box in Fig. [ even if one can fully capture both temporal and cross-channel dependencies. To address these challenges, many deep learning based methods have been proposed for various MTS tasks, such as forecasting, anomaly detection, and classification.
To model the seappoorl-ddeeedenciees of MTS, many dynamic methods based on recurrent neural networks (RNNs) have been developed (Malhotra et al. 2016 Zhang et al. 2019 Bai et al. 2019b Tang et al. 2020; Yao et al. 2018). Meanwhile, to take the stochasticity into consideration. some probabilistic dynamic methods have also been developed (Dai et al. 2021 2022, Chen et al. 2020
2022; Salinas et al.) 2020). With the development of Transformer (Vaswani et al. 2017 and
due to its ability to capture long-range dependencies (Wen et al.) 2022 Dosovitskiy et al. 2021 I
Dong al. 2018 Chen et al., 2021), and interactions, which especially attractive for time series modeling, there is a recent trend to construct Transformer based MTS modeling methods and have achieved promising results in learning expressive representations for down-stream tasks.
For example, for forecasting, Trans (Li et al., 2019) incorporates causal convolutions into self-attention layer to consider local temporal dependencies of MTS. Informer (Zhou et al. 2021) develops a probsparse self-attention mechanism for long sequence forecasting. AST et al.) 2020) further constructs a generative adversarial encoder-decoder framework for better predicting output distribution. In addition, there are also some other efficient Transformer-basee forecasting methods, such as Autoformer (Xu et al. 2021 I). FEDformer (Zhou et al. 2022), and TFT (Lim et al. 2021).

---

Under review as a conference paper at ICLR 2023
Temponl
Besides, for anomaly detection, Meng et al. (2019) illus- trate the superiority of using Transformer for anomaly de- tection over other traditional RNN- based methods. Fol- lowing it, some modified Transformer-basee have also been proposed for anomaly detection, such TransAnomaly (Zhang et al. (021)) ADTrans (Tuli et al.] 2022). and Anomaly Transformer (Xu et al., 2022). To ad- dress non-deterministic temporal dependence within MTS,
methods
Figure 1: The temporal dependency, chan- nel relationship and stochasticity within MTS.
Tang & Matteson (202T) further incorporate Transformer
structure into state-space space models and develop ProTrans.
Despite the attractive performance of existing Transformer-based models, their ultimate potentials have been limited by ignoring the cross-channel dependence of MTS.
To consider the relationships of different channels within MTS, MSCRED (Zhang et al.) 2019 introduces multi-scale convolutional recurrent encoder&decoder to learn spatial correlations and temporal characteristics in MTS and detects anomalies via the residual signature matrices. InterFusion 202I) incorporates recurrent and convolutional structures into unified framework to capture both temporal and
information. Recently, Graph neural networks (GNNs) have gradually attracted more attentions in exploring the relationships. Thus, some GNN-based methods for MTS have been developed (Deng & Hooi| [021] /Zhoo et al. f020) discovering expressive representations MTS. Deep variational graph convolutional recurrent network (DVGCRN) incorporates relationship modeling into hicrarchical generative process. Moreover, some graph based methods (Li et al. 20188]
Bai et al., 2019a| Yu et al.] 2018) Wu et al., 2019 Guo al., 2019 Pan al. 2019) have also been developed for MTS forecasting. Adaptive graph convolutional recurrent network (AGCRN) (Bai al. 2020) further learns node-specific patterns for MTS forecasting without requiring pre-defined
graph. However, these methods are still all non-dynamic or RNN based models, limiting their power in capturing complex relationships across long-distance time steps
Moving beyond the constraints of previous work, we first propose adaptive graph Transformer (G-Trans) module by incorporating a graph into the Transformer structure, which can model both temporal and cross-channel dependencies within MTS. Then, considering the stochasticity within MTS and enhancing the representative power of G-Trans, we further develop a Variational adaptive Graph Transformer (VG-Trans) which is well-defined probabilistic dynamic model obtained by combining G- T-Tanss with a proposed Embedding guided Probabilistic generative Module (EPM), illustrated in Fig. (b). We note that VG-Trans is able get the robust representations of MTS, which enables it to be combined with the existed methods and applied to both anomaly detection and forecasting tasks. In addition, we introduce an autoencoding variational inference scheme for efficient inference and a joint optimization objective that combines forecasting and reconstruction loss to ensure the expressive time- series representation learning. The main contributions of our work are summarized as follows:
For MTS modeling, we propose a G-Trans module, which incorporates channel-relationshi) learning into the Transformer structure.
We develop VG-Trans, a VAE-structured probabilistic dynamic model with G-Trans as encoder and EPM decoder, which can consider the non-deterministic within both temporal and cross-channe dependencies of MTS. VG-Trans can be combined with different methods and applied to different tasks of MTS.
To achieve scalable training, we introduce an autoencoding inference scheme with com- bined prediction and reconstruction loss for enhancing the representation power of MTS.
Experiments on both anomaly detection and forecasting tasks illustrate the efficiency of our model on MTS modeling
METHOD
We first present the problem definition, and then introduce the probabilistic channel embedding for measuring the relationships between different channels and present G-Trans by incorporating cross-channe dependence into Transformer. Finally, we develop VG-Trans, a novel variational dynamic model. The notations used in this paper are summarized in Table 4 in Appendix.
2

---

Under review as a conference paper at ICLR 2023
Preprocesing
Represestation
Common Tacks
Decodur
Add Normalization
Normalization
Anumaly detection
EPN
Feed Forward Network
VG-Trans
Normalizaiion
Dala Slicing
Forecasting
tape scsies
Mulli-heud allentioo
3 teciinal Eraedro
Prediction
G-Trans
Dynamic latent stales
Ialent variables
Probabilisdic
EPN
Gronh covelutional network
Channel Embedding
Recunsrrution
Trans
Some Enieddies
Add Normalization
Encader
Mulli-hrad Sell Faliiiio
(a)
(b)
Figure 2: (a) The whole framework of VG-nrans for MTS modeling; (b) Graphical illustration of inference (encoder) and generative (decoder) models (Blocks in gray and green represent input and probabilistic latent variables, while the colourless blocks represents deterministic dynamic latent states.); (c) The detailed structure of adaptive graph Transformer module.
Dr where n 1....N and is the number of MTS. is the duration of æn and the observation at time t, It,n E RV, is a V dimensional vector where V denotes the number of channels, thus Vn E RT*V, The modeling of MTS is to learn the robust representations of the input with powerful method, which can be served as plug and play framework and applied to different tasks.
Problem Definition: Defining the n-th MTS as 2n, = {I1,n= 22,n.
2.1 PROBABILISTIC CHANNEL EMBEDDING
To reflect the characteristics of different channels MTS and capture their non-deterministic relationships, introduce the probabilistic embedding vector for each channel in MTS as:
a; ~ N (p,, diag (ar,) E R [a, avl
where N-)) means Gaussian distribution. a refers to the channel embbedding for inputs. To fully take the advantage of uncertainties brought by -iittrnnddsttibuueee embeddings, we measure the distance between different channels with the expected likelihood kernel Jebara et al. 2004)) as
s(a;; a2) SacRe N (æ; μæ diag(or)) W (æ; My, diag(or)) dr = N (0; p. - M1 diagío, + +a,)
Clearly, symmetric similarity function, s(-) incorporates both the means and covariances of two Gaussians, considering the uncertainties within them. The parameters channel embeddings are initialized randomly and then trained along with the rest of the model.
2.2
ADAPTIVE GRAPII TRANSFORMER
To consider both temporal and cross-channel dependencies, we introduce graph structure into the Transformer framework and develop an G-Trans module. The detailed structure of G-Trans shown in Fig. 2 (c). Specifically, first introduce Multi-head Self Attention (MSA) block to capture the temporal dependence as
= MSA(X) = con (H Hm) H, SA (Q,, K,, V,) =softmax
Q Q
IV,
(I)
Vdx
where X E XT denotes the input and con(.) means concat operation, Qs = W E dd XT K; W E xT where i E 21, m) and is the number of heads. It is worth noting that we assign V; = X to keep the meaning of channels within MTS unchanged for capturing their relationships follow-up. O E (Vxxxxm)) is the output of MSA block. Then, different from traditional Transformer that uses feed forward network after multi-head self-attention block, which can not well capture the diverse relationships between different channels, we introduce an Adaptive Graph Convolutional Network (AGCN) block. Specifically, with the O from MSA block and channel embeddings a defined in Section 2.1 AGCN discoveries channel dependencies automatically as
log(s(a, a)), (s Conv(O))), Á = softmax (RUMnX)
h AGCN(a, 0) = In(1 exp W AH))

---

Under review as a conference paper at ICLR 2023
where A RVXV is the relational matrix calculated by symmetric similarity function and a is updated to be adaptive to the MTS data. Conv(.) means the convolutional operation, which is utilized to summarize the multi-head information within O into H E RVXI. Á is the normalized symmetric adjacent matrix. W R xV is the GCN filter. After combining temporal and channel-wise relational information of MTS into h E RK xT, multi-head self attention and feed forward network blocks are further applied for exploring expressive representations of MTS, as the red box shown in Fig. (c), and getting the dynamic latent states h E RRXT Then, can select reconstruction or forecasting decoder for different tasks.
2.3 VARIATIONAL ADAPTIVE GRAPH TRANSFORMER
Previous Transformer based MTS modeling methods are always equipped with deterministic gen- erative model, which ignores the stochasticity within MTS and has difficulty in modeling complex MTS with sophisticated distribution. To address this issue, we further develop VG-Trans, which is a novel VAE-structured dynamic model equipped with powerful Embedding guided Probabilistic generative Module (EPM) as decoder and G- Graarsss as encoder. The graphical illustration of the generation (decoder) and inference (encoder) operations of VG-Trans shown in Fig. (b).
Embedding guided probabilistic generation: To consider the stochasticity within MTS, we first introduce Gaussian-distribute latent variable 2t,n generative process as
RK at each timestep, and then define the
(Wh.ja Zt,n ~ N (M diag (or,,)) Mr, = S h. 1
Ft. ~ N (p diag (oi,)) M4, = (W Zp + hy ht-1)))
As illustrated in Fig. (a) and (b). 1t,, and Ot. are means and covariance parameters of žt,n- h E RR-K' denotes the deterministic latent states of G-Trans module. We combine 2t,n and h into generative process to consider the temporal dependencies and the stochasticity. ʃ(-) refers to the non-linear activation function. Moreover, to further capture the cross-channel dependencies of inputs and latent variables, we introduce channel embeddings into generation process by defining
W log (s (aB2)). W by softmax (a31)
(4)
The channel embeddings are incorporated into generative process by defining the factor loading matrices W and We, as the mapping function of them, which can capture the non-deterministic inter-relationships between channels, as introduced in Dieng et al. (2020), thus to improve the capacity of model in modeling complex MTS. Bg xax is the mapping matrix that transmit
Zt.n into the embedding space of Ft.n, while 3r E R is the mapping matrix that transmit hen into the embedding space of øt,n. We call our generative model as EPM. Compared with generative module of previous Transformer based methods for MTS modeling, EPM discovers the latent semantic structure of each channel as an probabilistic embedding vector and capturing the relationships between each other according to the similarity of channel embeddings, meanwhile, considering the -enemmteriii within them, thus to improve the representative capacity.
The purpose of inference module is to map the inputs H,, to zn. To consider the locality temporal dependencies, we first apply convolutional operation on it,n as it.,. Conv(at,,) ) E IR Then, with MSA and AGCN blocks, we summarize temporal and channel-wise relational information of input MTS as
Inference:
On = MSA(&,), hn = AGCN (On,, a))
(5)
Given time series hy we apply a linear projection and combine with a positional embedding to obtain
n MayyerNor((MLP(h.... + Position (1))
(6)
(0) With f as the input, we then apply multi-head self attention and feed forward network blocks get the dynamic latent states h., as shown in Fig. (c). Following VAE based models Kingma & Welling (2014), we define a Gaussian distributed variational distribution q(2t,,)) = NM diag( (t.g) to approximate the true posterior distribution p(z),nl-) and map the dynamic layent states h, their parameters as:
Hl, = (Cap hen + bau) ot. = Softplus (f (C (C he, + bya))
(7)
where Cryy Cuo E KXV Daps bao TR are all learnable parameters of the inference network.
4

---

Under review as a conference paper at ICLR 2023
3 MODEL TRAINING
As mentioned in Cao et al. 2020); Wen et al. 2022). the prediction-based model is expert in capturing the periodic information the MTS, while the reconstruction-base based model can explore the global distribution of the time series. To combine the complementary advantages of the two models for facilitating the representation capability of MTS, we formulate the optimization function as the combination of both prediction and reconstruction losses and define the marginal likelihood
P(Dla, W) f I p( Zt, a) + p(ZT,n 1, a dz)11.
where the first and the second term are reconstruction and prediction loss separately. Similar to VAEs, with the inference network and variational distribution in Eq. equation 77 the optimization objective of VG-Trans can be achieved by maximizing the evidence lower bound (ELBO) of the log marginal likelihood, which can be computed
E (==... [mmp (Ze, ze,) + [In (xr,,) h1T-110) In ( k2)..) ((
c -=
(8)
where > O is hyper-parametee to balance the prediction and the reconstruction losses, which chosen by grid search on the validation set. The parameters of channel embedding can be learned with Bayes by Backprop (Blundell et al.] 2015) as it can be reparameterized as s(a;, Q₃) = (1:: - -41 + (diag(o; + o;)) Eij- The detailed procedures of the optimization of VG-Trans are summarized in Appendix.
APPLICATION TO ANOMALY DETECTION TASK
Anomaly detection of MTS is defined as problem that determines whether observation from a certain task and at a certain time anomalous or not. Specifically, the model trained to learn normal patterns of multivariate time series, the more observation follows normal patterns, the more likely it can be reconstructed and predicted well with higher confidence. Our model is an unsupervised probabilistic generative model, which can be applied to unsupervised anomaly detection directly. Specifically, we apply the reconstruction and prediction error of æ, as the anomaly score to determine whether an observed variable is anomalous not, and it computed
(9) where Si,,, and S are reconstruction and prediction score, respectively. An observation 2; will be classified as anomalous when Sr,, is below a specific threshold. From practical point of view, we use the Peaks-Over- O (POT) (Siffer et al. 2017) approach to help select threshold.
St.n = (S + 7( S(,S( (1111111110 7),S7, ST log p( ,,,n),
(Ii,n 2t,0)
Moreover, we note that after collecting both non-enteriiiistic temporal and channel dependencies information within VG-Trans and get latent representations of MTS, can introduce more powerful structure for different specific tasks. Specifically, we combine our model with Anomaly Transformer (Xu et al., 2022 by introducing anomaly attention and association discrepancy into VG-Trans and get the (G-AAnomalyTrram
5 APPLICATION TO FORECASTING TASK
Considering forecasting task of MTS being formulated as finding the function to predict the next time steps (Bai al. 2020) given the past T' time steps
{ {0. 22.42. ,. 2. Fe (a...
2.1-T411
(10)
where refers to the learnable parameter. Since VG- Trans is a representation learning method, we can refer to as a play and plug framework, which can be applied to forecasting task by combining with corresponding methods. Inspired by the effectiveness of Autoformer in forecasting, Trans and get the VG-Autofommer. As shown in Fig. 3 VG- Autofororme first get latent representations of MTS
Channel Eubodding
Ershadding Gusbed Coneratie
Receedrattion
Kl Lon
Senmal lnk Tradk diaa
Candiadien
(Fibadlig Gabbel
MTS
Trandoenar
combine it with our proposed VG-
Deaxhe
Figure 3: The framework of VG-Autoformer.
5

---

Under review as a conference paper at ICLR 2023
Table 1: Fl-score performance of different methods.
CDN
SMD
MSL.
SMAP
Methods
R
FI_score
P
R
Fi_score
P
R
FI score
P
R
FI_score
OCSVM
10
0.508
0.640
0.566
0.443
0.767
0.562
0.598
0.866
0.708
0.538
0.591
0.563
bsotationForst
0.496
0.640
0.423
0.733
0.536
0.539
0.865
0.664
0.524
0.590
0.555
LOF
10
0.667
0.364
0.671
0.563
0.399
0.467
0.477
0.853
0.612
0.589
0.563
0.576
LSTM
10
0.585
0.614
0.599
0.786
0.852
0.818
0.855
0.825
0.840
0.894
0.781
0.834
VRNN
0,957
0.795
0.874
0.902
0.893
0.821
0.813
DOMI
10
0.938
0.878
0.907
0.943
0.913
0.927
0.896
0.639
0.746
0,864
0.567
0.685
BeaGAN
0.875
0.924
0.559
0.696
OmniAnomaly
10
0,983
0,875
0.926
0.837
0.868
0.852
0,890
0.864
0.877
0.925
0.820
0.869
SDEVAE
10
0.972
0.903
0.936
0.882
0.926
0.903
0.853
0.894
0.873
0.884
0,908
0.896
GNN
10
0.933
0.902
0.917
0.829
0.964
0.891
0,881
0.889
0.885
0.812
0,944
0.873
Interfusion
0.980 10
0.883
0.929
0.870
0.854
0.862
0.813
0.927
0.866
0.898
0.885
0.891
THOC
0.909
0.910
0.897
0.907
Trans
10
0.861
0.802
0.830
0.853
0.748
0.797
0.856
0.813
0.834
0.913
0.568
0.700
Anomaly-Tiae Trans
10
0.888
0.991
0.937
0.894
0.954
0.923
0.921
0.951
0.936
0.941
0.994
0.967
VG-Trans
10
0.903
0.998
0.948
0,972
0.895
0.932
0.930
0.959
0.944
0.984
0.928
0.955
VG-Anomaly-
0.953
0.925
0.933
0.938
0.964
0.949
0.974
0.964
16
23
15
25
Time
Time
Figure 4: Visualization of example channels and corresponding relational matrices a on SMD machine-2-3 dataset.
with VG-Trans, then transport them with Autoformer encoder. Finally, after by combining channel embedding with Autoformer decoder, we develop an embedding guided Autoformer decoder as:
(11) V and OT+1:744 VxT separately represent the seasonal and trend- cyclical outputs of Autoformer decoder, which is refined by channel embedding a via a learnable parameter B3 E Rd The intuition behind this is that long-term channel relationships may lie in
seasonal ET+1T++= W seasonal T+
Wtrend O Wtrend sof softmar(a(..)
where 07+ 7+1774
seasonal
frend
trend-cyclical part rather than seasonal part. The detailed structure of VG-Autoformer is introduced Appendix. To optimize VG-Autoformer for forecasting, we modify the loss in Eq. 12 as
N
q (2 Ft,n)
==
E B [In (It, z1,n) + (11p(pr-1T77 hI.T.) In
p p h)-1n))
6 EXPERIMENT
We conduct extensive experiments to evaluate the performance of our proposed models on forecasting and anomaly detection tasks of MTS.
Datasets and set up: We evaluate the effectiveness of our model on twofold datasets:1) four real- world datasets for anomaly detection, including CDN Dai et al. (202I). SMD, MSL and SMAP Xu et al. 2022); 2) four datasets for forecasting, including ETTh, E ETTm, Weather and ECL. (Zhou et al. 2021. The results are either quoted from the original papers reproduced with the code provided by the authors. The way of data preprocessing is the same as Dai et al. (021). where the window size T' and overlap are set T 20, o 5 for anomaly detection and 20,
for forecasting. Adam optimizer Kingma& & Ba (2015) is employed with learning rate of 0.0002, the batch size is set be 64. The number of heads V--Trans set as The summary statistics of these datasets and other implementation details are described in Appendix.
6.1 ANOMALY DETECTION
Similar to the previous studies (Dai et al. 2021. we employ Precision, Recall, and Fl-score as the evaluation metrics to indicate the anomaly detection performance of different methods. Particularly,

---

Under review as a conference paper at ICLR 2023
Table 2: Multivariate results with predicted length as (96, 168. 288, 336} on the five datasets and (24,48,96, 48,96, |68) on the Weather dataset, lower scores are better. Metrics are averaged over 5 runs, best results are highlighted in bold.
Models Metric
VG-autaformer
Autoformer
Informer
LogTrans
Reforee
LSTNet
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.420
0.435
0.426
0.442
0.925
0.761
1,008
0.875
0.774
0.637
1.257
0.983
168
L.24
1.138
1.997
1.214
258
0.502
0.483
0:514
0.494
1.120
0.835
1.141
0.972
0.910
0.700
1.528
1.376
336
0.494
0.482
0.505
0.484
1.128
0.873
1,362
0.952
2.177
1,280
2,655
1.369
96
1,297
1.656
1.054
3.367
1.982
ETTh2
168
0.429
0.443
0:157
0.455
3.189
1.515
4.070
1.681
1,660
1.846
3.212
2.513
258
0.459
0.463
10470
0.468
6.020
2,030
7.172
2.471
2,730
1 1,346
8.252
2.665
336
0.458
0.467
0.471
0.475
2.723
1.340
3,875
1,763
4,028
1 1,688
2.544
2.591
96
0.484
0,465
0,481
0.614
0.768
0.792
1.433
0.945
2.762
1.542
ETTml
288
168
0.505
0.520
0.492
0.477
0.634
0.573
0.528
0.505
0.748
1,056
0.789
0.634
0.886
1,462
0.759
1,320
0.915
1,830
0.696
1,094
1.124
1.257
0.897
2,076
336
1.786
1.598
96
01.446
0.320
0.255
0.339
0.365
0,453
0.768
0.642
0.658
0.619
3.142
1.365
ETTm2
288
168
0.328
0.259
0.365
0.312
0.342
0.2774
0.338
0.378
0.681
1,047
0.647
0.804
1,090
1,291
0.806
1,173
2.441
1.255
0.858
1.190
2.856
1.502
1.329
1.328
336
1,438
0.921
2,006
1.334
2.213
1.104
3,259
1,576
24
0.156
0.237
0.182
0.265
0.213
0.287
0.365
0.405
0,176
0.248
0.575
0,507
Weather
96
48
0.220
0.264
0.228
0.306
0.335
0.387
0.458
0.196
0.185
0.490
0.284
0.344
0.596
0.622
0.594
0.553
0.587
168
0.263
0.327
0.319
0.376
0.129
0.465
0.649
0.573
0.423
0.451
0.676
0.585
96
0.193
0.307
0,201
0.317
0.274
0.368
0.258
0.357
0.312
0.402
0.680
0,615
168
0.318
0.368
2588
0.231
0.339
0.234
0.343
0.315
0,410
0.319
0.425
0.353
0.437
0.481
0.612
336
0.266
0,358
0,270
0.361
0.300
0.394
0.280
0,380
0,350
0:133
0.828
0.727
F1-score is deemed as comprehensive indicator since balances precision and recall. We extensively compare model with 14 maaselines.includin the classic methods: OC-SVM (Tax & Duin, 2004).
IsolationForest (Liu et al. (2008). LOF (Breunig et al. (2000); recurrent structure methods: LSTM
(Hundman et al. 2018). THOC (Shen et al. 2020); probabilistic dynamic models: LSTM- PPar
ctt al. 2018)) VRNN (Chung et al. 2015).
DOMI (Su et al. 2021). BeatGAN (Zhou et af.] 2019).
Omn (Su et 2019), SDEVAE
(Dai al. (021): methods considering cross-channel
dependency: Interfusion (Li et al.
2021). GNN (Deng & Hooi| 202T); and Transformer based
methods: Transformer (Zerveas al. 202T). Anomaly-Trans etallal 2022). Anomaly-Trans is the
state-of-the-as method based on Transformer structure.
Firstly, we compare the proposed models with baselines on their detection performance and report the average Fl Fl-score results on five independent runs in Table the best results are highlighted in boldface. It is obvious that deep learning based methods outperform the classic methods for stronger representational power. Probabilistic dynamic methods achieve better results than LSTM, since they consider the stochasticity within MTS. Both recurrent and graph structures can boost the performance, indicating the effectiveness of temporal and cross-channel relationships on learning normal patterns of MTS. Original Transformer is not suitable for MTS modeling, leading to a poor performance, while Anomaly- -Tans achieves SOTA results before models, showing the efficient of Transformer structure in modeling long and complex temporal dependencies. Our proposed VG-Trans achieves the best detection performance among all methods most datasets, which demonstrates its effectiveness in modeling don-deteriiiistic temporal and cross-channel dependencies, thus to learn more expressive representations of the normal pattern of MTS. Finally, the performance of VG-trans can be further improved by combining it with Anomaly- - for VG-Anomaly-Tee -rann...
To better demonstrate the effectiveness of capturing the channel-wise relationships of VG-Trans, we further visualize some channels of xn, in Fig. and presents a subset of the corresponding relational matrices to these channels in Fig. (middle). As we can see, relational matrices can effectively reflect the similarity and correlation between channels, such as channels 21, 23, 25 of æ,,, which illustrates the capacity of VG-Tram in capturing the cross-channel dependencies within MTS.
6.2 FORECASTING
We deploy two widely used metrics, Mean Absolute Error (MAE) and Mean Square Error (MSE) (Zhou et al., (02T). to measure the performance of forecasting models. Six popular methods are compared here, including recurrent structure models: LSTnet (Lai et al. 2018). LSTMa (Bahdanau
7

---

In addition to quantitative evaluation, we also list the case study of prediction results by autoformer and VG-autoformer in Fig As shown Fig (a), similar with anomaly detection task, relational matrices in forecasting task can also effectively reflect the similarity and correlation between channels, Under review as a conference paper at ICLR 2023
0.00
100
200
300
100
200
200
(a) Input MTS of Th (Top) and corresponding relational mainx (Bottom)
(b) Predictions on charnel by autoformer (Top) and V-C
(c) Predictions on the 2-st channel by autoformer (Top) and VG- autoformer (Bottom)
(d) Predictions on the channel by autoformer (Top) and V C-C
autoformer (Bottom)
autoformer (Bottom)
Figure 5: Visualizations ETThl dataset of (a) the input MTS under the input- 200 setting and the learned relational matrix among different channels; (b), (c), (d): predictions on different channels by autoformer and VG-autoformer.
al. 2015); graph structure model: AGCRN Bai et al. 2020): Transformer based models: Reformer (Kitaey et al. 2019), LogTrans (Li et al. 2019), Informer (Zhou et al. (2021) and Autoformer (Kingma
& Welling 2014). We note that the experiment settings used here same with Zhou et al. (2021).
presents the overall prediction performance, which are aver- age MAE and MSE on five independent runs, and the best results highlighted in boldface. As we can see, RNN based methods under- perform Transformer based methods on most setting, especially on long-term forecasting, suggesting the efficiency of Transformer struc- ture modeling complex and long-distance temporal dependencies. By incorporating graph convolutional structure into network, AGCRN outperforms other RNN based methods, demonstrating the importance of modeling cross- cross-channel dependency explicitly. Combining VG- Trans and autoformer, VG-Autoformer achieves much better results on almost all datasets, suggesting efficient of VG-Trans in capturing the -onddmteriiiistcc complex temporal dependencies and complex cross-channel relationships MTS, which helps to get robust representations, thus ensuring the promising predictions. Besides, also test the effect of embedding size on forecasting performance with Weather dataset, as shown in Fig. 6 we set d = 4, 8, 16, 24, 32 and test the MAE of VG-Trans with different d. We can find that the selection of embedding size is affected by the length of horizons, higher embedding size performs better on longer horizons for more complex channel-wise relationships.
0.4
0.35
MAE
0.3
0.2
24
18
168
Timestep
Figure 6: The influcnce of bedding size on forecasting.
such as channels 1,2 and 3. By the aid of the relationships between different channels captured by VG-autoformer, the prediction results of the related channels can guide and revise each other, thus to ensure the forecasting performance. As illustrated in red boxes Fig 5 (b), (c) and (d), VG-autoformer achieves more accurate prediction than autoformer.
6.3 ABLATION STUDY
We conduct ablation study to analyze the importance of each component in our model, including graph structure, variational scheme, embedding guided generation and the combined optimization objective. The results on both anomaly detection and forecasting tasks are listed in Table 3 Firstly, on both tasks, all the structural components contribute to the performance of the framework. Specifically, by introducing powerful variational generative module for Transformer graph Transformer, their performance gain a significant improvement for being able to get robust representation of MTS with complex distributions. Meanwhile, as shown in the last two lines, incorporating cross-channel
8

---

Under review as a conference paper at ICLR 2023
Auummd Scare
Auaraty Score
-20
Anonsdy
-20
-60
60
Time
Time
Time
Figure 7: Case study of anomaly scores by Transformer (left), G-Trans (middle) and VG-Trans (right) on machine 2-3 of SMD dataset. Regions highlighted in red and blue represent the groundtruth anomaly segments and misjudgement segments by methods, red lines refer the threshold selected according to the rule that all anomalies can be detected.
Table 3: Ablation study of VG-Trans on anomaly detection and forecasting tasks.
Variational Embedding/
Optimization
Anomaly Detection
Forecasting
Architecture
Generation
Guided Generation
Ohjective
CDN
SMD
MSL.
SMAP
ETTh1
ETTh2
ETTml
F1-score Fl-score Fl-score Fl-score]
168
288
168
288
168
288
Prediction
0.830
0.797
0.834
0.700
[0.490
0.514
0.457
0.470
0.573
0.634
Transformer
Reconstruction
0.907
0.911
0.922
Combined
0.922
0.920
0.921
0.937
X
X
Reconstruction
0.910
0.907
0.908
0.923
Graph Transformer
Prediction
0.908
0.478
0.509
0.448 0.465 0.547 0.602
Combined
0.919
0.917
0.920
0.931
Combined
0.940
0.928
0.935
0.949
10.470
0.506
0.441 0.463 0.537 0.589
Combined
0.932
0.944
0.955
0.445
0.502
0.459 0.505 0.520
dependency modeling into generative process can further improve the generative capacity of models, thus resulting better performance on two tasks. In addition, by comparing the 1-st and 5-th lines, the 3-rd and 6-th lines, the efficiency of graph structure can be proved. The combined optimization objective is also beneficial to learn expressive representation of MTS as shown in the 5-th and 7-th lines. These verify that each module of our design is effective and necessary.
To further show the efficiency of different components of VG-Trans in capturing the robust represen- tations of MTS intuitively, focusing on anomaly detection task, we visualize the anomaly score of the case study. We compare the anomaly scores by Transformer, G-Trans and VG-Trans, the results are visualized in Fig. 7 As the deterministic methods, Trans and G-Trans get turbulent anomaly scores since they ignore the stochastic of MTS. Considering the inter-relationssiip within MTS, G-Trans exhibits more distinct spikes in the regions of anomalies, thus leading to less regions misjudged to anomalies. As probabilistic methods, VG-Trans realizes smoother anomaly scores than previous methods for considering stochasticity within MTS. Meanwhile, it exhibits considerable spikes in the regions of anomalies for considering both temporal and cross-channel dependencies. These properties enable that only one region is misjudged by VG-Trans, thus leading to better Fl-score. The anomaly score of this case further demonstrates the capability of graph structure and powerful probabilistic generative process in VG-Trans in learning expressive representations of complex MTS, which echoes the numerical results listed in Table 3
7 CONCLUSION
In this paper, towards MTS modeling tasks, we propose novel variational dynamic model named VG-Trans, which consists of G-Trans module that can capture both cross-channel and long-distance temporal dependencies by incorporating graph into Transformer, and an embedding guided probabilis- tic generative module to consider the stochasticity within MTS and enhance the capacity of modeling MTS with complex distribution. To achieve efficient optimization, we introduce autoencoding variational inference scheme with a combined prediction and reconstruction loss. VG- "G-Trans is able to get the robust representations of MTS, which enables it to be combined with the existed methods and applied to different tasks of MTS. Experiments on both anomaly detection and forecasting tasks of MTS illustrate the effectiveness of our model in both extracting expressive representations of MTSS

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. in ICLR, 2015.
Lei Bai, Lina Yao, Salil Kanhere, Xianzhi Wang, Quan Sheng, et al. Stg2seq: Spatial- graph to sequence model for multi-step passenger demand forecasting. in AAAI, 2019a.
Lei Bai, Lina Yao, Salil S Kanhere, Zheng Yang, Jing Chu, and Xianzhi Wang. Passenger demand forecasting with multi-task convolutional recurrent neural networks. in PAKDD, pp. 29-42, 2019b.
Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. NeurlPS, 13:17804-17815 2020.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. in ICML, pp. 1613-1622, 2015.
Markus M Breunig, Hans-Peter Kriegel, Raymond Ng, and Jörg Sander. Lof: identifying density- based local outliers. in SIGMOD, pp. 93 104, 2000.
Defu Cao, Yujing Wang. Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate seme-s forecasting. in NeurIPS, 33:17766- 17778, 2020.
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. in CVPR, pp. 1270-12280. 2021.
Wenchao Chen, Bo Chen, Yicheng Liu, Qianru Zhao, and Mingyuan Zhou. Switching poisson gamma dynamical systems. in IJCAI, 2020.
Wenchao Chen, Bo Chen, Yicheng Liu, Chaojie Wang. Mingyuan Zhou, and Hongwei Liu. Infinite switching dynamic probabilistic network with bayesian nonparametric learning. in TSP, 2022.
Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Doctor ai: Predicting clinical events via recurrent neural networks. pp. 301-318, 2016a.
Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. in NeurlPS, 29, 2016b.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. Advances in neural information processing systems, 28, 2015.
Liang Dai, Tao Lin, Chang Liu, Bo Jiang, Yanwei Liu, Zhen Xu, and Zhi-Li Zhang. Sdfvae: Static and dynamic factorized vae for anomaly detection of multivariate cdn kpis. in WWW, pp. 3076-3086, 2021.
Liang Dai, Wenchao Chen, Yanwei Liu, Antonios Argyriou, Chang Liu, Tao Lin, Penghui Wang. Zhen Xu, and Bo Chen. Switching gaussian mixture variational mn for anomaly detection of diverse edn websites. in INFOCOM, 2022.
Ailin Deng and Bryan Hooi. Graph neural network-basee anomaly detection in multivariate time series. in AAAI, 35(5):4027-4033 2021.
Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Transac- fions of the Association for Computational Linguisties, 8:439 2020.
Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformen model for speech recognition. in ICASSP. pp. 5884-5888, 2018.
no-recurrence sequence-to-sequene
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. in ICLR, 2021.
10

---

Under review as a conference paper at ICLR 2023
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. in NeurIPS, 29, 2016.
Yuechun Gu, Da Yan, Sibo Yan, and Zhe Jiang. Price forecast with high-frequency finance data: An autoregressive recurrent neural network model with technical indicators. pp. 2485-2492, 2020.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. AAAI, 33(01):922-929, 2019.
Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using Istms and nonparametric dynamic thresholding. pp. 387-395, 2018.
Tony Jebara, Risi Kondor, and Andrew Howard. Probability product kemels. in JMLR, 18819-84,, 2004.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. in ICLR, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. in ICLR, 2014.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. in ICLR, 2019.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. pp. 95-104, 2018.
Shiyang Li. Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. in NeurIPS, 32, 2019.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. ICLR, 2018.
Zhihan Li, Youjian Zhao, Jiaqi Han, Ya Su, Rui Jiao, Xidao Wen, and Dan Pei. Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. in SIGKDD, pp. 3220-3230, 2021.
Bryan Lim, Sercan Ö Ank, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. in International Journal of Forecasting, 37(4): 1748-1764, 2021.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hue Zhou. Isolation forest. in ICDM, pp. 413-422, 2008.
Iwao Maeda, Hiroyasu Matsushima, Hiroki Sakaji, Kiyoshi Izumi, David deGraw, Atsuo Kato, and Michiharu Kitano. Effectiveness uncertainty considcration in noural-network-beee financial forecasting, in AAAI, pp. 673-678, 2019.
Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. Lstm-based encoder-decoder for multi-sensor anomaly detection. in ICMI, 2016.
Hengyu Meng, Yuxuan Zhang, Yuanxiang Li, and Honghua Zhao. Spacecraft anomaly detection via transformer reconstruction error. in ICASSE, pp. 351-362, 2019.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard I. Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. in NeurIPS, 28, 2015.
Zheyi Pan, Yuxuan Liang, Weifeng Wang, Yong Yu, Yu Zheng, and Junbo Zhang. Urban traffic prediction from spatio-temporal data using deep meta learning. in SIGKDD, pp. 1720-1730, 2019.
Dachyung Park, Yuuna Hoshi, and Charles C Kemp. A multimodal anomaly detector for robot- assisted feeding using an based variational autoencoder. IEEE Robotics and Automation Letters, 1534:1544455111 2018.
11

---

Under review as a conference paper at ICLR 2023
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. in International Journal of Forecasting, 36(3): 1181-1191, 2020. 2020.
Lifeng Shen, Zhuocong Li, and James Kwok. Timeseries anomaly detection using temporal hierar- chical one-class network. in NeurIPS, 13:13016-33006 2020.
Alban Siffer, Pierre-Alain Fouque, Alexandre Termier, and Christine Largouet. Anomaly detection in streams with extreme value theory. in SIGKDD, pp. 1067-1075, 2017.
Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. in SIGKDD, pp. 2828-2837, 2019.
Ya Su, Youjian Zhao, Ming Sun, Shenglin Zhang, Xidao Wen, Yongsu Zhang, Xian Liu, Xiaozhou Liu, Junliang Tang, Wenfei Wu, et al. Detecting outlier machine instances through gaussian mixture variational autoencoder with one dimensional cnn. in IEEE Transactions on Computers, 2021.
Binh Tang and David S Matteson. Probabilistic transformer for time series analysis. in NeurlPS, 34: 2021.
Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Charu Aggarwal, Prasenjit Mitra, and Suhang Wang. Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values. in AAAI, 34(04))5956-5940 2020.
David MJ Tax and Robert PW Duin. Support vector data description. Muchine learning, 54(1): 45-66, 2004.
Shreshth Tuli, Giuliano Casale, and Nicholas Jennings. Tranad: Deep transformer networks for anomaly detection in multivariate time series data. in VLDB, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. in NeurlPS, 30, 2017.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma. Junchi Yan, and Liang Sun. Transformers in time series: A survey. in arXiv preprint arXiv:2202.07125, 2022.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlatio for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22430 2021.
Sifan Wu, Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse transformer for time series forecasting. in NeurIPS, 33:17105-171155 2020.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. in AAAI, 2019.
Jiehui Xu, Jianmin Wang, Mingsheng Long, et al. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. in NeurlPS, 34, 2021.
Jichui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. in ICLR, 2022.
Huaxiu Yao, Fei Wu, Jintao Ke, Xianfeng Tang, Yitian Jia, Siyu Lu, Pinghua Gong, Jieping Ye, and Zhenhui Li. Deep multi-view -empaal-emppoal network for taxi demand prediction. in AAAI, 32(1), 2018.
Bing Yu, Haoteng Yin, and Zhanxing 7hu. Spatio-temporal graph convolutional networks: deep learning framework for traffic forecasting. in IJCAI, 2018.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer- based framework for multivariate time series representation learning. in SIGKDD, pp. 2114-2124, 2021.
12

---

Under review as a conference paper at ICLR 2023
Chuxu Zhang, Dongjin Song. Yuncong Chen, Xinyang Feng, Cristian Lumezanu, Wei Cheng, Jingchao Ni, Bo Zong, Haifeng Chen, and Nitesh V Chawla. A deep neural network for unsuper- vised anomaly detection and diagnosis in multivariate time series data. in AAAI, 33011):1499-141 2019.
Hongwei Zhang, Yuanqing Xia, Tijin Yan, and Guiyang Liu. Unsupervised anomaly detection in multivariate time series through transformer-base variational autoencoder. in CCDC, pp. 281-286, 2021.
Hang Zhao, Yujing Wang, Juanyong Duan, Congrui Huang, Defu Cao, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Multivariate time-series anomaly detection via graph attention network. in ICDM, pp. 141- 2020.
Bin Zhou, Shenghua Liu, Bryan Hooi, Xueqi Cheng, and Jing Ye. Beatgan: Anomalous rhythm detection using adversarially generated time series. in IJCAI, pp. 4433-4439, 2019.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time-series forecasting. in AAAI, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Fre- quency enhanced decomposed transformer for long-term series forecasting, in arXiv preprint arX1v:2201.12740, 2022.
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 ALGORITHM
The autoencoding variational inference algorithm for VG-Trans is described in Algorithm. [
It is a great optimization challenge to train a dynamic VAE structured model in practice, due to the well-known posterior collapse and unbounded KL divergence in the objective. Here, we utilize two approaches for stabilizing the training.
Warm-up: The variational training criterion in Eq. equation 12 contains the likelihood term p(a, ( ) and the variational regularization term. During the early training stage, the varia- tional regularization term causes some of the latent units to become inactive before their learning useful representation ?. We solve this problem by first training the parameters only using the reconstruction error,and then adding the KI. loss gradually with temperature coefficient:
T C ΣEΣ ((E..) Imp (z zt,n) + [In (IT, -,))-1m)) ,In ( ( p(2 h
(12)
where 3 is increased from 0 to 1 during the first N training epochs.
Gradient clipping: Optimizing the unbounded KL loss often causes the sharp gradient during training, we address this by clipping gradient with large L2-norm above a certain threshold, which we set 20 all experiments. This technique can be easily implemented and allows networks to train smoothly.
A.2
THE DERIVATION OF ELBO
P (D w)
T
(Rtm 21,m, * (xT, 1urr-1,, a) (Ze, 1r- (...i.dddr.r den I W)
(Rem
2t, a) (xr. ha.r- a (ze, ( dæi.T
(Ze, (2t) hir--1)) (Zam 1.)) aeeen Iem) d2
(13)
T
p ( ( har P (zem hT-1n)) glan sin)
In
E (=t..)
T
Inp 2.,) + (In (xT, h1T-11)) pzr, h(-1,))) g (2e, Fen)
B
g (Ze, I(n) (2t,m he-1,
Σ E (((e
Inp(ren 24,a) + + (y [1T--10]]
A.3 THE NOTATION TABLE OF OUR PAPER
To better understand the proposed model, we summarize the notations used in this paper in Table
A.4 DATASETS
Anomaly detection: For anomaly detection task, we conduct extensive experiments on four datasets: one real-world dataset named CDN multivariate KPI dataset and three public datasets named SMD,
MSL and SMAP that were released by the works Su et al. (009)) and Hundman et al. 2018). respectively. The basic statistical information of datasets is reported in Table and Table 6 CDN multivariate KPIs dataset is collected from a large internet company China, and the dataset contains 12 websites monitored with 36 KPIs individually. These websites are different from each other in
14

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Autoencoding Variational Inference of VG-Trans
Set mini-batch size M, the number of convolutional filters K and hyperparameters: Initialize the parameters of inference networks Ω2 EPM v. G-Trans 0 and the channel embeddings a (0:1);
repeat
Randomly select a mini-batch of M MTS consist of T subsequences form a subset (ZiTs))=
from uniform distribution for sampling latent states (= T.M 1,n=1 and channel embbedings {a,))
Draw random noise (Ct,ntteI,ns {c M
Calculate V (S2, V; Et,ns C,, 0, a) according Eq. (8), and update parameters of inference
module Ω. EPM L G-trans module 0, as well as the channel embeddings a jointly;
until convergence
return global parameters (Ωt 0, o.).
Table 4: Notation table for our paper.
symbol
meaning
symbol
meaning statistics of a;
a,
ith channel embedding of inputs MTS
[4g. d;
s(u,v)
symmetric similarity function of and
Q K, Vi
query, key and value of the ith head (normalized) symmetric adjacent matrix (Convolutional) latent variable at each timestep
H.
output of the ith head
A, A
h. h
(dynamic) latent states
Pt. di.
statistics of zt.n
generated MTS at each timestep mapping matrix between lec. and
Mi. oin
statistics of æt..
8
hi, hg.
dynamic latent states of z and at
h.
concatenated dynamic latent states
Of, 0
output MSA block for and
O
output of MSA block
types of services, c.g., Video on Demand (VoD) or live streaming video, etc. Besides, for each website, KPIs span about one and a half months and are collected every 60 seconds. In our experiments, for each website, the first half of the KPIs are used for training, while the second half are used for testing. Note that the ground-trute truth anomalies in the test set of CDN have been confirmed by human operators. For the public datasets, Server Machine Dataset (SMD) that contains data from 28 server machines that are monitored by 38 KPIs individually. In addition, for each server machine, KPIs span about five weeks. Mars Science Laboratory (MSL) rover Dataset is also a real-world public and expert-labeled dataset from NASA Hundman et al. (2018) containing the data 27 entities each monitored by 55 metrics. Note that the other Soil Moisture Active Passive (SMAP) satellite dataset is also released NASA Hundman al. (2018), and both MSL and SMAP are collected from spacecraft where the first dimension is the value of telemetry channel, while the rest dimensions are command information that encoded as 0 or
real-world public dataset [Su et al. (2019)
For forecasting task, we use four datasets, including {ETTh1, ETTh2) and (ETTml, ETTm1], created in Zhou et al. (202)) based on Electricity Transformer Temperature (ETT), which is a crucial indicator in the electric power long-term deployment. ETT used here is created by collecting 2 years data from two separated counties in China. To explorer the granularity on the LSTF problem, Zhou et al. (202I) creates separate datasets as [ETThI, ETTh2 for 1-hour level and (ETTm1, ETTm2] for 15-minutes-level Each data point consists of the target value "oil temperature" and power load features. The train/val/test is 12/4/4 months, which is the same setting as in Zhou et al. (202I). Weather is the dataset that contains local climatological data for nearly 1,600 U.S. locations, 4 years from 2010 to 2013, where data points are collected every hour. Each data point consists of the target value "wet bulb" and 11 climate features. The train/val/test is 28/1281/110 18/10/10 months. ECL (Electricity Consuming Load) dataset collects the electricity consumption (Kwh) of 321 clients. Due to the missing data, we use the data released by Zhou et al.) (202T) that converts the dataset into hourly consumption of 2 years and set "MT 320' as the target value. The train/val/test is 15/3/4 months.
Forecasting:
15

---

Under review as a conference paper at ICLR 2023
Table 5: Basic statistics of anomaly detection datasets
Anomaly Detection
Statisties
CDN
SMD
MSL
SMAP
Dimensions
12*36
28*38
27*55
$5*25
Granularity (sec)
60
60
Training set size
344,843
708,405
58,317
135,181
Testing set size
344,844
708,420
73,729
427,617
Anomaly ratio (%)
3.44
4.16
10.72
13.13
Table 6: Basic statistics of forecasting datasets.
Forecasting
Statisties
ETT
ECL
Weather
96
168
288
336
96
168
288
336
24
48
96
168
Channels
321
21
Training size
8449
8377
8257
8209
18221
18149
18029
17981
36768
36744
36696
36624
Validation size
2785
2713
2593
2545
2537
2465
2345
2297
5247
5175
5103
Testing set size
2785
2713
2593
2545
5165
5093
4973
4925
10516
10492
10444
10372
ETT means the full benchmark of ETTh1, ETTh2, ETTml and ETTm2.
A.5 EXPERIMENT SETTING
Hyper-parameters: The hyper-parametert of data preprocessing are set as 20, o for anomaly detection and T 20, o = 0 for forecasting. We employ Adam optimizer with learning rate of 0.0002, batch size of 64, total training epochs of for all tasks. The balance factor y for anomaly detection and forecasting are set as 0.1 and 0.5, respectively. The length of forecasting is set to be 12, 24, 48, 96, and 168 in turn.
For anomaly detection experiment, the dimensions of channel embeddings for inputs 02 are all set as 256, that is d 256. The dimension of mapping matrix is set as 256 x 10, that K' = 10. We employ a deconvolutional neural network (DCNN) and CNN in generation and inference process separately, both CNN encoder and DCNN decoder are with convolutional layers, whose filters and strides are set according to the number of timesteps of the observed variables. Then, the latent variables and inputs are feed into MSA blocks, respectively. Thirdly, the aggregated dynamic information h. inputs z and latent variables can be calculated by AGCN blocks. And the architectures of MSA and AGCN are shown in Table. 7 where V represents the inputs' dimension, various from different datasets as discussed in A.6. Finally, employ well-developed Transformer architecture Li et al. (2019) to derive the dynamic latent states h. The specific architecture can found Table. [8] The head VG-Trans set as For forecasting experiment, select the hyperparameters, including embedding size, latent size, latent dimension and so on, with validation data.
Model architecture:
Table 7: MSA and AGCN blocks for inputs and latent variables
Layer
Size
Norm/Act.
Input
25/V
Linear Query
25/V
Linear Key
25/V
Identity Value
25/V
Lidear+Add
25/V
BN
GCN
using a
Linear+Add
25/V
16

---

Under review as a conference paper at ICLR 2023
Table Transformer for dynamic latent states
Layer
Size
Norm/Act.
Input
25+V
PosEmbd
256
Linear Query
25+V+256
Linear Key
25+V+256
Linear Value
Anear_Valuc
25+56256
Lidcar+Add
25+V+256
BN
GCN
using
Linear+Add
25+V+256
BN
Graph Transtomer
Transformer
Self- A Attention
Series Decomp
Graph Convolutional
Sectes Decomp
Feed Forward
Series Decomp
Decomp
Abeetion
series
Embedding- Guided
Channel Embedding
A utoformer Encoder
Generation
Embedding Incorpoating
Foed Forward
Latent Vartable
Aute- Comelation
Sertes Decomp
Feed Forward
Series Decomp
Series
Seriie Decomp
Feed Forwurd
Serfes Decomp
Emmissiing Incompraatngg
-
Decomp
Channel Embedding
Tmand eychoal Imt
Embedding Guided Autformmrr Decoder
Figure The detailed structure of VG-Autoformer.
A.6 MODEL. PROPERTIES
Comparing with previous Transformer based methods for modeling MTS, our proposed VG-Trans has the following properties:
I): Introducing Gaussian-distributee channel embeddings, modeling the stochasticity of different sensors and their dependencies in MTS;
2): Introducing graph network into transformer structure for capturing the diverse relationships between different channels within MTS;
3): Enhancing generalization capacity by incorporating channel embeddings into generative process;
4) Being optimized with both prediction and reconstruction losses, VG-Trans can explore both the periodic information and global distribution of MTS extracting expressive representations;
5): The prior of the latent variable in EPM is conditioned on the previous latent states of G-Trans for captureing temporal dependency, thus to help learning richer representations.
All of these properties ensure more expressive and robust representations of MTS and enhance generative capacity for complex MTS, thus achieves accurate detection and forecasting.
A.7 VARIATIONAL GRAPH AUTOFORMER
We claim that the proposed Variational Graph Transformer (VG-Trans) is a flexible framework having better compatibility with other methods, such as Anomaly-Tree -rans (please refer to results of Table in the text) and Autoformer Wu et al. (202T). In this section, we mainly discuss how to extend the VG- Trans with the help of Autoformer. the pipeline of which is illustrated in Fig.
17

---

Under review as a conference paper at ICLR 2023
148
-22
Figure 9: Prediction of simple MTS from the ETT, Weather and ECL datasets (from left to right). Top: results of Autoformer, Bottom: results of VG-Autoformer.
MY t
MV
230
Figure 10: Prediction cases of complex MTS from the ETT dataset. Top: results of Autoformer;
Bottom: results of VG-Autoformer.
Specifically, replace the standard Transformer with Autoformer encoder. Then add Autoformer decoder for forecasting task while keep the VG-Trans decoder and Graph Transformer unchanged. We find that is important keep the Seasonal Init and Trend-cycl ical Init t directly derived from the raw input Dn to maintain Autoformer's capability for forecasting. On the other hand, considering full ELBO and channel-wise relationships enhances the robustness of capturing long-term dependencies, thus achieves better prediction results, as shown in Table. 2 in which u/v represents the author reported result and our reproduced result v.
A.8 MORE VISUALIZATION RESULTS
We illustrate more prediction cases of simple and complex MTS on forecasting task in Fig. and Fig. 10, respectively. And the improvement areas are highlighted with red dotted boxes. Our model gives best performance on different datasets compared with Autoformer. Specifically, VG-Autoformer complements more missing details in Autoformer thanks to its design of capturing both temporal and channel dependencies the same time.
A.9 BALANCE BETWEEN PREDICTION AND RECONSTRUCTION
As discussed in Sec. B we combine prediction and reconstruction losses on VG- Trans, which enables models learn expressive representations efficiently, and introduce parameter y balance
18

---

Under review as a conference paper at ICLR 2023
0.96
CDN
SMD
0.94
Fl- F1-score
MSL
SMAL
0.92
0.9
0.88
0.01
0.1
0.5
5
10
Balance parameter
Figure 11: The influence of balance parameter 7 on anomaly detection task.
the effect of them. Here, we evaluate the influence of y to anomaly detection task, the results are reported in Fig. [1] As we can see, excessively small and large 7 will lead to weaker performance, illustrating the effectiveness of both reconstruction and prediction losses. Besides, relatively small weight for prediction loss is good for detection for exploring the global distribution of the MTS.
19


Reviews:
[{"review_id": "x2XDEAjXUf", "reviewer": "Reviewer_Z2ri", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an approach to model the time dependencies, cross-channel dependencies, and stochasticity for multivariate time series (MTS) modeling. Specifically, an adaptive graph Transformer is first designed to learn the cross-channel relationships in MTS. To model the stochasticity, a channel embedding guided probabilistic generative module is further proposed. Lastly, the authors propose an autoencoding inference scheme that combines prediction and reconstruction loss for better representation learning. Experiments on the time series forecasting and anomaly detection task demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "Given the strength and weaknesses mentioned above, this paper is below the level of ICLR and needs to be improved.", "strength_and_weaknesses": "Pros:\n1. The problem studied in this paper is interesting and practical.\n2. The paper is well-structured.\n3. The experiments verify the effectiveness of the method.\n\nCons:\n1. I suggest that the author should further clarify the first contribution (i.e., the G-Trans module). Since incorporating relationship learning between different time series have actually been explored by many works in the field of spatial-temporal forecasting. What’s the difference in the proposed G-Trans ? \n2. The term “dynamic” is mentioned many times in the paper, with no clear explanation. For example, in related works, “However, these methods are still all non-dynamic or RNN based models, limiting their power in capturing complex relationships across long-distance time steps”\n3. The authors claim that “VG-Trans is able to get the robust representation of MTS”. How to prove it?\n4. In Figure 2(c), the authors use Graph Convolution network to replace the first Feed Forward Network, by stating that GCN can better capture the diverse relationship than FFN. If so, why not use GCN to replace all FFN in Transformer? Will the result get better by doing so?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "vCXD1_v25c", "reviewer": "Reviewer_RixC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to combine graph structure with transformer to model multivariate time series, and use probabilistic modeling method and variational inference to remove random noise in time series. The experiments on the task of anomaly detection and time series forecasting show the effectiveness of the proposed method.", "summary_of_the_review": "This work is just the fusion and utilization of existing techniques to realize multivariate time series modeling. The innovation of the method is limited, the experimental design is not fine enough, and no enlightening insights are given.\n\n", "strength_and_weaknesses": "Strength: The organization of this manuscript is clear and it's easy to read. The proposed approach performs better in anomaly detection and time series forecasting tasks compared to some baselines.\n\nWeaknesses: \n\n--Some claims and motivations lack theoretical support or explanation. E.g., it is hard to understand that \"However, Transformers in general do not well capture the diverse relationships between different channels within MTS and have difficulty in modeling MTS with complex distributions due to the lack of stochasticity.\" and \"As probabilistic methods, VG-Trans realizes smoother anomaly scores than previous methods for considering stochasticity within MTS.\"\n\n--The novelty of the idea in this work is very limited, and the proposed method is not very insightful. It is a natural idea to use graph structure in time series, and many works have tried, such as \n[1] Graph neural network-based anomaly detection in multivariate time series, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2021.\n[2] MST-GAT: A multimodal spatial–temporal graph attention network for time series anomaly detection. Information Fusion 2023.\nJust changing RNN to Transformer is not inspiring.\n\n--The motivation for using probabilistic modeling and variational inference is also unclear, and no experimental results have shown the existence of stochasticity and the advantage of elimination it. \n\n--The experimental comparison is still superficial, and only focuses on whether the performance is improved, without in-depth analysis and experimental verification of the claimed assumptions about the advantage of modeling channel relationship and modeling stochasticity.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lOxemDZM8-F", "reviewer": "Reviewer_XSSh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a G-Trans module, which incorporates channel-relationship learning into the Transformer structure for multivariate time series modeling. The authors further added  a VAE-structured module to model the non-deterministic within both temporal\nand cross-channel dependencies of MTS. Experiments on both anomaly detection and forecasting tasks illustrate the efficiency of our\nmodel on MTS modeling.", "summary_of_the_review": "A novel approach on MTS by introducing graph structure and VAE encoder.", "strength_and_weaknesses": "Strengths:\n- The proposed idea looks novel to me, though I'm not very familiar with related work in this subfield.\n- The authors conducted thorough experiments and the results look competitive.\n\nWeaknesses:\n- It's better to include std in the experiments to help understand how significant the improvements are.\n- In Table 3, it is a bit confusing to me why the reconstruction loss leads to better results than prediction loss. Why is it the case?\n- Why there are some entries unavailable in Table 3?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject