Multivariate Time-series Imputation with Disentangled Temporal Representations
Authors: SHUAI LIU
Xiucheng Li
Gao Cong
Yile Chen
YUE JIANG
Venue: ICLR 2023 poster (2023)
Keywords: multivariate time-series imputation
disentangled representation

Abstract: Multivariate time series often faces the problem of missing value. Many time series imputation methods have been developed in the literature. However, these methods all rely on an entangled representation to model dynamics of time series, which may fail to fully exploit the multiple factors (e.g., periodic patterns) contained in the time series. Moreover, the entangled representation usually has no semantic meaning, and thus they often lack interpretability. In addition, many recent models are proposed to deal with the whole time series to capture cross-channel correlations and identify temporal dynamics, but they are not scalable to large-scale datasets. Different from existing approaches, we propose TIDER, a novel matrix factorization-based method with disentangled temporal representations that account for multiple factors, namely trend, seasonality, and local bias, to model complex dynamics. The learned disentanglement makes the imputation process more reliable and offers explainability for imputation results. Moreover, TIDER is scalable to large datasets. Empirical results show that our method not only outperforms existing approaches by notable margins on three real-world datasets, but also scales well to large datasets on which existing deep learning based methods struggle. Disentanglement validation experiments further demonstrate the robustness of our model in obtaining accurate and explainable disentangled components.

Published as a conference paper at ICLR 2023
MULTIVARIATE TIME-SERIES IMPUTATION WITH DIS- ENTANGLED TEMPORAL REPRESENTATIONS
SHUAI LIU
Xiucheng Li
School Computer Science and Engineering Nanyang Technological University 50 Nanyang Avenue, Singapore, 639798 wwt. edt..edu..s. SHUAI004 wtw.::00eee..t..eleeeeee
School of Computer Science and Technology Harbin Institute of Technology (Shenzhen) No.6, Pingshan 1st Road, Nanshan District, Shenzhen, Guangdong, China, 518055 1 1ix Lixiuchenge hit.edie cn
Gao Cong & Yile Chen & Yue Jiang
School of Computer Science and Engineering Nanyang Technological University 50 Nanyang Avenue, Singapore, 639798 gaocong @ntu.edu
ABSTRACT
Multivariate time series often faces the problem of missing value. Many time series imputation methods have been developed in literature. However, they all rely on an entangled representation model dynamics time series, which may fail to fully exploit the multiple factors (e.g.. periodic patterns) presented in the data. Moreover, the entangled representations usually have no semantic meaning, and thus they often lack interpretability. In addition, many recent models are proposed to deal with the whole time series to identify temporal dynamics, but they are not scalable to long time series. Different from existing approaches, we propose TIDER, novel method with disentangled temporal representations that
matrix factorization-batt account for multiple factors, namely trend, seasonality, and local bias, model complex dynamics. The learned disentanglement makes the imputation process more reliable and offers explainability for imputation results. Moreover, TIDER is scalable to long time series. Empirical results show that our method outperforms existing approaches
three typical real-world datasets, especially on long time series, reducing mean absolute error by up to 50%. It also scales well to long datasets on which existing deep learning based methods struggle. Disentanglement validation experiments further highlight the robustness and accuracy of our model.
INTRODUCTION
Multivariate time series analysis (e.g., forecasting (Zeng et al. 2021) and classification (Li et al. 2022) has wide spectrum of applications like traffic flow forecasting (Liu al. 2020), electricity demand prediction (Kaur et al. 202I). motion detection (Laddha et al. 2021). health monitor- ingeTonekaboni al. 2021), etc. Most of these multivariate time series analysis approaches typically assume intact input for building models. However, real-world multivariate time series tends to have missing values caused by factors like device malfunction, communication failure, costly measure- ment, leading to impaired performance of these approaches, or even rendering them inapplicable.
In light of this, many time series imputation methods have been proposed to infer missing values from the observed ones. A multivariate time series, denoted as X E RNxT consists of N univariate time series (called channels) spanning over time steps. Hence, it offers two perspectives for imputation: modeling cross-channel correlations and exploiting temporal dynamics. Earlier methods (Batista et al. 2002; Acuna Rodriguez] 2004 Box et al. 2015)) either aggregate observed entries across channels by estimating similarity between distinct channels, solely exploit local smoothness linear assumption in the same channel to fill in missing values. Since these methods lack the ability of modeling nonlinear dynamics and complex correlations, they may not perform well in practice.

---

Published as a conference paper at ICLR 2023
In recent years, deep learning based methods (Liu et al. 2019 Tashiro al. 2021 Cini al.
2022) were proposed for time series imputation. These methods typically employ Recurrent Neural Networks (RNNs) the backbone to jointly model nonlinear dynamics along temporal dimension (by updating hidden states nonlinearly over time steps) and cross-channel correlations (by mapping X, R.V from data space to hidden state space). These RNN-hased imputation models achieve better
results by further combining with multi-task loss (Cao et al. 22018] Cini et al. (022) or adversarial training (Liu et al. 2019; Miao et al. 202T). Despite their success, these methods rely solely on a single entangled representation (hidden state) to model dynamics. However, for many real-world multivariate time series, their dynamics are rich combinations of multiple independent factors like trend, seasonality (periodicity), and local idiosyncrasy (Woo et al. 2022). Modeling combinations of these factors with a single entangled representation may not give good performance, as the entangled representation has to compromise itself to explain multiple orthogonal patterns (like local changes or exogenous interventions vs global patterns) together (Bengio et al. 2013). Furthermore, it becomes exacerbated when seasonal patterns dominate, as RNNs lack the inductive bias to initiatively capture periodicity (Hewamalage et al. 202T). In addition, the hidden states that these models learned entangled and complex combination of various components. Thus it is difficult for them to provide interpretable information to explain imputation. Moreover, these methods require the entire time series to be fed in their models at each forward step, to capture temporal dynamies, This is prohibitively costly for large T. Therefore, these methods are not applicable to long datasets.
To address these limitations, in this paper, we propose a novel multivariate time series imputation method, Time-series Imputation with Disentangled tEmpraal Representations (TIDER), in which we explicitly model the complex dynamics of multivariate time series with disentangled representations to account for different factors. We employ low-rank matrix decompasition framework, and achieve the disentanglement by imposing different forms of constraints on different representations which compose the low-rank matrix. In particular, we introduce a weighhoringsmoothheee matrix to explain the trend, a Fourier series-based representation matrix to define periodic inductive bias, and local bias representation matrix to capture idiosyncrasies specific to individual time step. These disentangled representations offer greater flexibility and robustness to model the factors contributed to the dynamics in time series. Another notable benefit of these semantically meaningful disentangled representations is that they offer TIDER an interpretable perspective on imputation. Moreover, TIDER is scalable model. It can be applied to time-series datasets with large T and achieve much better imputation performance. In summary, our contributions are as follows.
representation
We propose TIDER, new multivariate time series imputation model, which is featured with effective and explainable disentangled representations to account for various factors that characterize the complex dynamics of time series. To the best of our knowledge, TIDER is the first model to learn disentangled representations for multivariate time series imputation.
TIDER is the first imputation model that introduces a learnable Fourier series-based repre- sentation to capture periodic patterns inherent in time series data.
Extensive experiments show that our proposed method outperforms the baseline imputation methods in terms of effectiveness and scalability. Especially, for imputation task in long time series, TIDER achieves more than 50% improvement in MAE, compared with the best imputation baseline approach. Furthermore, TIDER is a scalable model. It can easily handle long multivariate time series while existing deep-learning methods struggle. We also demonstrate the explanability of the disentangled representations with several case studies.
2 RELATED WORK
Early time series imputation methods based on simple statistical strategies focus on exploiting local smoothness of temporal dimension as well as the similarity between different channels. For example, SimpleMeanSSipleMedii
(Fung 2006) imputes missing values by averaging, and KNN (Batista al., 2002) aggregates cross-channel observations to fill in missing slots with k-nearest neighbors. Linear dynamics- based imputation methods, including linear imputation and state- models (Durbin & Koopman| 2012). have also been employed. MICE (Van Buuren & Groothuis Oudshoom] (0011) explores the idea of estimating missing slots with chain equations. These methods typically lack the ability of exploiting nonlinear dynamics and complex correlation across channels.
2

---

Published as a conference paper at ICLR 2023
1
D
Vi+Vs
X
N
U
1
X
Db
Vb
Do
J
Figure Architecture of the proposed method TIDER. X the multivariate time series matrix. U represents a correlation-dccoppeed matrix. V,, V.. and V, denote trend representation matrix, seasonality representation matrix and bias representation matrix, respectively.
recent trend is to apply deep learning, particularly RNNs, to better capture the nonlinearity. BRITS (Cao et al. 2018) treats missing values variables which participate in bidirectional RNN pack-propagation and are updated with hidden states. GAIN (Yoon et al. 2018) combines Generative Adversarial Nets (GAN) and RNNS for imputation. NAOMI (Liu al. 2019) merges RNNs and adversarial training strategies. SSGAN (Miao et al. 2021 is semi- supe-super GAN-based model which uses label information. GRIN (Cini et al., 2022) fuses graph message passing into GRU structure to learn spatial-tempont patterns. All these methods model dynamics based on a single entangled representation, which is insufficient to capture multiple factors underlying time series, especially when seasonality emerges, since RNNs lack the inductive bias to initiatively capture periodicity (Hewamalage et al. 2019). Morcover, these approaches are not scalable to long datasets since they have to process time series of whole length 7 at each forward step to capture temporal dynamics. Thus, they do not perform well, or are even not applicable to long time-series data.
Our proposed TIDER is based on low-rank matrix factorization (MF) Yu et al. 2016 Bjorck et al. 2021). Vanilla MF-based models impute missing entries by learning latent factors U. to exploit low-rank structure. However, they overlook the temporal continuity. To address this, TRMF (Yu et al. 2016) imposes autoregressive constraints on temporal factor V. However, similar to aforementioned methods, TRMF also relies entangled representation to account for all factors underlying the dynamics. In contrast, TIDER introduces multiple disentangled representations. It achieves the disentanglement by enforcing different forms of constraints into different representations.
PROBLEM STATEMENT
Given univariate time series data X1, X2,..., XN E T collected over time steps, we represent it with multivariate time series matrix X E RNT whose n-th row represents n-th univariate time series (channel) x, and f-th column denotes the observation of all time series at time step t. The multivariate time series matrix X is incomplete and a fraction of entries are missed. We aim infer missing values from the observed ones, and we denote the mask matrix as M E {0, 10,1 where
if Xı is observed, 11, {o. 0, otherwise.
Mij =
(1)
METHODOLOGY
4.1 METHOD OVERVIEW
The core idea of TIDER decompose multivariate time series X into two latent factors and V, such that U only preserves features unique to each channel whereas V is determined by multiple disentangled representations that jointly capture temporal dynamics. We employ such factorization for two main reasons: 1) univariate time series channels (rows of X) are usually highly correlated; 2) observations at different time steps (columns of X) exhibit strong temporal dynamics. The benefits of our design twofold: 1) cross- chasscchannel correlations channel- -sunnee- features 2) time-related information is isolated into V. and it enables us to model the potentially complex temporal dynamics with multiple explainable disentangled representations.
decoupled since U only preserves
3

---

Published as a conference paper at ICLR 2023
Decompesition additive time series
20-
10-
1e/o0
(d)
Figure 2:
Visualization of time series decomposition. (a) is the raw time series, whereas (b), (c), and (d) are its decomposed trend, seasonality, and residual component. The grey bars on the right of each subplot show the relative scales of the components. Each grey bar represents the same length.
Figure [ shows the architecture of TIDER. We adopt low-rank matrix decomposition framework to factorize multivariate time series matrix X into two latent factors U, V. U is a correlation- decoupled matrix which accounts for channel-specifie patterns, while V is matrix for time-related information. Since temporal dynamics underlying real-world time series can be rich and complex combinations of multiple factors, e.g., trend and seasonality, modeling them merely through an entangled representation matrix will lead to model degradation. Woo et al. (2022) proposes that under mild assumptions, seasonality and trend can be treated as independent factors in time series generating process, [ohen (2013) suggests that independence can be used as proxy criterion for disentanglement. Inspired by them, we propose to model V with multiple disentangled repre- sentations, each accounting for one particular factor. We achieve the disentanglement by enforcing distinct forms of constraints on different representations, which introduce distinct inductive biases into these representations and make them more liable to capture specific patterns. More specifically, we consider three important factors: trend, seasonality, and bias, which specified by three representation matrix V,, V.. Vo, respectively. Figure illustrates example of decomposing a time series into these three factors. Trend representation matrix V, E R DXT captures the intrinsic trend which changes gradually and smoothly, and seasonality representation matrix V. E DxT illustrates the periodic patterns hidden in temporal dynamics. V. and V. jointly determine the dynamics driven by endogenous factors. Bias representation matrix Vo IR Px characterizes variations specific to each time step. Intuitively, Vo explains the individual idiosyncratic behaviors time steps, which are orthogonal to global dynamics but shared across channels at given time step. Hence, we treat it differently from V., V. and interpret it as residual term matrix, ie.. X U(V++ + V.) x INDD, V,, Mathematically, we formulate the objective of TIDER as
minimize ||(X - U.V) O MI \V.. + Avfo(V&) + | V
U. = U 1 Nx(++D)) v + V
E (D++D.XXT
(2)
where U. is the augmented matrix of latent factor NXD ft and fo are corresponding inductive bias constraint functions imposed on V, and V,, M is the mask matrix introduced in Sec | m2ll 2 are used to regularize the magnitude of latent factors, and dr. Ab, 7/1 7/2 are corresponding weights for each term. When training is completed, we use the learned U to get U.. and the learnt V,, V.. and V& to form V. U. and V are then used to generate the imputed time series X as
1-=(Xan Xiji
M,j 1, My = 0.
X. --- ((..).)
(3)
We are now ready elaborate on the details of the three disentangled representation matrices.
4.2 TREND REPRESENTATION MATRIX
Trend representation matrix V, characterizes the intrinsic trend of time series. The evolution patterns dominated by V, are supposed to change gradually and smoothly in the absence of accidents
4

---

Published as a conference paper at ICLR 2023
extreme events such as holidays (we will consider these external interventions and exogenous influences in Vo). Based on this, we impose a smoothness constraint on Ve as
f((V.) Σ liv J=2
where v! is the j-th column of v,. Equation encourages close representations of adjacent time steps in latent space, which will result in smooth change in data space. We only impose constraints on two consecutive time steps account for short-term patterns here whereas long-term patterns are explained by V.. This is in contrast with TRMF (Yu et al. 2016) which uses one temporal matrix to account for both short-term and long-term patterns by imposing regression constraint.
4.3 SEASONALITY REPRESENTATION MATRIX
Real-world time series often demonstrate seasonal patterns. For instance, traffic flow exhibits strong daily and weekly seasonality due to regularity of human activities, such as commuting patterns. Solar power production presents clear periodic characteristics caused by climate seasonality and metcorological conditions. Motivated by this and Fourier analysis Section 1.2 we propose to model the seasonality of time series by parameterizing representation matrix V. with Fourier basis.
V. is a matrix with size D T, we represent each row with a superposition 2K sinusoidal waves (K < T). More formally, let A, B Px be two learnable coefficient matrices, and Dsin- Pcos E IR TxX be the corresponding Fourier basis matrices, which are defined as
sin ).
A
a1
a2
aK
Øsin =
sin(lwt) sin()
(5)
B
bj
b2
bx
Ocos
=
cos(1wst) cos(2wt)
cos(Kwt)
(6)
where t
[1....,TT
TJT For one specific time-series with period P. is calculated 2T/P.
Then the seasonality representation matrix V. can be defined as
V A A + BΦ
(7)
In other words, V. is spanned by Fourier basis Φsin and Pece. In particular, the d-th row of V.. denoted by (v?)² has the form
Σ Aas sin(kwtt) + Σ Ba cos(kwt)
8
which is truncated Fourier series with coefficients A d. Ba This elaborate design of V, provides meaningful periodic inductive bias, which enables our model to capture seasonal patterns more accurately and effectively, by learning coefficient matrices A, B from data.
4.4 BIAS TEMPORAL REPRESENTATION MATRIX
The representation matrices V., V. presented so far jointly determine the dynamics driven by endogenous factors of multivariate time series. However, there are also various external factors (e.g., holidass, weekdays/weekends) that could affect real-world time series. These external factors usually occur specific time points and yield local variations within short time period, thus they independent of endogenous dynamics and cannot be captured by V, and V,. And also, these factors impact nearly equally all channels. To account for these local variations, inspired by the idea of user and item bias explored in collaborative filtering (Lü et al.) 2012), we propose to learn another bias representation matrix V, E IR DxxT, where the representation of specific time step shared by all channels. In addition, we impose an autoregressive constraint on Vü in temporal dimension since the impact caused by local variation usually lasts for short duration. Let v; be the t-th column of Va and L be the maximum time lag indicating duration, we define the constraint function as follows,
fo(Vv)
Σ
W Wiv
t=++1
5

---

Published as a conference paper at ICLR 2023
where W (Wi R DsXDs -== 1... L} is group of learnable parameters. In our setting, Do and L are small numbers, thus group W only incurs very few extra parameters.
4.5 ADAPTIVE WEIGHT FOR TREND AND SEASONALITY
In Section 4.1 we use the additive form Vr V. to characterize the influence of endogenous impacting factors. This implicitly assumes that trend and seasonality components contribute equally to endogenous dynamics. But in practice the importance of trend and seasonality can vary drastically across data sources, which is illustrated in Figure 2 To address this, we adopt learnable parameter E (0, 1) to adaptively adjust the weight for these two components, which leads to a weighted additive form a V, (1-a) V.... The temporal matrix V in Equation 2 then becomes:
a + (1 (1-a)V,
V
(10)
Vo
5 EXPERIMENTS
In this section, we evaluate the performance of TIDER by comparing with existing multivariate time series imputation methods, terms of imputation accuracy and scalability. We also show the explanability of TIDER with several case studies. Hyperparameter sensitivity experiments are included well to show that TIDER performs steadily under different hyperparameter settings. The code of TIDER is available at https: http:///gt.aaaacoa//////////////
5.1 EXPERIMENTAL SETUP
Baseline Methods We compare our model with popular baselines used in the literature and recently- proposed methods, including statistical models (S333 impleMean, KNN. MICE), MF-based methods (MF, MF+L2, Softimpute, TRMF), and deep learning approaches (BRITS, GAIN, NAOMI, Sin- gleRes, SAITS, CSDI). Details and settings of these baselines can be found in Section A.3 In addition, we also include TIDER (no W), variant of TIDER without the learnable parameter to verify the effectiveness of the adaptive weight introduced in Section 4.5
Datasets We use three typical real-world datasets. Guangzhou is a small one where all methods can fit in. Solar-energy shares along time-span while Westminster one with large N. These datasets represent three different types of multivariate time series (small, large T, large N), and these three types can cover most of the multivariate time series. For more details, please refer to Section A.2
Evaluation Metries We adopt RMSF, MAE, and MAPE to evaluate the imputation accuracy of all compared methods. Details of these three metrics can be found in Section A.4
Training Setup We randomly remove a subset of entries from X as validation and test datasets separately. Let r be the missing rate variable, the ratio of For each model over these runs. All experiments are conducted on a Linux workstation with a 32GB Tesla V100 GPU. For more detailed hyperparameter settings, please refer Table 7
(0.9 - r)/0.1// //r experiments 7 rounds every dataset and report the imputed results averaged
5.2 IMPUTATION ACCURACY COMPARISON
Table and Table 2 show the imputation accuracy of all methods on three datasets with different missing rate OOM indicates out of memory. The meaning of asterisk and improvement are illustrated in Section A.3 Our proposed TIDER achieves the best performance in most cases in terms of the three metrics. The superiority of TIDER is much more significant in Solar-energy, the dataset with large T'. In addition, obtain similar improvements other long time series. We report experiments on another two datasets Section A.5 addition, we observe that the imputation accuracy of most methods drops missing rate increases, which is expected since fewer patterns available.
Among all baseline methods, CSDI has the best performance on most datasets, and deep-learning methods (BRITS, GAIN, SAITS, CSDI) usually perform better than other baselines. However, TIDER outperforms them on each dataset. Especially on the solar-energy dataset, where time span T' is 52, 560 and many deep learning methods run into COM 32GB-memory GPU, TIDER is

---

Published as a conference paper at ICLR 2023
Table 1: Imputation accuracy of different methods with missing rate 0.2
Method
Guangzhou MAE
Solar-energy MAE
Westminster
(r = 0.2)
RMSE
MAPE
RMSE
МАPE
RMSE
MAE
МАРE
SimpleMean
10.24
8.023
0.396
3.352
2,433
4.527
5.233
4.247
0.410
KNN
6.540
4.731
0.242
2.701
1.748
2.324
2.249
1.611
0.147
MF
5.642
4.301
0.188
3.769
2.447
4.243
2.275
1.641
0.137
MF+L2
5,400
4.074
0.177
2.761
1.918
2.760
2.109
1.523
0.134
Softlmpute
5,876
4.465
0.217
2.692
1.903
3.051
2.347
1.645
0.141
MICE
5.097
3.859
0.161
2.647
1.763
2.372
2.638
2.014
0.169
TRMF
4.563
3.407
0.148
3.212
2.010
2.937
1.903
1.370
0.119
BRITS
4.416
3.003
0.139
2.617
1.861
2.677
2.154
1.488*
0.136*
GAIN
4.976
3,451
0.154
2.803
1.917
2.501
1.947
1,403
0.121
SAITS
4.407
3.025
0.140
OOM
OOM
OOM
1.893
1.366
0.119
CSDI
4.301
2.991
0.135
OOM
OOM
OOM
1.886
1.361
0.116
NAOMI
5.173
4.013
0.167
OOM
OOM
OOM
OOM
OOM
OOM
SingleRes
4.997
3.979
0.172
OOM
0OM
OOM
OOM
OOM
OOM
TIDER (no W)
4,431
3.229
0.142
1.872
0.893
2.522
1.981
1,426
0.
TIDER
4.168
3.098
0.132
1.676
0.874
2.227
1.867
1.354
0.115
Improvement (%)
3.092
2.222
35.96
50.00
4.174
1.007
0.514
0.862
Table 2: Imputation accuracy of different methods with missing rate 0.4
Method
Guangzhou MAE
Solar-energy MAE
Westminster
(r = 0.4)
RMSE
MАPE
RMSE
MAPE
RMSE
MAE
MAPE
SimpleMean
9.949
7.901
0.384
3.259
2.412
4.889
5.243
4.259
0.407
KNN
6.712
4.955
0.252
2.724
1.835
2.874
2.391
1.714
0.159
MF
7.671
5.844
0.246
4.111
2.887
5.851
2.476
1.807
0.153
MF+L2
7.324
5.553
0.235
2.617
1.80
2.977
2.157
1.593
0.134
SoftImpute
5.811
4.465
0.209
2.536
1.849
3.311
2.333
1.637
0.145
MICE
7.119
5.417
0.235
2.756
1.823
3.517
2.755
2.113
0.175
TRMF
6.119
4.617
0.198
3.407
2.199
3.586
2.014
1,453
0.126
BRITS
4.874
3.335
0.158
2.842
1.985
3.146
2.180
1.527*
0.138*
GAIN
5.550
3.671
0.192
2.639
1.883
3.117
2.337
1.711
0.145
SAITS
4.839
3.391
0.159
OOM
OOM
OOM
1.998
1.453
0.129
CSDI
4.813
3.202
0.157
OOM
OOM
OOM
1.982
1.437
0.124
NAOMI
5.986
4.543
0,222
OOM
OOM
OOM
OOM
OOM
OOM
SingleRes
6.051
4.705
0.252
OOM
OOM
QOM
OOM
OOM
QOM
TIDER (no W)
4.708
3,469
0.155
1.697
0.878
3.152
2.013
1.466
0.125
TIDER
4.764
3.527
0.152
1.679
0.838
2.735
1.959
1.422
0.121
Improvement (%)
2.182
3.185
33.79
54.03
4.836
1.160
1.055
2.419
still applicable and achieves the best performance by large margins. This could be attributed to our model's 's ability to model multiple factors in representation space in a disentangled way.
We observe that MF-based models can all work for long time series. The difference between these models lies in constraints on U and V. Compared with other MF-based baselines, TRMF outperforms others most cases, since intuitively captures auto-regressive dynamics. However, since TRMF employs merely entangled representation, it cannot model complex dynamics well. TIDER (no W) and TIDER utilize disentangled features like trend, seasonality and local bias to rebuild patterns for time-series, which exploit explanatory temporal factors, thus achieving better imputation result. Furthermore, TIDER performs better than TIDER (no W). This consistent with our previous analysis that trend and seasonality components might not contribute equally to global dynamics.

---

Published as a conference paper at ICLR 2023
TOEE
23000
Emparte5
Trawing TeReOo
MAC
18.00
1800
2000
2003
4006
5000
100
7:30
302
430
530
600
1E01
0001
SegiITES NAOVI
SATT
DITS
(c)
Figure 3: Scalability test on Westminster dataset: (a) Memory usage of different methods varies over N from 0 to 5000 with fixed 7 100; (b) Memory usage of different methods varies over T from 100 to 700 with fixed N = 100; (c) The average running time taken by different methods for every 100 epochs when processing 100 x 100 matrix.
Table 3: Ablation Analysis of TIDER on Guangzhou dataset.
Name
RMSE
MAE
MAPE
V, V
7.241 (7.497)
5.388 (5.459)
0.272 (0.294)
VI + Vo
5.680 (6.392)
4.311 (4.899)
0.190 (0.214)
V V.
5.297 (5.999)
4.014 (4.520)
0.170 (0,201)
TIDER
4.168 (4.764)
3.098 (3.527)
0.132 (0.152)
5.3 SCALABILITY ANALYSIS
We study the scalability of different methods in memory usage and training time. particular, we compare method with state-of-ther of-thh-art models, BRITS, NAOMI, SingleRes, SAITS, and CSDI. Figure (a) and Figure (bb) present the memory footprints of different methods against channel number N and time span T. It can be seen from Figure 3 (a) that memory usage of NAOMI and SingleRes grow rapidly with very steep slopes, and that of BRITS also grows quickly whereas TIDER needs much less memory. Similarly, (b shows that the memory usage of NAOMI, SingleRes, and SAITS also grows fast as T increases. Again, TIDER needs the least amount of memory. There is also gap between the curves of CSDI (BRITS) and TIDER in Figure 3 which is overwhelmed by the magnitude of NAOMI and SingleRes. We present these gaps in Section A.9 Lastly, Figure (c) demonstrates the running time of different methods for 100 epochs when processing 100 100 matrix. TIDER runs much faster than RNNs-based methods. Notably, TIDER outperforms BRITS by almost an order of magnitude. This also true for the total time taken by entire training process. Furthermore, the space complexity of BRITS is (N + +T) (Vaswani ct al. 2017) while that of TIDER is O(N +T). The complexity of NAOMI was not established and tricky to analyze due to its complex divide-and-conque strategy, but its complicated procedure yields more intermediate variables and thus results in more memory consumption, as empirically validated by experiments. In conclusion, this experiment demonstrates the scalability of our model.
5.4 ABLATION ANALYSIS
To verify the effectiveness of proposed disentangled representation matrices, we conduct ablation study on Guangzhou dataset by removing one of the trend, scasonality, and bias representation matrices while leaving the rest of the model unchanged. The ablative results with missing rate r = 0.2 (0.4) are presented in Table We find that the model performance drops no matter which representation is removed, which validates that all of our proposed disentangled representations play an important role in imputation and jointly enhance the performance of the final model.
5.5 CASE STUDY OF DISENTANGLEMENT
This section is to demonstrate that the learned representations achieve disentanglement and offer explanability. We first visualize a time-series from Guangzhou dataset and its learnt patterns, namely, trend patterns Vt, seasonality patterns u V as and bias patterns Vb. where u is the time-sert series representation (a row of U). Figure shows that the learned representations generate semantically meaningful patterns. Figure (bb) shows gradually changing curve that corresponds to trend
8

---

Published as a conference paper at ICLR 2023
data
(8)
100
200
30
010
300
(b)
-20
506
V
[c]
me
200 X0
40
500
2.5
/
by MMWM
60
500
Step
Figure 4: Case study of disentanglement. (a) is the original time-series, whereas (b)(() and (d) represent the generated intrinsic trend, seasonality, and local variation within a period of time.
patterns, whereas Figure (c) exhibits clear periodicity, and Figure 4 (d) manifests more idiosyncratic behaviors. Each of the displayed patterns captures information from particular semantic perspective, which verifies that the learned representations V.. V.. V exhibit independence and disentanglement.
In order to further show TIDER's interpretability, we conduct experiments on synthetic time-series where one pair time-series are composed of the same trend but different seasonality, while another pair of time-series are composed of the same seasonality but a different trend. We visualize the ground truth trend and seasonality patterns, together with the learned ones. Their results are depicted in Figure and Figure 6 We can clearly find that the learned disentangled components to the ground truth, while the disentangled patterns that are supposed to be close to each other indeed look similar (similar slopes the trend components, together with similar amplitudes and periods on the seasonality components). This further supports TIDER's explainability of disentanglement.
very similar
5.6 HYPERPARAMETER SENSITIVITY
We analyze the impact of four key hyperparameters (D, K, Da. and P) on performance of TIDER, and present the results in Figure 7 8 9 and respectively. We observe that TIDER is relatively stable under different hyperparameter settings. Even for hyperparameter P (period), TIDER exhibits consistent imputation performance, although interpretability may be compromised. Therefore, if accurate imputation is the primary objective, extensive hyperparameter tuning may not be necessary, as TIDER robust to variations in hyperparameters. However, when both accuracy and interpretability are important, determining the appropriate time series period is critical. This can often be achieved
through prior knowledge or other periodic detection models (Fan et al. 2022: Wang et al.) 2022).
6 CONCLUSION
In this paper, we propose a scalable multivariate time series imputation method, TIDER, with multiple novel inductive biases under the framework of low-rank matrix factorization. In contrast to existing imputation approaches, TIDER adopts semantically meaningful disentangled representations to account for multiple factors time series. In particular, it enables capturing periodicity with novel Fourier basis-based representation and allows to identify local time variation with a time bias representation. TIDER's superiority is verified by experimental results. Moreover, it scales well to long time series where the existing methods struggle or even cannot fit. In the future, it is interesting to investigate how to use the meaningful disentangled representations for forecasting tasks and how to design constraints on U when channel information is available. Furthermore, since hyperparameter tuning techniques to decide P in V., how to obtain adaptively in a data-driven way is also worth further investigation.

---

Published as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
We provide an open-source implementation of our proposed model, TIDER, at https: / thub com/ 1i liwwwwie /2000/ /TIDE Hyperparameter setting is shown in Section A.8 Users can download the code and run TIDER easily.
8 ACKNOWLEDGEMENT
This study is supported under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well cash and in-kind contribution from Singapore Telecommunications Limited (Singtel), through Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU), This research/project is also supported by the National Research Foundation, Singapore under its AT Singapore Programme (AISG Award No: AISGG-TTC-201-0001)
Xiucheng Li is supported in part by the National Natural Science Foundation of China under Grant No. 62206074, Shenzhen College Stability Support Plan under Grant No. GXWD202088111923000
The main part his job is done when is in Nanyang Technological University.
REFERENCES
Edgar Acuna and Caroline Rodriguez. The treatment of missing values and its effect on classifier accuracy. In Classification, clustering, and data mining applications. Springer, 2004.
Melissa J Azur, Elizabeth A Stuart, Constantine Frangakis, and Philip J Leaf. Multiple imputation by chained equations: what is it and how does it work? International jnternatiionalouraall journal of methods in psychiatric research, 20(1):40-49 2011.
Gustavo EAPA Batista, Maria Carolina Monard, al. study of k-nearest neighbour as imputation method. 2002.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE TPAMI, 2013.
Johan Bjorck, Anmol Kabra, Kilian Q. Weinberger, and Carla Gomes. Characterizing the loss landscape in non-negative matrix factorization. In Thirty- AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-
Conference on Innovative Applications of Artificial In- telligence, IAAI 2021, The Eleventh Symposium Educational Advances in Artificial Intelli- gence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 0768-677666 AAAI Press, 2021. URL AAAIAAAAAAAAA
https: ojs .aw.i..a.aaaa.................aa org/ index..ix.. http////aaaattcle/ee///////
George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Wei Cao. Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. NewrlPS, 2018
Jiawei Chen, Can Wang, Sheng Zhou, Qihao Shi, Jingbang Chen, Yan Feng, and Chun Chen. Fast adaptively weighted matrix factorization for recommendation with implicit feedback. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAA/ 2020, New York, NY, USA, February 7-12, 2020, pp. -470-3477, AAAI Press, 2020. URL https:: aaai. wwtg/idee....ooooooeee...e.eee.e phpp
AAAI AAAI/art T.AT/aticce/view///
Xinyu Chen, Yixian Chen, and Zhaocheng He. Urban traffic speed dataset of guangzhou, china, March 2018. htt.lhttts:///
woog/10. 12111...221222mdde 1205229
Andrea Cini, Ivan Marisca, and Cesare Alippi. Filling the g_ap_s: Multivariate time series imputation by graph neural networks. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event. April 25-29, 2022. OpenReview.net, 2022. URL
https:///ppeneviie net httpaarrrreeeeeee//oooiiiiaaee
10

---

Published as a conference paper at ICLR 2023
Taco Cohen. Learning transformation groups and their invariants. PhD thesis, PhD thesis, University of Amsterdam, 2013.
Wenjie Du, David Côté, and Yan Liu. Saits: Sclf-attention based imputation for time series, arXiv preprint ar 200.2200.088166, 2022.
James Durbin and Siem Jan Koopman. Time series analysis by state space methods, volume 38, OUP Oxford, 2012.
Chunlei Fan, Qun Ding, and Chi K. Tse Evaluating the randomness of chaotic binary se- quences via novel period detection algorithm. Int. J. Bifure. Chaos, 3255):2500075:1 2250075:11, 2022. doi: 10. 10.1142/502281227255500533 URL Utthttss::/ S02181 0TP288124425007550
David S Fung. Methods for the estimation of missing values in time series. 2006.
Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. Recurrent neural networks for time series forecasting: Current status and future directions. CoRR, abs/1990.005900 2019. URL
00590
Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. Recurrent neural networks for time series forecasting: Current status and future directions. International Journal of Forecasting, 37 (1):388-421) 2021.
Manpreet Kaur, Shalini Panwar, Ayush Joshi, and Kapil Gupta. Residential electricity demand prediction using machine learning. In Sarika Jain and Sven Groppe (eds.), Proceedings of the International Semantic Intelligence Conference 202/ (ISIC 2021). New Delhi, India, February 25-27, 2021, volume 2786 of CEUR Workshop Proceedings, pp. 331-340, CEUR-WS.org. 2021. URL http: ceur ws org/ / /oll 201--7888 20a-222266PPper5/ .pff
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2014.
Ankit Laddha, Shivam Gautam, Gregory P. Meyer, Carlos Vallespi-Gonzaled and Carl K. Wellington. Range view based fusion of time-series lidar data for joint 3d object detection and motion forecasting. In JEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021, Prague, Czech Republic, September 27 Oct 1, 2021, pp. doi: 10. 109//R0511..1.... 4021.9636083. URL https: //oii htt.:/ii..rrgg///// 10. 1109/ 111.:011119991R111100010 IROS51168
Ry-fusenet:
IEEE, 2021.
2021 1 6083
Guozhong Li, Byron Choi, Jianliang Xu, Sourav S. Bhowmick, Daphne Ngar-yin Mah, and Grace Lai- Hung Wong. IPS: instance profile for shapelet discovery for time series classification. In 38th IEEE International Conference on Data Engineering, ICDE 2022, Kuala Lumpur; Malaysia, May 1-, 2022, pp. 1781-1793. IEEE, 2022. doi: 10. 1009//DD5337552222.00001 https: doii..i.../ www.oogg/.... (WW.0/9/CCE33111 222 00179.
URL
Shuai Liu, Guojie Song, and Wenhao Huang. Real-time transportation prediction correction using reconstruction error in deep learning. ACM Truns. Knowl. Discov. Data, 17:11:7: 2020. doi: 10.1145/3369871 URL httlhhttt:///oo wtt.................... 111.111316397777
Yukai Liu, Rose Yu, Stephan Zheng, Eric Zhan, and Yisong Yue. Naomi: Non-autoregressive multiresolution sequence imputation. NeurIPS, 2019.
Linyuan Lü, Matúš Medo, Chi Ho Yeung, Yi-Cheng Zhang, Zi-Ke Zhang, and Tao Zhou. Recom- mender systems. Physics reports, 2012.
Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. JMLR, 2010.
Xiaoye Miao, Yangyang Wu, Jun Wang, Yunjun Gao, Xudong Mao, and Jianwei Yin. Generative semi-supervised learning for multivariate time series imputation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Thrrd Conference on Innovative Applications of Artificial Intelligence. IAAI 2021, The Eleventh Symposium Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 8983 8991 AAAI Press, 2021. URL wtt.:http/////t waa.saaaa. aaai wwweaaaa..rrgaaaam wwwg//ind.....oo A / e/v 17086

---

Published as a conference paper at ICLR 2023
Elias M Stein and Rami Shakarchi. Fourier analysis: introduction, volume 1. Princeton University Press, 2011.
Gábor Takács, István Pilászy, Bottyán Németh, and Domonkos Tikk. Matrix factorization and neighbor based algorithms for the netflix prize problem. In RecSys, 2008.
Yusuke Tashiro. Jiaming Song. Yang Song. and Stefano Ermon. Csdi: Conditional score-based diffu- sion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804 24816, 2021.
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In ICLR, 2021.
Stef Van Buuren and Karin Proothuis-Ousshoon mice: Multivariate imputation by chained equations in I. Journal of statistical software, 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, 2017.
Xu Wang. Zhaoshui He, Zhijie Lin, Yang Han, Taiheng Liu, Jun Lu, and Shengli Xie. Pažnet: Period- aware attention network for robust fetal ecg detection. IEEE Transactions on Instrumentation and Measurement, 71:1-12, 2022. doi: 10.1 20.11009/M...22..31189
Qingsong Wen, Kai He, Liang Sun, Yingying Zhang, Min Ke, and Huan Xu. Robustperiod: Robust time-frequency mining for multiple periodicity detection. In Proceedings of the 2021 International Conference on Management of Data, pp. 2328-23233 2021.
Gerald Woo, Chenghao I Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting, In The Tenth In- ternational Conference on Learning Represennattinss ICLR 2022, Virtual Event, April 25-29, 2022. net. forum? witpummmidiPPP 1
OpenReview.net, 2022. URL URL httpttttttt/:sseeeeeee
Jinsung Yoon, James Jordon, and Mihaela Schaar. Gain: Missing data imputation using generative adversarial nets. In International conference on machine learning, pp. 5689-5698. PMLR, 2018.
Hsiang-Fu Yu, Nikhil Rao, and Inderjit S Dhillon. Temporal regularized matrix factorization for nigh-dimmnsional time series prediction. NIPS, 2016.
Sebastian Zeng, Florian Graf, Christoph Hofer, and Roland Kwitt. Topological attention for time series forecasting. In NeurIPS, 2021.
A APPENDIX
A.1 PRELIMINARY
A.1.1 LOW-RANK MATRIX FACTORIZATION
Given a matrix X NxT with rank & min 73. the low-rank matrix factorization aims to factorize X as the product of two low-rank matrices U E R x and V E gXxT
=V
(11)
This low-rank matrix factorization can be extended to the cireumstance where the true rank of X larger than k. In such a case, we use a metric & to measure the discrepancy between X and the approximated product UV
(X, UV) = X - UVI
(12)
where || || denotes the Frobenius norm a matrix. This low-rank approximation fairly reasonable in practice, since many real-world structural observation matrices are low rank. This has been widely adopted and extensively studied in recommender systems (Takács et al. 2008 Chen et al. 2020).
In the context of multivariate time series data, we propose adopt low-rank matrix factorization for two reasons: 1) univariate time series channels (the rows X) are usually highly correlated; 2)
observations different time steps (the columns of X) exhibit strong dependencies.
12

---

Published as a conference paper at ICLR 2023
Table Summary of dataset statistics.
Name
Number of channels (N)
Number of time steps (7)
Guangzhou
214
500
Solar-Energy Westminster
137
52,560
7,489
744
A.1.2 FOURIER SERIES
states that any real-valued periodic function f(z) can be represented by an infinite series of sinusoidal functions, known as the Fourier series. These sinusoidal functions have their own coefficients and distinct frequencies. More formally, the Fourier series for function (x) with its period P can be described as
Fourier analysis(Stein Stein & Shakarchi, 2011 1
f(z) an cos cos(mwat (nwr)
bn sin(nez),
(13)
where w 2tT PP. and an; bn E R are the corresponding coefficients. Since sin(nwr)) and cos (oww)) functions are orthogonal to each other and are able to span over the entire functional space, they are also called Fourier basis.
A.2 DATASET DETAILS
Guangzhou Traffic Data. This dataset(Chen al. 2018 contains traffic speed of 214 anonymous urban road segments for days with a 10-minute sampling rate in Guangzhou, China. results in 214 x 500 multivariate time series matrix.
Solar-Energy Production Data. This dataset consists of solar power production records of 137 PV plants in Alabama, USA sampled every 10 minutes. It results in 137 x 52, 560 data matrix.
Westminster Uber Movement Data. This dataset contains hourly averaged speed of road segments in Westminster, in Jan 2020, released by Uber. Dimension of its data matrix is 7, 7,489 744.
The dataset statistics are summarized in Table
A.3 DETAILS BASELINE MODELS
The details of baseline methods are briefly summarized follows. For SimpleMean, KNN, and Softlmpute, we use their implementation provided by the package fancyimpute whereas for BRITS, GAIN, SAITS, CSDI, NAOMI and SingleRes, we use the source codes released by their authors.
SimpleMean & Rodriguez 2004 imputes missing entries with mean values of corresponding columns.
KNN Batista et al. 2002 first finds nearest rows with the highest similarity score to the target row, and then uses the weighted sum of these & rows for imputation.
SoftImputed (attuupeueeMaaumderr Mazumder et al.
2010 6 It is matrix completion approach based on iterative soft thresholding of Singular Value Decomposition (SVD).
MICE(Azur et
2011 It is the abbreviation for Multivariate Imputation by Chained
Equations, a widel y- used R package for imputation.
MF(Takács et al.; 2008). applies low-rank matrix factorization without any constraint on the latent factors.
httes://grtthob..om///mmo///o//
13

---

Published as a conference paper at ICLR 2023
MF+ Takács et al. 2008). applies low-rank matrix factorization with L2 regularization on and V.
TRMF Yu et al. constraint imposed on temporal matrix V.
2016).
applies low-rank matrix factorization with an autoregressive
BRITS(Cao et al.
2018
It is time series imputation method based on Bidirectional
Recurrent Neural
Nets and
a time decay mechanism.
GAIN Yoon et al.
2018
It employs Generative Adversarial Nets (GAN) for imputation.
SAITS Du et al.
2022 10
It imputes missing values based on self-attention mechanism.
CSDI Tashiro et al.
2021
It utilizes score-based diffusion models to explore correlations
between observed values and for imputation.
NAOMI(Liu al. 2019 It combines Bidirectional Recurrent Neural Nets with adversarial
training to offer non-autoregressiv style imputation. The divide-and-conque strategy is adopted.
SingleRes Liu et al. 2019 It is the autoregressive counterpart of NAOMI.
In our experiments, we find that BRITS with the suggested batch size and RNN hidden size will result in OOM (Out Memory) on the Westminster dataset. Thus we reduce the batch size to and the dimension of RNN to 10 such that it could just fit the 32 GB GPU on that dataset, we use an asterisk symbol to indicate its results in Table and Table 2 as follows.
in which the Improvement is calculated
Improvement best baselin TIDER* 100%
(14)
best_baseline
where best baseline represents the best performance among all the compared baseline models, TIDER" stands for the better one between TIDER and TIDER (no W).
A.4 DETAILS OF METRICS
Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE) are adopted to evaluate the imputation accuracy of all compared methods. These three metrics defined as
Σ Enjen(X,, (Xis X2y)² [22
Enen X X
X
X
is
RMSE
MAE
MAPE
Ω
Ω
X,y
iER
(15)
where Xiş denotes the ground-truth values, Xij is the imputed values, and Q the index set of missing entries to be evaluated.
A.5 ADDITIONAL EXPERIMENTS FOR IMPUTATION ACCURACY COMPARISON ON LONG TIME SERIES
As Table [ and Table shows, TIDER outperforms the baseline models by large margins on long time-series dataset Solar-Energy. order further verify its performance stability in imputing long multivariate time-series data, conduct additional experiments another two long time-series:
This dataset contains every minute's household electric consumption measurements gathered house located Sceaux from December 2006 to November 2010. The size of this long time-series matrix is 7 x 2075259.
HouseHold Power dataset 14
thousehol 1oeeholooee d+e1 tielect eeeectri ri c+
poxer4consumption
14

---

Published as a conference paper at ICLR 2023
bonfonn Way Mornon M Martoryyy
(6)
wore crigs
(c)
leamod
(d)
(e)
28
20
Figure 5: Disentanglement validation on synthetic dataset. Time series are composed of the same trend but different seasonalities. (a) shows the raw time series. (b) and (d) show the trend and seasonality components TIDER has learned. (c) and (e) depict the ground trend and seasonality components which compose the raw time series.
/
(a)
(b)
hmlmm//////m
(c)
(d)
Ter orgg
(a)
Figure 6: Disentanglement validation on synthetic dataset. Time series are composed of different trends but the same seasonality. (a) shows the raw time series. (b) and (d) show the trend and seasonality components TIDER has learned. (c) and (e) depict the ground trend and seasonality components which compose the raw time series.
15

---

Published as a conference paper at ICLR 2023
Table Imputation accuracy different methods with missing rate r 0.2, for long time series.
Method
Hous Power
Beijing Climate
(r = 0.2)
RMSE
MAE
MAPE
RMSE
MAE
MAPE
SimpleMean
100.6
70.07
10.69
418.5
295.9
6.521
KNN
91.57
38.55
4.313
406.9
184.5
1.141
MF
87.88
33.75
0.885
404.3
177.4
1.005
MF+L2
82.62
29.79
0.791
393.5
170.4
1.033
SoftImpute
85.61
33.75
0.651
386.5
172.9
1.043
MICE
79.21
31.27
0.642
292.2
111.9
0.827
TRMF
89.14
35.39
0.685
312.7
110.1
0.811
BRITS
66.98
25.61
0.335
101.9
66.42
0.611
GAIN
70.04
25.92
0.498
125.4
62.37
0.607
SAITS
OOM
COM
OOM
OOM
COM
OOM
CSDI
OOM
OOM
OOM
COM
COM
OOM
NAOMI
OOM
OOM
OOM
OOM
COM
OOM
SingleRes
OOM
0OM
OOM
OOM
COM
OOM
TIDER (no W)
56.36
20.20
0.351
185.8
99.78
0.754
TIDER
53.95
18.29
0.284
60.65
32.77
0.532
Improvement (%)
24.15
28.58
15.22
40.48
47.45
12.28
Table 6: Imputation accuracy of different methods with missing rate = 0.4, for long time series.
Method
HouseHold Power
Beijing Climate MAE
(r 0.4)
RMSE
MAE
MAPE
RMSE
MAPE
SimpleMean
105.2
68.17
9.064
459.8
313.8
6.454
KNN
96.07
42.68
5.362
397.8
179.4
1.053
MF
90.70
36.72
1.494
388.4
164.9
1.015
MF+L2
86.71
34.06
1.262
384.7
162.8
1,013
SoftImpute
85.72
33.61
0.698
385.1
168.0
0.912
MICE
83,46
32.48
0.672
281.1
111.3
0.820
TRMF
101.5
57.36
0.985
312.5
110.5
0.807
BRITS
69.82
27.82
0.398
163.0
75.69
0.676
GAIN
74.47
26.93
0.512
177.3
73.79
0.658
SAITS
OOM
OOM
COM
COM
CSDI
OOM
COM
COM
OOM
OOM
COM
NAOMI
OOM
COM
OOM
COM
COM
SingleRes
0OM
COM
COM
0OM
COM
COM
TIDER (no W)
64.44
25.53
0.391
221.4
112.0
0.872
TIDER
59.07
20.01
0.356
80.87
55.85
0.581
Improvement (%)
15,40
25.70
10.55
50.39
24.31
11.70
Beijing Climate dataset This dataset records the PM2.5 concentration, dew point, temperature, pressure, combined wind direction, cumulative wind speed, cumulative hours of snow, and cumulative hours of rain in Beijing, from Jan 1st, 2010 Dec 31st, 2014. The size of this long time-series matrix is 7 x 43824.
The imputation accuracy of all models on these two long time series is shown in Table 5 and Table 6 Similar to the observation made in Section 5.2 many deep-learming-bee based models that perform well in short time-series dataset Guangzhou (eg, CSDI, SAITS) run into out of memory on these two datasets, while our proposed method can easily handle them and achieve significantly better performance. These results further support the scalability of our model, as well as its effectiveness in imputing long time-series datasets.
"https://archive" iss usi www.ii.ddmmmmdaaaeeetteeee ng+ PM2. 5+Data
16

---

Published as a conference paper at ICLR 2023
5.00
4.00
0.15
4.75
0.75
3.50
RMSE
MAE
MAPE
4.25
5.45
9.33
3.75
275
3.30
250
40
50
70
0:11
30
50
D
(a)
(b)
(c)
Figure 7: The performance of TIDER changes against D.
5.00
4.00
0:15
4.50
3.50
RMSE
MAE
MAPE
6:25
5.25
0.33
4.00
3.00
0.12
3.70
277
3.50
K
K
(a)
(b)
Figure 8: The performance of TIDER changes against K.
A.6 DISENTANGLEMENT VALIDATION ON SYNTHETIC DATASET
In order to further show TIDER's interpretability, we conduct experiments on synthetic time-series where one pair of time-series are composed of the same trend but different seasonality, while another pair of time-series are composed of the same seasonality but a different trend. We visualize the ground truth trend and seasonality patterns, together with the learned ones. Their results are depicted in Figure and Figure We can clearly find that the learned disentangled components are very similar to the ground truth, while the disentangled patterns that are supposed to be close to each other indeed look similar (similar slopes the trend components, together with similar amplitudes and periods on seasonality components). This further supports TIDER's explainability of disentanglement.
A.7 HYPERPARAMETER SENSITIVITY
In this section, we study the performance change of TIDER under different hyperparameter settings. Figure 7 shows TIDER's hyperparameter sensitivity under different D. Figure & draws the perfor- mance of TIDER changing against K, the number of sinusoidal waves V s. Figure depicts accuracy curves of TIDER varying over different Da. Figure 10 used to study the performance stability of TIDER under differentt periods in V.. It can be seen that TIDER is rather stable under different hyperparameter settings. Therefore, if our target is merely for accurate imputation, not need to pay much effort or spend much time on hyperparameter tuning, since TIDER is stable among different hyperparameter settings. If we target both accuracy and interpretability, then the only hyperparameter we need to figure out is the period of time series. This often can be acquired by our prior knowledge or by other periodic detection models Wen et al. (202T).
deuuecss the
might
5.00
1.00
0.15
0.14
RMSE
4.50
3.50
MАPE
4.00
3,00
0.12
3.75
3.50
20
250
011
Db
Dy
Do
(a)
(b)
Figure 9: The performance of TIDER changes against Da.
17

---

Published as a conference paper at ICLR 2023
5.00
4.00
06.5
4.75
3.78
3.50
RMSSE
MAPE
4.25
5.43
0.13
3.75
275
3.60
12
100
100
200
500
250
24
30
160
200
500
0.11
50
100
100
100
(a)
(b)
(c)
Figure 10: The performance of TIDER changes against
Table Hyperparameters of TIDER for the three datasets.
Name (r 0.2)
Guangzhou
Solar- Bnear-Energy
Westminster
batch size
32
32
512
epoch
500
500
500
D
50
50
150
D3
2
2
2
0.01
0.01
0.01
n2
0.01
0.01
0.01
0.1
0.01
0.01
A
0.2
0.1
0.1
K
20
2
15
w
84
84
Name (r 0.4)
Guangzhou
Solar-Energy
Westminster
batch size
32
32
512
epoch
500
500
500
D
50
50
150
Do
2
2
2
Th
0.01
0.01
0.01
m2
0.01
0.01
0.01
d
0.1
0.01
0.01
0.2
0.1
0.1
K
20
2
15
84
A.8 HYPERPARAMETER SETTING
implement TIDER using Python 3.6 Pytorch 1.9, and optimize the model parameters using Adam (Kingma & Ba
2014) with learning rate 1e 3. We gird search to select the optimal Da. P) on the validation datasets. However, as we have observed in Section 5.6 and Section A.8 the imputation performance of TIDER is relatively steady among
hyperparameters (D,
different hyperparameter settings. Thus other hyperparameter sets might also offer desirable results.
The general principle and guidelines parameter setting follows:
This hyperparameter is involved in the seasonality representation matrix. w is close relation with the time-series period P. At present, P is chosen by jointly using our prior knowledge of different time series and hyperparameter tuning techniques. We first construct a candidate list of P based on our knowledge of these time series and select the best one according to performance on validation datasets. We for Guangzhou. Solar-Energy Energy and Westminster datasets as 168, 168, and 24 respectively. In other words, the optimal w for these datasets are 84
84 and
12'
D: Dimension for matrix and V, is also an important hyperparameter. Small will lack the ability to learn enough information while large D is prone overfitting.
18

---

Published as a conference paper at ICLR 2023
Model Name
TIDER
Monoory Usage/MB)
1980
Venntry UsageMBB
1340
BRITS
1980
Model Name
1540
1620
1520
1500
1480
0
1000
2000
3000
4000
5000
100
200
300
400
500
600
/00
Dimension of Time- (erie-Srres(())
Temporal Length(T)
(a)
(b)
Figure 11: Supplementary scalability test on Westminster dataset. (a) Memory usage between TIDER and CSDI varies over N from 0 to 5000 with fixed T 100; (b) Memory usage between TIDER and BRITS varies over T from 100 to 700 with fixed N 100.
Ar, Ap: These are the corresponding weights for the constraint functions of the trend repre- sentation matrix and bias representation matrix, respectively. We tune them by making choice from a scale set (0.01, 0.1, 0.2, 0.5,
By following this principle, the optimal hyperparameters we obtained are listed Table 7
A.9 SUPPLEMENTAL SCALABILITY ANALYSIS
there is a little gap between memory usage of CSDI and TIDER Figure Ba), and little gap between memory usage of BRITS and TIDER in Figure 3 In Figure (a) Figure (b)) the memory usage gap between CSDI (resp. BRITS) and TIDER is overwhelmed by the large magnitude of other methods. Thus, we zoom in give comparison in this section. As depicted in Figure [I] there are still large memory usage gaps between TIDER and them, which demonstrates the scalability superiority of our proposed TIDER.
In Section 5.3
(a) (resp.
19


Reviews:
[{"review_id": "X6mcWBJF48u", "reviewer": "Reviewer_iGko", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a time-series imputation technique, called TIDER, which utilizes matrix factorization with disentangled temporal representations. The proposed method is designed to learn disentangled representations to explain seasonality and trend and expected to scale well to large datasets. The authors validate their models with pre-existing imputation methods (e.g., KNN, BRITS, CSDI) on three real-world datasets.", "summary_of_the_review": "The authors provides a good description about their motivation and proposed method to solve the targeted problem. Although the suggested method is interesting, I have a concern with its originality as the suggested method seems to be a simple combination of low-rank matrix factorization with a notion of seasonability and trend in time-series dataset. Further, the experimental results are not quite convincing since the suggested method slightly outperforms its baselines on most datasets.", "strength_and_weaknesses": "The proposed model provides two benefits. First, cross-channel correlations can be decouples such that it enables to selectively optimize channel-wise model parameters, which scales well to large datasets. Second, the proposed method models complex temporal dynamics with explainability as time-related information can be isolated into the latent vector V.  \n\nThe experimental results on imputation accuracy with missing rates show that the proposed method slightly performs better than its baselines. It is hard to say the difference in accuracy performance between TIDER and its baselines statistically significant.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Fbu95_K2CZ", "reviewer": "Reviewer_3zzf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of imputing missing values in multivariate time series. As opposed to existing methods which depend on learning a single entangled representation to impute missing values, this paper proposes a disentangled matrix factorization based approach that aims to capture trend, seasonality and local bias as independent factors. This allows the proposed methods to be interpretable as well as scalable. The experiments claim to show improvement over existing methods.", "summary_of_the_review": "The paper is good and can conceivably make it to ICLR. It is a worthwhile contribution but the extensive hyperparameter tuning involved here without clear guidance is a sticking point.", "strength_and_weaknesses": "Strengths:\n- The paper is overall well written, well motivated, the problem and the solution both are very clearly articulated. \n- This is a problem worth solving well as multivariate time series data are present in crucial applications and almost always require imputation. (Health monitoring is perhaps the most critical and obvious application of this)\n- The experiments include comparisons with a wide variety of competing and baseline methods. \n- The solution results in interpretable factors and is scalable.\n\nWeakness:\n- There is quite a bit of redundancy in the writing. e.g. the point that disentangled representations are better than entangled representations is made unnecessarily too many times. e.g. the first 6 lines of first paragraph in section 4.1 are not really needed in my opinion as that has already been said thrice earlier.\n\n- I'd have loved to see more discussion of the interpretability/explainability aspect in the main paper, more than just a single case study in experiments section. If redundancy is removed there's plenty of room for discussion about this in the paper.\n\n- My biggest concern is the objective function involves a lot of hyperparameters. And while the authors did a good job (from reproducibility perspective) of writing out all hyperparameter choices it is very much unclear how: 1) those choices can actually be made in practice (the current explanation in the appendix is too vague) and 2) how sensitive are the empirical results to hyperparameter choices. This second point is especially crucial since wrong choices can conceivably wipe out whatever improvement is gained from this method. I will be willing to reconsider my rating if this particular issue is resolved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Cbwb0xMR1C5", "reviewer": "Reviewer_tDGy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a disentangled matrix factorization method for time series imputation. This paper achieves better accuracy and better interpretability compared to several baselines. ", "summary_of_the_review": "This paper is novel overall and the experiments are promising. ", "strength_and_weaknesses": "Pros\n1. Using matrix factorization for time series imputation is novel. \n2. This method is scalable to long time series\n3. Figure 4 shows how this method is more interpretable than deep learning based methods. The figure clearly shows  trend, seasonality, and local variation of the dataset\n\nCons\n1. I am not sure the assumption of  trend, seasonality, and local variation hold for all the datasets. There may be some datasets that don't have trends or seasonality and some datasets contain very unpredictable pattern\n2. I am not sure how this work can impute missing data by factorization. Could you explain more?\n3. The sentence \"but also scales well to large datasets on which existing deep learning based methods struggle.\" in abstract is misleading. What the author actually want to say is those deep learning methods is not scalable to long time series rather than large datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2AieS7on-1w", "reviewer": "Reviewer_CdFR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the multivariate time-series imputation problem. Existing methods all rely on entangled representations. In contrast, this paper contributes a novel methods that explicitly models 3 disentangled temporal representations, making the imputation process more reliable and offers interpretability.", "summary_of_the_review": "After evaluating the novelty of method, the writing quality, and experiments, I think this paper is marginally below the acceptance threshold.", "strength_and_weaknesses": "Strengths: \n1) This work is significant for the multivariate time-series imputation community. It focuses on two rarely studied directions -- explaining the temporal representations and using the disentangled temporal representations to improve imputation performance, which is still a challenging task in both classical and deep learning based methods.\n2) The paper is well written. Experiments are sufficient and comprehensive and can prove the reliability as well as the explainability of the proposed method.\n\nWeaknesses:\n1)  The proposed method is based on matrix decomposition and may not be novel enough for ICLR. Also, disentangled factors are pre-defined and hard coded in the model -- Trend, seasonality, and bias factors. In reality, however, the hidden temporal factors of time-series data may be unknown, might be more complex, and should be learned as well. The paper may have limitations if given high dimensional temporal data or if multivariate time-series have more complex inter-channel correlations and variations.\n2) The training is specific for individual time-series data. The structure of latent space is not regularized over dataset to guarantee generalizable disentanglement.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster