TimeAutoML: Autonomous Representation Learning for Multivariate Irregularly  Sampled Time Series
Authors: Yang Jiao
Kai Yang
shaoyu dou
pan luo
Sijia Liu
Dongjin Song
Venue:  (2021)
Keywords: representation learning
AutoML
irregularly sampled time series
anomaly detection
clustering

Abstract: Multivariate time series (MTS) data are becoming increasingly ubiquitous in diverse domains, e.g., IoT systems, health informatics, and 5G networks. To obtain an effective representation of MTS data, it is not only essential to consider unpredictable dynamics and highly variable lengths of these data but also important to address the irregularities in the sampling rates of MTS. Existing parametric approaches rely on manual hyperparameter tuning and may cost a huge amount of labor effort. Therefore, it is desirable to learn the representation automatically and efficiently. To this end, we propose an autonomous representation learning approach for multivariate time series (TimeAutoML) with irregular sampling rates and variable lengths. As opposed to previous works, we first present a representation learning pipeline in which the configuration and hyperparameter optimization
are fully automatic and can be tailored for various tasks, e.g., anomaly detection, clustering, etc. Next, a negative sample generation approach and an auxiliary classification task are developed and integrated within TimeAutoML to enhance
its representation capability. Extensive empirical studies on real-world datasets demonstrate that the proposed TimeAutoML outperforms competing approaches on various tasks by a large margin. In fact, it achieves the best anomaly detection
performance among all comparison algorithms on 78 out of all 85 UCR datasets, acquiring up to 20% performance improvement in terms of AUC score.

Under review as a conference paper at ICLR 2021
TIMEAUTOML: AUTONOMOUS REPRESENTATION LEARNING FOR MULTIVARIATE IRREGULARLY SAM- PLED TIME SERIES
Anonymous authors Paper under double- blind review
ABSTRACT
Multivariate time series (MTS) data are becoming increasingly ubiquitous in di- verse domains, e-g., IoT systems, health informatics, and 5G networks. To obtain an effective representation of MTS data, it is not only essential to consider un- predictable dynamics and highly variable lengths these data but also important to address the irregularities in the sampling rates of MTS. Existing parametric approaches rely on manual hyperparameter tuning and may cost a huge amount of labor effort. Therefore, it is desirable to learn the representation automatically and efficiently. To this end, we propose an autonomous representation learning ap- proach for multivariate time series (TimeAutoML)) with irregular sampling rates and variable lengths. As opposed to previous works, we first present a representa- tion learning pipeline in which the configuration and hyperparameter optimization are fully automatic and can be tailored for various tasks, e.g., anomaly detection, clustering, etc. Next, a negative sample generation approach and an auxiliary classification task are developed and integrated within TutoAt to enhance representation capability. Extensive empirical studies on wal-world datasets demonstrate that the proposed TiteA outperforms competing approaches on various tasks by large margin. In fact, it achieves the best anomaly detection performance among all comparison algorithms on 78 out of all 85 UCR datasets, acquiring up to 20% performance improvement in terms of AUC score.
INTRODUCTION
The past decade has witnessed a rising proliferation Multivariate Time Series (MTS) data, along with plethora of applications in domains as diverse IoT data analysis, medical informatics, and network security. Given the huge amount of MTS data, it is crucial to learn their representations effectively as facilitate underlying applications such as clustering and anomaly detection. For this purpose, different types of methods have been developed to represent time series data.
Traditional time series representation techniques, e.g., Discrete Fourier Transform (DCT) (Faloutsos) et al. 1994). Discrete Wavelet Transform (Cha))Chhn & Fuj 1999), Piccewise Aggregate Approx- imation (eAg) al., 2001). etc., represent raw time series data based on specific domain knowledge/data properties and hence could be suboptimal for subsequent tasks given the fact that their objectives and feature extraction are decoupled.
More recent time series representation approaches, e.g., Deep Temporal Clustering Representation (DTCR) (Ma et al., 2019), Self OegfnOrrgnizing Map based Variational Auto Encoder (SOM-VAL VAE) (For-) tuin et al. 201 8), etc., optimize the representation and the underlying task such as clustering in an end-to-end manner. These methods usually assume that time series under investigation are uniformly sampled with a fixed interval. This assumption, however, does not always hold in many applications. For example, within a multimodal IoT system, the sampling rates could vary for different types of sensors.
Unsupervised representation learning for irregularly sampled multivariate time series is a challeng- ing task and there several major hurdles preventing from building effective models: i) the design of neural network architecture often employs a trial and error procedure which is time con- suming and could cost substantial amount of labor effort; ii) the irregularity in the sampling rates

---

Under review as a conference paper at ICLR 2021
constitutes major challenge against effective learning of time series representations and render most existing methods not directly applicable; iii) traditional unsupervised time series represen- tation learning approach does not consider contrastive loss functions and consequently only can achieve suboptimal performance.
To tackle the aforementioned challenges, we propose an autonomous unsupervised representation learning approach for multivariate time series to represent irregularly sampled multivariate time se- ries. TimeAutoML differs from traditional time series representation approaches in three aspects. First, the representation learning pipeline configuration and hyperparameter optimization are carried out automatically. Second, a negative sample generation approach is proposed to generate negative samples for contrastive learning. Finally, an auxiliary classification task is developed to distinguish normal time series from negative samples. this way, the representation capability of TimeAu- toML. is greatly enhanced. We conduct extensive experiments on UCR time series datasets and UEA multivariate time series datasets. Our experiments demonstrate that the proposed TimeAu- toML outperforms comparison algorithms on both clustering and anomaly detection tasks by large margin, especially when time series data is irregularly sampled.
2 RELATED WORK
Unsupervised Time Series Representation Learning Time series representation learning plays an essential role in a multitude of downstream analysis such as classification, clustering, anomaly detection. There is growing interest in unsupervised time series representation learning, partially because no labels are required in the learning process, which suits very well many practical ap- plications. Unsupervised time series representation learning can be broadly divided into two cate- gories, namely 1) multi-stage methods and 2) end-to-end methods. Multi-stage methods first learn a distance metric from a set of time series, or extract the features from the time series, and then perform downstream machine learning tasks based on the learned or the extracted features. Eu- clidean distance (ED) and Dynamic Time Warping (DTW) are the most commonly used traditional time series distance metries. Although the ED is competitive, it is very sensitive to outliers in the time series. The main drawback of DTW is its heavy computational burden. Traditional time series feature extraction methods include Singular Value Decomposition (SVD), Symbolic Aggregate Ap- proximation (SAX), Discrete Wavelet Transform (DWT)(Chan & Fu 1999). Piecewise Aggregate Approximation
et al. 2001), etc. Nevertheless, most of these traditional methods for regularly sampled time series, so they may not perform well on irregularly sampled time series. In recent years, many new feature extraction methods and distance metrics are proposed to overcome the drawbacks mentioned above, For instance, Paparrizos & Gravano (2015): Petitjean et al. (201T) combine the proposed distance metrics and K-Means algorithm to achieve clustering. Lei et al. (017)) first extracts sparse features of time series, which is not sensitive to outliers and irregular sampling rate, and then carries out the K-Means clustering. In contrast, end-to-end approaches learn the representation of the time series in an end-to-end manner without explicit feature extraction or distance learning (Fortuin et al. 2018: Ma et al.) 2019). However, the aforementioned methods need to manually design the network architecture based on human experience which is time-consuming and costly. Instead, we propose in this paper a representation learning method which optimizes an AutoML pipeline and their hyperparameters in a fully autonomous manner. Furthermore, we con- sider negative sampling and contrastive learning in the proposed framework to effectively enhance the representation ability of the proposed neural network architecture
Irregularly Sampled Time Series Learning There exist two main groups of works regarding ma- chine learning for irregularly sampled time series data. The first type of methods impute the missing values before conducting the subsequent machine learning tasks (Shukla & Marlin 2019 Luo et al.) 2018 2019: Kim & Chi, 20188) The second type directly learns from the irregularly sampled time series. For instance, Che et al. (2018); Cao et al. (2018) propose memory decay mechanism, which replaces the memory cell of RNN by the memory of the previous timestamp multiplied by a learn- able decay coefficient when there are sampling value at this timestamp. Rubanova et al. (2019) combines RNN with ordinary differential equation to model the dynamic of irregularly sampled time series. Different from the previous works, TimeAutoML makes use of the special characteris- tics of RNN (Abid & Zou 2018) and automatically configure a representation learning pipeline to model the temporal dynamics of time series. It is worthy mentioning that there are many types of
2

---

Under review as a conference paper at ICLR 2021
irregularly sampled time series, which may be caused by sensor failure sampling time error. And what we put emphasis on analyzing in this paper a special type of irregularly sampled time series, which have many missing timestamps compared to regularly sampled time series.
Au Automatic Machine Learning (AutoML) aims to automate the time-consuming model development process and has received significant amount of research interests recently. Previous works about AutoML mostly emphasize on the domains of computer vision and natural language processing, including object detection (Ghiasi et al. 2019; Xu et al.) 2019) Chen et al.), semantic segmentation (Weng al.) 2019; Nekrasov et al.] 2019; Bae et al., 2019). translation (Fan et al., 2020) and sequence labeling (Chen et al., 2018a). However, AutoML for time series learning is an underappreciated topic far and the existing works mainly focus supervised learning tasks, e.g., time series classification. [kki & Bandyopadhyay propose an AutoML pipeline for automatic feature extraction and feature selection for time series classification. Van Kuppevelt al. (2020) develops an AutoML framework for supervised time series classification, which involves both neural architecture search and hyperparameter optimization. Olsavszky et al. (2020) proposes a framework called AutoTTS, which performs time series forecasting of multiple diseases. Nevertheless, to our best knowledge, no previous work has addressed unsupervised time series learning based on AutoML.
Summary of comparisons with related work We next provide comprehensive comparison be- tween the proposed framework and other state-of -hr methods, including (WaRTEm (Mathew et al. 2019). DTCR (Ma et al. 2019). USRLT (Franceschi et al. 2019) and BeatGAN (Zhou al. 2019 ), as shown in Table | In particular, we emphasize on a total of seven features the com- parison, including data augmentation, negative sample generation, contrastive learning, selection of autoencoders, similarity metric selection, attention mechanism selection, and automatic hyperpa- rameter search. TimeAutoML is the only method that has all the desired properties.
Table Comparisons with related methods
WaRTEm
DTCR
USRLT
BeatGAN TimeAutoML
Data augmentation
Negative sample generation
Contrastive training
Autoencoder selection
Similarity metric selection
Attention mechanism selection
Automatic hyperparameter search
TIMEAUTOML FRAMEWORK
3.1 PROPOSED AUTOML FRAMEWORK
Let X (m) (. m denote set of N time series in which æ, E R.T,, where T; is the length of the 7th time series x,- We aim to build an automated time series representation learning framework to generate task-aware representations that can support a variety of downstream machine learning tasks. In addition, we consider negative sample generation and contrastive self-supervised learning. The contrastive loss function focuses on building time series representations by learning to encode what makes two time series similar or different. The proposed TimeAutoML framework can automatically configure an representation learning pipeline with an array of functional modules, each of these modules is associated with set of hyperparameters. We assume there are total of M modules and there are Q options for the ith functional module. Let k E (0, 1) denote an indicating vector for 7th module, with the constraint 1 TT,k=>> Σタ ki = 1 ensuring that only a single option is chosen
for each module. Let 0., be the hyperparameters of 7th option in 7th module, where o and 0., are respectively the continuous and discrete hyperparameters. Let O and K denote the set of variables to optimize, i.e., θ e ;. Vi E [M] E [Q1]} and K = [kr....
}. We further let fK, Q) denote the corresponding objective function value. Please note that the objective function differs for different tasks. For anomaly detection, we use Area Under the Receiver Operating Curve (AUC) as objective function while we the Normalized Mutual Information (NMI) objective function for
3

---

Under review as a conference paper at ICLR 2021
clustering. The optimization problem of automatic pipeline configuration is shown below.
max J(K, ) K,6
k (0. 10, Vi [M], o C. Li E Da.. Vi E [M],j E [Q.]
subject to
We solve problem 1 by alternatively leveraging Thompson sampling and Bayesian optimization, which will be discussed follows.
Original Time Series Set
Data Augmentation
X
D(X,, X
Nogative Sample Generation
Attention
do
1K
fat
EN
E(y)
nrg
tla
BCE
11
The representation learning pipeline of TimeAutoML. There are totally eight modules forming this pipeline, namely data augmentation, encoder fen, attention, decoder Saee similarity se- lection ssim: estimation network fest EM estimator fEM. auxiliary classification network Selns- And fECE represents the binary cross entropy computation, D x!""). Ely,) and Leei represent the reconstruction loss, energy and self-supervised contrastive loss of input sample 2,, respectively.
Figure 1:
3.1.1 PIPELINE CONFIGURATION
We first assume that the hyperparameters @ are fixed during the pipeline configuration. We aim at selecting the better module option K to optimize objective function f(K, +), can delineate it as a K max problem:
max f(K, + XX((K, Xx(() { O, Kf E
max K
(2)
-0, else
where K is the feasible set, i..., K (K K = (k.,., E (0.1)%; 1 l = 1, Vi E [M]} and Xx (K) is penalty term that makes sure K fall in the feasible region.
Thompson sampling is utilized to tackle problem sumes the sampling probability of every option in each module follows Beta distribution, and the one corresponding
In every iteration, Thompson sampling as-
the maximum sampling value in each module will be chosen construct the pipeline. After that, Beta distribution of the chosen options will be updated according to the perfor- mance of the configured pipeline. Due to space limitation, more details about Thompson sampling and the search space for pipeline configuration are shown in Appendix B and Appendix C respec- tively.
The representation learning pipeline consists of cight modules, namely data augmentation, auxiliary classification network, encoder, attention, decoder, similarity selection, estimation network and EM estimator, elucidated in Figure [1 The goal of data augmentation is to increase the diversity of samples. The auxiliary classification network aims at distinguishing the positive samples from gen- erated negative samples, which will be discussed in detail in Section 3.2 And we combine encoder, attention, decoder and similarity selection together to generate the low-dimensional representation of the input time series. Given an input time series E;, we can generate the latent space repre- sentation y, which is an concatenation of the output of h; and reconstruction error z,, as shown below:
h - fon (x,),
rec fao (h.), Z, = fsim (z,, x00)
y
h 2,1.
where fan and far refer to an encoder and a decoder, respectively. There are three options for the en- coder and decoder, namely, Recurrent Neural Network (RNN), Long Short Term Memory (LSTM),
4

---

Under review as a conference paper at ICLR 2021
Gated Recurrent Unit (GRU). fsim is similarity function that characterizes the level of similarity between the original time series and the reconstructed one. Three possible similarity functions are considered in this paper, i.e., relative Euclidean distance, Cosine similarity, concatenation of both.
After obtaining the latent space representation of the input time series, EM algorithm is then invoked to estimate the mean and convariance of GMM. Assuming there are H mixture components in the GMM model, the mixture probability, mean, covariance for component in the GMM module can be expressed as oh Ps En, respectively. Assuming there total of N samples, the key parameters of GMM can be calculated as follows:
y- fest (ys) [N], OR = E Ia vh E [H], N. (E)) vA [H],
(4)
E.
IN
where fest the estimation network which is multi-layer neural network, and Y E R is the mixture-component membership prediction vector. fEM is the EM estimator which can estimate the mean and convariance of GMM via the EM algorithm. The h entry of this vector represents the probability that y belongs to the h mixture component. The sample energy E(y, is given by.
ON exp( (
y 1
Ely,) - log(
(5)
2
The sample energy can be used to characterize the level of anomaly of an input time series, that is, the sample with high energy will be deemed that TimeAutoML may suffer from the singularity problem as in GMM. In this case, the training algorithm may converge trivial solution the covariance matrix is singular. We prevent this singularity problem by adding le 6 to the diagonal entries of the covariance matrices.
unusual time series. It is worth noticing
3.1.2 HYPERPARAMETERS OPTIMIZATION
Once the representation learning pipeline is constructed, we then emphasize on optimizing the hy- perparameters for the given pipeline. Here we make use of the Bayesian Optimization (BO) (Shahri-) ari et al. 2015) to tackle this max task, as given below,
max S(K, (0C, xx)))+x((0 +xx(()) 00,90
O, O EC xD((p)= 00, else
if E
(6)
Xo -00, else
where the set C' and D denote respectively feasible region of the hyperparameters, and f(-) is the objective function given in problem (1). (c(@C) and (D( are penalty terms that make sure the hyperparameters fall in the feasible region. Unlike random search (Bergstra Bengio 2012) and grid search (Syarif et al. 2016). BO is able to optimize hyperparameters more efficiently. More dddrits the main steps in TimeAutoML details about BO are discussed in Appendix A Algorithm depicts
and more details are given in Appendix B
Algorithm 1: TimeAutoML. with Contrastive Learning
Input: Maximum iterations
Bayesian Optimization iterations B.
for / - 1, 1,2, ...... L do
Configuring complete AutoML pipeline accoding Thompson Sampling.
for b= 1,2 ..................... do
Hyperparameters optimization: Update hyperparameters according to Bayesian Optimization and obtain the objective function value.
end
Update parameters of Thompson Sampling each module according to the obtained objective function values.
end
Output: Configured AutoML pipeline and optimized hyperparameters.
5

---

Under review as a conference paper at ICLR 2021
3.2 CONTRASTIVE SELF-SUPERVISER Loss
According to Zhou et al. 2019);; Kieu et al. (2019): Yoon et al. (211)). the structure of the encoder has direct impact on the representation learning performance. Take anomaly detection as ex- ample, the semi-supervised anomaly detection methods (Pang et al. 2019 Ruff et al., 2019) assume that there are few labeled anomaly samples in the training dataset, which more effective repre- sentation learning than unsupervised methods. Instead, the proposed contrastive self- self-supervised loss does not require any labeled anomaly samples. It uses generated negative samples as anomalies for model building. The goal is to allow the encoder to distinguish the positive samples from generated negative samples.
Given a normal time series L. RR, which is deemed as positive sample. We then generate the negative sample z neg by adding noise randomly over a few selected timestamps of 2 that is, E _, g(z,), where g(-) is the negative sample generation trick. In the experiment, the noise amplitude is randomly selected within the interval [min(<;), max(x;)).
In the experiment, generate one negative sample _. for each positive sample 2,- The proposed contrastive self-supervised loss Luelf aims to distinguish the positive time series sample from the negative ones, which can be given as:
h (xx, h ncg Sea (2E; negg)
Los felas [L) (os), pog Sclas nog L (xx)) = TBCB(O, pos
,00 + fBCE (o; heg
1),
where 4.000 IS and 1 E R are respectively the latent space representations of positive samples and negative samples, S represents the length of latent space representation. selss is the auxiliary classification network, e E R'' and o, neg E E are the outputs of the classifier. fECE represents binary cross entropy and we label the positive time series and negative time series as and 1, respectively. More details about the proposed self-supervised loss Tuclt shown in Figure I we can see that minimizing Lueff allows the encoder to distinguish the positive samples from the negative samples in the latent space, and consequently entails better latent space representations.
3.3 OVERALL LOss FUNCTION AND JOINT OPTIMIZATION
Given a dataset with N time series, for fixed pipeline configuration and hyperparameters, the neural network is trained by minimizing overall loss function containing three parts:
Loverall N Σ D TIC +
N D E(y. + d2 ΣM", (z,)) (8)
N
where D(E x:(0)) represents reconstruction error. Eyy,) is the sample energy function which rep- resents the level of abnormality for a given sample. [2,))(22) is the proposed contrastive self- supervised loss. A and A2 are weighting factors governing the trade-off among these three parts.
4 EXPERIMENT
The performance of the proposed time series representation learning framework has been assessed via two machine learning tasks, i.e., anomaly detection and clustering. The primary goal is to answer the following two questions in the experiment, 1) Effectiveness: can the proposed represen- tation learning framework effectively model and capture the temporal dynamics of a time series? 2) Robustness: can TimeA remain effective the presence of irregularities in sampling rates and contaminated training data?
Dataset We first conduct experiments on total of 85 UCR univariate time series datasets Chen et al. 2015) to assess the anomaly detection performance. Next, we also assess the performance of the proposed TimeAutoML on multitude of UEA multivariate time series datasets (Bagnall et al.) 2018). We follow the method proposed in Chandola al. (2008) to create the training, validation, and testing dataset. AUC (Area under the Receiver Operating Curve) is employed to evaluate the anomaly detection performance. For clustering, carry out experiments a total of 3 UCR univariate datasets and 2 UEA multivariate datasets. NMI (Normalized Mutual Information) is used to evaluate the clustering results.

---

Under review as a conference paper at ICLR 2021
Baselines
For anomaly detection, the proposed TimeAutoML is compared with set of state-of-
the-art methods including latent ODE
(Rubanova et al.
2019). Local Outlier Factor (LoF) (Bre- One-Class SVM (OCSVM) (Schölkopi
unig et al.] 2000), Isolation Forest (IF)
(Liu et al. 2008).
et al.
2001). GRU-AF (Malhotra et al.
2016). DAGMM (Zong et al. 2018) and BeatGAN (Zhou
al.
2019). For clustering, the baseline algorithms for comparison include K-means, GMM, K-
means+DTW, K-means+EDR (Chen et al.
2005), K-shape (Paparrizos & Gravano| (0015), SPIRAL
(Lei et al.] 2017). DEC (Xie et al. 2016). IDEC
(Guo et al. 2017). DTC (Madiraju et al.
2018).
DTCR (Ma et al. 2019).
4.1 ANOMALY DETECTION
We present the AUC scores of the proposed TimeAutoML and other -late-of-the-art anomaly detec- tion methods for the 85 univariate time series datasets of UCR archive (Chen et al. al. (0013). Due to the space limitation, tection results are summarized in Table 2 The anomaly detection results for the remaining datasets summarized in Appendix [ Table A2
choose portion of the time series datasets and the corresponding anomaly de-
A3 It is seen that Ti achieves best anomaly
detection performance over the majority > 90% of the UCR datasets no matter the time series are regularly or irregularly sampled. In addition, we evaluate the performance of TimeAutoML on a multitude of multivariate time series datasets from UEA archive (Bagnall et al. 2018).
Effectiveness We assess the anomaly detection performance when time series are irregularly sam- pled (3 = 0.5) and regularly sampled (3 = 0), where 3 is the irregular sampling rate representing the ratio of missing timestamps to all timestamps (Chen et al. 201 8b). Note that we put emphasis on analyzing time series with missing timestamps, which is a special type of irregular sampling time series. Table 2 presents the AUC scores of the proposed TimeAutoML and offte-of-the-ar anomaly detection methods on a selected group of UCR datasets and UEA datasets. We observe that the performance of BeatGAN severely degrades in the presence of irregularities in sampling rates since it is designed for fixed-length input vectors. We also notice that the proposed TimeAA exhibits superior performance over existing state-of-the-ard anomaly detection methods in almost all cases for irregularly sampled time series. In addition, the negative sampling combined with the contrastive loss function can further boost the anomaly detection performance.
Table 2: AUC scores of TuLoA and state-of-the-art anomaly detection methods when time seires are regularly sampled (3- = 0) and irregularly sampled (3 = 0.5). Bold and underlined scores respectively represent the best and second-best performing methods.
Model
BO0RseDays GasPiee
DuyyD
Melkellrages
MaxSnii
FrgeeMoereeh
Rocketipents PhoternSpptrs
Heathoot
J-
2-066
LOF
0.5211 Usiss 05583 04856 05133 0.392 (1506 0,3307 05555 0.395 05:32 04491 0.589 C5499 0.5492 06492 01415 0.4118 035466 015546 0.52)
0,552)
0.5095 0ess 1597I DASS1 0-557 1LST B.119 05009 0.5181 06317 0.005 0.536
0.576
0188 08185 0.9133
0.5155
05123 0.575 0.5124
GR01-110
[13001
0ss0s (13112 018558 0.657 (15277 (18289 DAS25
01619
0.9A6
10004
01349
0.911
CAI
111012 (1836 07181 01811
0.5451
(15241
06609
0/0722
DAGNM
0,5729
05732 0.5358 04570 0.700 (1759L 0,5359 0.443
0.5512
0,5155
0.577 0.5332
6.5332
05115 0.5511 0,9553 03363
0.5562
05562
0.004
0.5814
BeaGAN
0.5411
0.5621 07557 01666 (19756 06214 0.635
(18201
01716 0.0955
6.5994
0.77%) 06856 06688
0,6674
0.195
0.631 0/151
ODE
06637
wither
(19422 9012 (1985) (19199 01F377 (1963 (LSRT9 D.S191 (172407 0.2000 19935) (19867 0.997 0.33 0.700 (17149 006635 0.907 02967 (15159 07781 07750
TiaAAmnM. 0,9722 8.339 0.993 6/519 8.862 8,5093 (,9959 03811 05831 0,7093 69336 099166 0.945 6,6677 IL7ING 07837 6/9933 0,905 6,8177 05686 93031 6.7783
Iagervanett 12.47% 11.7%% 9.51% 28.65 923% 9U86 181% 1889% 12.85 14019 1135% 16185 17.35% 18.8% 5.3% 9.29% 6.01% 7.00 2014 21,45%
14545 12359
Robustness We investigate how the proposed TimeAutoML responds to contaminated training data when time series are irregularly sampled with the rate 3 0.5. AUC scores of the proposed TimeAutoML. when training on contaminated data are presented in Appendix[D] Table A5 We observe that the anomaly detection performance of TimeAutoML slightly degrades when training data are contaminated. Next, we investigate how AutoM responds to different irregular sam- pling rate, i.e., when |3 varies from 0 to 0.7. The AUC scores of TimeAutoML and ofate-of-the-ar anomaly detection methods on ECGFiveDays dataset are presented in Fig 2 and the results other datasets are presented in Appendix D Fig Al A2 We notice that TimeAutoML performs well robustly across multiple irregular sampling rates.
7

---

Under review as a conference paper at ICLR 2021
4.2 VISUALIZATION
In this section, we use synthetic dataset to elucidate the underlying mechanism of TutoAu model for detecting time series anomalies. Figure Figure[]shwws shows the latent space representation learned via AuL model from synthetic dataset. In this dataset, smooth Sine curves are normal time series. The anomaly time series is created by adding noise to the normal time series over a short interval. It is evident from Figure S that the latent space representations of normal time series lie in a high density area that can be well characterized by GMM; while the abnormal time series appears to deviate from the majority of the observations in the latent space. In short, the proposed encoder-decoder structure allows us to project the time series data in the original space onto vector representations in the latent space. In doing so, can detect anomalies via clustering -based methods, c.g., GMM, and easily visualize as well as interpret the detected time series anomalies.
10
Savee: E dOWVEE 3S lateal merrentaaoo Larve 1paTe diDristtoo
TimoRAtoMIL
20
0.8
Latent ODE
-03
BANEGAN
Sample
Map ongrral
GRU-ED
0.6
05
Sumple Innarna
30
69
reprerriaiion locatsele the Loan gpai www.ceouuulle..... builebi
0.5
-1.0
Semple
0.2
0.3
Rate
0.4
0.5
0.6
0.7
Original space
hore Danalyy Latert Space Latent space
Figure 2: AUC scores of TimeAutoML on ECGFiveDays dataset when irregular sam- pling rate varies from 0.7.
Figure 3: Anomaly interpretation via analy- sis in latent space.
4.3 CLUSTERING
Apart from anomaly detection, TimeAutoML. can be tailored for other machine learning tasks as well, e.g., multi-class clustering. In particular, the clustering process is carried out in the latent space via the GMM model, along with other modules in the pipeline.
We evaluate the effectiveness of Time on three univariate time series datasets as well two multivariate time series datasets. The NMI of TimeAu and state-of-the-art clustering meth- ods are shown in Table 3 We observe that TimeAutoML generally achieves superior performance compared to baseline algorithms. This is because: i) it can automatically select the best module and hyperparameters: ii) the auxiliary classification task can enhance its representation capability.
Table 3: NMI scores of AutoML and state-of-the-art clustering methods. Bold and underlined scores respectively represent the best and second-best performing methods.
Model
GunPuint
BOGFweDaas
ProximaPOAAG
AtralFibiillation
Epilepey
3
8 0.5
3-
2 (1
0.5
3- 0
B 01.5
K-means
0.0011
0.185
0.0007
0.0020
0.4842
0.0076
0
0
0.0760
0.1370
GMM
0.065
0.090
0.0130
0.0019
0.5258
0.0164
0
0
0.1276
010828
K me
0.2100
0.0766
0.2508
0.0811
0.4830
0.4318
0.0650
0.1486
0.1654
0.1534
K-meens-FDR
0.0656
00892
0.1614
0.0622
0.1105
0.0260
0.2025
0.16670
0.30664
0.2934
K-oltape
0.0111
00280
0.745%
0.855
0.4344
0.0257
0:5402
0.2541
0.2589
0.1732
SPIRAL
0.000
00019
0.0215
0.0800
0.5457
0.0143
0.2349
0.1475
0.1600
0.1912
0.1155
0.1415
0.1242
0.2236
0.1122
0.1132
0.1242
DTC
0.154
0.3036
0.086
0.0564
0.3992
0.488
0.3591
0.2583
TimeAutoML
without fouir
0.3262
0.2791
0,5914
0.3220
0.5915
0.5051
0.6623
0.5469
0.5073
0.1735
TimeAntoML.
0.332
0.2541
0.6108
0.3476
0.5981
0.5170
0.6871
0.6649
0.5419
0.5856
8

---

Under review as a conference paper at ICLR 2021
5 CONCLUSION
Representation learning on irregularly sampled time series is an under-explored topic. In this paper we propose TumeAu framework to carry out unsupervised autonomous representation learn- ing for irregularly sampled multivariate time series. In addition, we propose self-supervised loss function to get labels directly from the unlabeled data. Strong empirical performance has been ob- served for MutoML on plurality of real-world datasets. While tremendous efforts have been undertaken for time series learning in general, AutoML. for time series representation learning is still in its infancy and we hope the findings in this paper will open up venues along this direction and spur further research efforts.
REFERENCES
Abubakar Abid and James Zou. Autowarp: Learning a warping distance from unlabeled time series using sequence autocncoders. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 10568-10578, 2018.
Woong Bac, Seungho Lec, Yeha Lee, Beomhee Park, Minki Chung, and Kyu-Hwan Jung. Resource optimized neural architecture search for 3D medical image segmentation. In International Confer- ence on Medical Image Computing and Computer-Assisted Intervention, pp. 228-236. Springer, 2019.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv: 1.0007, 2018.
James Bergstra and Yoshua Bengio. Random search for hyper-paramett optimization. The Journal of Machine Learning Research, 13(1):281-305, 2012.
Aaron Bostrom and Anthony Bagnall. Binary shapelet transform for multiclass time series clas- sification. In Transactions Large-Scale Data-and Knowledge 24 46. Springer, 2017.
Chowledge-Centered Centered Systems XXXII, pp-
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. LOF: Identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93- 104, 2000.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. BRITS: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, pp. 6775- 6785, 2018
K. Chan and A. W. Fu. Efficient time series matching by wavelets. In ICDE, pp. 126-133, 1999.
Varun Chandola, Varun Mithal, and Vipin Kumar. Comparative evaluation of anomaly detection techniques for sequence data. In 2008 Eighth IEEE international conference data mining, pp- 743-748. IEEE, 2008
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Junkun Chen, Kaiyu Chen, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Exploring shared structures and hierarchies for multiple NLP tasks. arXiv preprint 1808. 07658. 2018a.
Lei Chen, M Tamer Özsu, and Vincent Oria. Robust and fast similarity search for moving object trajectories. In Proceedings of the 2005 ACM SIGMOD international conference Management of data, pp. 491-502, 2005.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 6571-6583, 2018 2018b.

---

Under review as a conference paper at ICLR 2021
Yanping Chen, Eamonn Keogh, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, and Gustavo Batista. The UCR time series classification archive, July 2015. www. CS uur.
edd.-eamonn/ time series series_datall data/
Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. DETNAS:
Neural architecture search object detection.
C. Faloutsos, M. Ranganathan, and Y. Manolopoulos. Fast subsequence matching in time-series databases. In SIGMOD, pp- 419-429, 1994.
Yang Fan, Fei Tian, Yingce Xia, Tao Qin, Xiang- Yang Li, and Tie- Yan Liu. Searching better archi- tectures for neural machine translation. IEEF/ACM Transactions Audio, Speech, and Language Processing, 2020.
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. SOM-VAE: Interpretable discrete representation learning on time series. In International Confer- ence on Learning Representations, 2018.
Jean-Y Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In Advances Neural Information Processing Systems, pp- 4650-4661, 2019.
Golnaz Ghiasi, Tsung-Y Yi Lin, and Quoc Le. NAS-FPN: Learning scalable feature pyramid ar- chitecture for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7036-7045, 2019.
Xifeng Guo, Long Gao, Xinwang Liu, and Jianping Yin. Improved deep embedded clustering with local structure preservation. In ISCAI, pp. 1753-1759, 2017.
E. Keogh, Chakrabarti, M. K., Pazzani, and S. Mehrotra. Locally adaptive dimensionality reduction for indexing large time series databases. In SIGMOD, pp. 151-162, 2001.
Tung Kieu, Bin Yang, Chenjuan Guo, and Christian S Jensen. Outlier detection for time series with recurrent autoencoder ensembles. In ICCAI, pp. 2725-2732 2019.
Yeo-Jin Kim and Min Chi. Temporal belief memory: Imputing missing data during training. In In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI-2018), 2018.
Qi Lei, Jinfeng Yi, Roman Vaculin, Lingfei Wu, and Inderjit S Dhillon. Similarity preserving representation learning for time series clustering. arXiv preprint arXiv:1702.033: 1702.03544 2017.
Jason Lines and Anthony Bagnall. Time series classification with ensembles of elastic distance measures. Data Mining and Knowledge Discovery, 29(3):565-599) 2015.
Jason Lines, Sarah Taylor, and Anthony Bagnall. Time series classification with hive-cote: The hierarchical vote collective of transformation-beee baaed ensembles. ACM Transactions on Knowledge Discovery from Duta, 12(5), 2018
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. 2008 Eighth IEEE Interna- lional Conference on Data Mining, pp. 413-422. IEEE, 2008
Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, al. Multivariate time series imputation with generative adversarial networks. In Advances in Neural Information Processing Systems, pp- 1596-1607, 2018
Yonghong Luo, Ying Zhang, Xiangrui Cai, and Xiaojie Yuan. E2GAN: End-to-end generative ad- versarial network for multivariate time series imputation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3094-3100. AAAI Press, 2019.
Qianli Ma, Jiawei Zheng, Sen Li, and Gary W Cottrell. Learning representations for time series clustering. In Advances in neural information processing systems, pp. 3781-3791, 2019.
10

---

Under review as a conference paper at ICLR 2021
Naveen Sai Madiraju, Seid M Sadat, Dimitry Fisher, and Homa Karimabadi. Deep temporal clus- tering: Fully unsupervised learning of time-domain features. arXiv preprint arXiv: FrXXi:18022.0059, 2018.
Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. -STM-based encoder-decoder for multi-sensor anomaly detection. arXiv preprint arXiv:1607.00148, 0484, 2016.
Anish Mathew, Sahely Bhadra, et al. Warping resilient time series embeddings. In Proceedings of the Time Series Workshop at ICML, 2019.
Vladimir Nekrasov, Hao Chen, Chunhua Shen, and Ian Reid. Fast neural architecture search of com- pact semantic segmentation models via auxiliary cells. In Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 9126-9135, 2019.
Victor Olsavszky, Mihnea Dosius, Cristian Vladescu, and Johannes Benecke. Time series analysis and forecasting with automated machine learning on a national ICD-10 database. International journal of environmental research and public health, 17(14):4979, 2020.
Guansong Pang, Chunhua Shen, and Anton van den Hengel. Deep anomaly detection with deviation networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 353-362, 2019.
John Paparrizos and Luis Gravano. K-shape: Efficient and accurate clustering of time series. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp- 1855-18700 2015.
François Petitjean, Alain Ketterlin, and Pierre Gançarski. A global averaging method for dynamic time warping, with applications to clustering. Pattern Recognition, 44(3):678-693, 2011.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularl-sampple sampled time series. In Advances in Neural Information Processing Systems, pp. 2019.
Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus- Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. arXiv preprint 1906:1990..2640 2019.
Patrick Schäfer. The boss is concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery, 1955)150 2015.
Bernhard Schölkopf, John C Platt, John Taylor, Alex J Smola, and Robert Williamson. Estimating the support of a high-dimensional distribution. Neural computation, 13(7):1443-147)
2001.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1); 148-175, 2015.
Satya Narayan Shukla and Benjamin M Marlin. caappollatio-predictiin networks for irregularly sampled time series. arXiv preprint arXiv:1909.0778. 2019.
Ian Syarif, Adam Prugel-Bennett, and Gary Wills. SVM parameter optimization using grid search and genetic algorithm to improve classification performance. Telkomnika, 15(4):1502, 2016.
Arijit Ukil and Soma Bandyopadhyay. AutoSensing: Automated feature engineering and learning for classification task of time-series sensor signals.
D van Kuppevelt, C Meijer, F Huber, A van der Ploeg, S Georgievska. and VT van Hees. Mcfly: Automated deep learning on time series. SoftwareX, 12:100548. 2020.
Yu Weng, Tianbao Zhou, Yujie Li, and Xiaoyu Qiu. NAS-UNET: Neural architecture search for medical image segmentation. IEEE Access, 7:44247-44257, 2019.

---

Under review as a conference paper at ICLR 2021
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pp. 478 487, 2016.
Hang Xu, Lewei Yao, Wei Zhang, Xiaodan Liang, and Zhenguo Li. Auto-FPN: Automatic network architecture adaptation for object detection beyond classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6649-6658, 2019.
Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative adversarial net-
works. In Advances Neural Information Processing Systems, pp.
2019.
Bin Zhou, Shenghua Liu, Bryan Hooi, Xueqi Cheng, and Jing Ye. BeatGAN: Anomalous rhythm detection using adversarialyy generated time series. In ICAII, pp. 4433 4439, 2019.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Inter- national Conference on Learning Representations, 2018.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX A: BAYESIAN OPTIMIZATION
Let f(0) denote the objective function. Given function values during the preceding T iterations [f())): f(02)..... [Or]], we pick up the variable for sampling in the next iteration via solving the maxmization problem that involves the acquisition function i.c., expected improvement (EI) based on the postetior GP model.
Specifically, the objective function is assumed to follow GP model (Shahriari et al. 2015 and can be expressed as f() GP K). where m(0) represents the mean function. And K represents the covariance matrix of {0: ) namely, K; = k(0;, f2), where k(::)) is the kernel function. particular, the poster probability of f() at iteration + 1 is assumed to follow Gaussian distribution with mean (=()) and covariance o² "(")), given the observation function values 2:
(1)0) K²" K o v. a² (0)) x00))=(00r00000 (0) K K + -
(9)
where x is vector of covariance terms between 0 and (0.) and o denotes the noise variance. We choose the kenel function ARD Matérn 5/2 kernel (Shahriari et al. 2015) in this paper:
10 ppp p') = =o (x))(- + v5r + 5 3²
(10)
where p and p' are input vectors, n = Σ i=1 p. D /r3 and a = o..) are the GP hyperparameters which are determined by minimizing the negative log marginal likelihood log(v/ eb)
min log det K + T. + w ("(KK T
(11)
Given the mean p(0") and covariance (²(")) in 99, 8r+1 can be obtained via solving the following optimization problem:
0 = arg max ETT(A)
--0") arg max + 000" o -0) ((0))
(12)
g*
where y+ =fx[((( ((2) f(Or)) represents the maximum observation value in the previ- ous T iterations. is normal cumulative distribution function and ø is normal probability density function. Through maximizing the EI acquisition function, we seek improve f(1++1 monotoni- cally after each iteration.
13

---

Under review as a conference paper at ICLR 2021
B APPENDIX B: DETAILED VERSION OF TIMEAUTOML
Algorithm 2: Detailed Version of TimeAutoML
Input: L: pre-defined threshold for maximum number of iterations. B: pre-defined threshold for maximum Bayesian optimization iterations. at and & pre-defined Beta distribution priors. fupp and flow: pre-defined upper bound and lower bound to objective function f.
Set: (t) E R (1) E Q the cumulative reward and punishment for ith module for the t iteration, specifically, (1) ag and [, Po-
for 1......... do
for i = 1.2, .M do
Sample w, (t)
end
Obtain the TimeAutoML pipeline configuration by solving the following optimization problem:
M
maxmize K
Σ (k:) w, subject to k; E {0,119; 1'k, = 1, e [M],
for b = 1,2, ... do
Hyperparameters optimization: Update hyperparamettrs according to Bayesian optimization framework given in Appendix and the obtained objective function value is denoted as f(K(t), @x(t)), where ((t) denote the module options in th iteration and (,, denote the hyperparameters in tthh Bayesian optimization iteration in 1th iteration.
end
Update Beta distribution of the options in the configured pipeline:
1. Let f(t) = m fK{(t), (t)., 1,... B} denote the performance of TimeAutoML. model the th iteration.
2. Compute the continuous reward r:
ma f(t)-flow Jupp flom
3. Obtain the binary reward Bernoulli()). for 1,2, .. .M do
at = = k. ((t+))=B((t) + k -1-11
end end
Output: A TimeAutoML model with optimized hyperparameters.
It is seen that TimeAutoML. consists of two main stages, i.e., pipeline configuration and hyperpa- rameter optimization. In every iteration TimcAutoML, Thompson sampling is utilized to refine the pipeline configuration at first. After that, Bayesian optimization is invoked to optimize the hy- perparameters of the model. Finally, the Beta distribution of the chosen options will be updated according to the performance of the configured pipeline.
In the experiment, the upper limit to number of entire TumeAL iterations. BO iterations are set 40 and 25 respectively. The Beta distribution priors respectively on 10 and 30 10.
14

---

Under review as a conference paper at ICLR 2021
C
APPENDIX C: SEARCH SPACE: OPTIONS AND HYPERPARAMETERS
Table A1: Modules, options, and hyperparameters of TimeAutoML.
Module
Options
Hyperparameters
Scaling
Continuous hypergarameten
Data augmentation
Shifting
Discrete hyperparameters
Time-warping
Discrete hyperparameters
RNN
Discrete hyperparameters
Encoder
LSTM
Discrete hyperparameters
GRU
Discrete hyperparameterss
None
Atteation
Self-atteation
None
RNN
Discrete hyperparameters
Decuder
LSTM
hyperparameters
GRU
Discrete hyperparameters
EM Estimator
Gaussian Mixture Model
Discrete hyperparameters
Relative Eaclidean distance
None
Similarity Selection
Casine similarity
None
Both
None
Estimation Network
Multi layer feed forward neural network
Discrete hyperparameters
Auxiliary Classification Network
Multi-layer feod-forward neural network
Discrete hyperparameters
C.I DATA AUGMENTATION
increasing or decreasing the amplitude of the time series. There are two hyper- parametes, the number of data augmentation samples Naug 100) and the scaling size hamp E [0.5, 1.8).
Scaling:
Shifting: cyclically shifting the time series to the left or right. There are two hyperpa- rametes, the number of data augmentation samples Naug [0, 100) and the shift size hshift E 1-10 10).
Tarp-ng:
randomly "slowing down" some timestamps and "speeding up" some timestamps. For each timestamp to "speed up". we delete the data value that timestamp. For each timestamp to "slow down", we insert a new data value just before that timestamp. There are two hyperparametes, the number of data augmentation samples Naug E [0, 100] and the number of time-warping timestamps t E [T/10,T/4]
C.2 ENCODER
For all encoders, i.e. RNN, LSTM, and GRU, there is only one hyperparameter, i.e., the size of the encoder hidden state. And we assume it is no larger than 32, i.e., hene E [1.
C.3 ATTENTION
Self-attention mechanism has been considered in this framework.
C.4 DECODER
For all decoders, ie. RNN, LSTM, and GRU, there is only one hyperparameter, i.e., the size of decoder hidden state doee For univariate time series, assume it is no larger than 32, i.e. h [1, 32). And we assume dee E [n font 4 n feat] for multivariate time series, where n feat represents the dimension of the multivariate time series.
15

---

Under review as a conference paper at ICLR 2021
C.5 EM ESTIMATOR
In this module, we provide a statistical model GMM to carry out latent space representation distri- bution estimation. There is one hyperparameter, the number of mixture-component of GMM. EM algorithm is used to estimate the key parameters of GMM.
C.6 SIMILARITY SELECTION
We offer three similarity functions for selection, including relative Euclidean distance, cosine simi- larity, or the concatenation of both.
C.7 ESTIMATION NETWORK
We utilize multi-layer neural network as the estimation network in our pipeline. There are two hyperparameters to optimized, i.e., the number of layers elayer E [1, 5] and the number of nodes in each layer emode E 128). V1 I i S elayer.
C.8 AUXILIARY CLASSIFICATION NETWORK
We utilize a multi-layer neural network as the auxiliary classification network in our pipeline. There hyperparameters to be optimized, i.e., the number of layers dayer E [1, 5] and the number of nodes in each layer eooee E [8, 128), v1 S i S clayer
16

---

Under review as a conference paper at ICLR 2021
D APPENDIX D: RESULT
D.I ANOMALY DETECTION PERFORMANCE FOR UNIVARIATE TIME SERIES
Table A2: AUC scores of TimeAu and state-of-the-an anomaly detection methods on UCR time series dataset when time series are regularly sampled (8 = 0). Bold and underlined scores respectively represent the best and second-best performing methods.
BEaRIAN DACAIM GRILED
TXF
Adiac
0.3155
ArroHHead
0.9676
0,892
0.7923
0.872
0:4008
0.7899
0,442
Beef
0.8353
0.4167
ReaileFly
0:35
0.4
BirdChicken
0.8
0.9
0.6
01.5
0.4
Car
0,6233
0.3346
0.2855
0.4231
CBP
0,6573
0.9919
0.7985
0,8006
0.6408
0.9399
ChlorineC
0.6653
0.5672
0.529T
0.5724
0.5048
0.5449
0.5899
CioCECGTosoo
0.5951
D.6761
0.9966
0.7908
0.5959
0.6749
0:9641
Coffee
0.333
0.75
0.7167
Compusers
0.8354
0.244
0,738
0.6533
0.7686
0.468
0.5714
CricketX
0.9744
D8754
08123
0.7892
0.7405
0.6282
CricketY
0.954
0.9228
039977
0.331
0.8161
0.9827
Cricke7Z
0.9583
0,8255
0.6897
0.8333
0.6521
0.6349
DiatonSizeRedaction
0.8571
0.9913
0.9946
0.9912
0.8331
0,333
0.5626
08333
0,5342
0.6721
0.6198
0.6244
0.76
DisullPtalanxTV
0.143
0.9643
Eariquakes
0.8418
0.7421
0.6221
0.5599
0.8003
0.5671
0.5428
DCG5000
0.9981
0.5648
0.9923
0.4755
0.8958
0.9301
0,5136
ElectneDevices
0.8427
0.5626
0.:31
0.1122
0.7588
0.5518
0.552
FaszAll
0,7674
0,9821
0.9841
0:9844
0.7677
0.7847
Faceffour
0.9236
0.9236
0.4286
UCR
0.6368
0.9226
0,9065
0.8756
0.6750
0,8256
HofdyWoods
08187
099890
0.9901
1
0.3643
0,807
Fish
09704
08523
0
0:6212
FundA
0.629
0.601
0,5896
03619
0.6306
0/9933
0.1708
FuniB
0.008
066212
0.5999
0.0011
0.5949
0.5949
0.4971
Ham
0.5961
0.8579
0.6556
037677
0.6358
0.6348
0.0296
HondChllines
0,8808
0.9131
03524
0.5679
0.7349
0.7415
Haptics
0
0.8579
0.7266
0.6688
0.5826
0.6674
0.5167
Hering
0.8333
0.8006
0.7231
0.7105
0.8099
0.65
07147
0.4223
0.6254
InserrWingheeSSond
0.21
0.6574
0.9055
0.9735
0.754M
0.7861
0:9333
LargeKinchenAgglaanee
0
0,720
0.5887
0.2975
(1.5025
0.5289
Lightning?
0,9242
06661
0.7574
0.5758
0.909
0.7197
Lighnning?
1
1
0,4211
Mallat
0.99%
0.6639
0.9079
0.9701
0,5728
0.8377
0.8811
Meat
0.975
0,7001
0.054
0,9673
Midkke/hullaxDullii Coment
0,9242
0,7355
0,4001
0.7011
0.
0.5725
MikKI:PhallaxTTWW
0
09167
1
0.8611
0,9028
091667
0.9722
OlivOOii
09187
09167
099167
0.9583
OSULeea
0.8964
08112
03802
038302
0.335
0.6823
Conle
0.7423
0.2949
0,4321 1
02520
1.5192
0.6629
Phoneme
0.9148
0.7054
0.5043
Plane
ProtimalPhalaanCDa Autineege
0.0024
0.975
02225
0.775
0.9255
0.6882
023825
07221
0
0.7674
Provinelltoolamss TW
0.8664
199601
018890
0.9311
RefigeraiioDDevices
0.9323
0.7483
0.7264
0.522
0.5150
1
0.1665
0.5714
ScreenType
0,8572
0,7453
07455
03443
0,4021
0.5289
ShapeletSim
0,7431
0.5611
1
ShapesAll
SmallKiichheAyyliann
0.9586
0.7151
09590
07320
1,9621
0.6812
0.6563
Sony Suuffieel
0.9998
0.6866
09834
1002
0
0.9
0
0.9907
0.6211
01823
0.5981
0.2152
Steef Light Curves
0.9135
0.5448
018356
8.8161
0.5025
Strawberry
0.7805
0.6766
06769 1
0.5699
0.8144
0.4738
0.4433
SwedishLeaf
0.9913
0.9304
06963
0.5758
0.6566
0.6212
0.6212
Symbols
0.9987
0.7669
(9888
099922
0.9942
SyntheieCContel
0.756
0.6344
0.5299
TeeSegmennttiiol
0,7112
0.8119
06264
0.5726
0.5226
0.6708
TeeSegmenttioon2
0.9907
0.8255
893588
0,3743
0.6157
0.5617
2020
Trace
1
0.9211
0.3211
0.9959
0.6485
08759
06941
0.8641
0159667
0.8274
TwoPuttens
0.9996
0,3899
039936
0.7163
0.9299
(1.411
0.7371
WWWaeeGenaae ibrary All
0.9941
0.6467
00035
0.9888
0.8106
0.9343
0.7896
U (WaveGestureiiraa X
0.7477
0.6136
06563
0.6766
0.5626
0.4696
U U'WaaCiesnaeell ibrrry
0.9845
0.6256
00744
05626
0:9357
DA687
0.9883
0.9602
0.9101
0.8671
Waler
0.9983
0.4947
0.9586
0.6761
0.9416
0.5599
Wine
0.9536
5
0.7531
0.429
0.6689
WordSynoeyms
0.9929
0.7862
09621
09621
0.8245
0.8226
0.8442
Worms
0.9%6
0.8485
037677
0.7126
0.5341
0.5896
WormeTwoClass
0.95%3
0.9355
0.6937
0.7501
0,4021
0.4432
Yoga
0.7538
0.5623
0.6883
0.6766
0.5884
0.5421
0.6257
17

---

Under review as a conference paper at ICLR 2021
Table A3: AUC scores of TimeAutoML and state-of-the-an art anomaly detection methods on UCR
time series dataset when time series are irregularly sampled (3 = 0.5). Bold and underlined scores respectively represent the best and second-best performing methods.
dabaet
TimeAutoML Lateni ODE BeRGAN
DAGMM
GRU-ED
IF
LDF
0.25
013355
ArroHHead
0.9816
0.8095
0.7633
0.8671
0.3478
0.7547
Reef
0.4167
Beetlekly
0.9
0.75
0.35
0.35
BindChicken
0.
0.6
0.
0.4
Car
0.6154
0.3027
0.6538
0.2692
0.4231
CBF
0.9933
0.6322
0,8725
0.7819
0.7638
0.5281
0.8549
ChlorineCoonccsraatt
0.5954
0,5669
0.4916
Ci
0.6679
0,9037
0.7881
Cofiee
0,4333
Computers
0,9188
0.6613
0.72
0,46
0.5714
CricketX
0.9743
0.6751
0.8077
0.7051
0.7572
0.6282
CricketY
0.9997
0.908
0.9699
0.3966
B.6997
0.8161
0.7089
CrickeZZ
1
0.9166
0.8125
0.6875
0.8333
0.6458
0,4575
DattnnizzeRecsttion
0,9793
0.8467
0.8104
0.0772
0.4848
0.9516
0.8395
0.7821
0.333
0.764
0.5167
0.4941
DisallftalanxTW
0.9107
0.6613
Eanhquaks
0,8191
0.6628
06083
0.2214
0.5841
0.5350
0.5248
RCG5000
0.9765
0,5422
08817
0.6995
0.8537
0.2008
0.5142
Eleetteeeeviesss
0.7481
D.5621
0,9744
0.6679
0.5361
0.5331
FaceAll
0.6545
0,8552
0.6671
0.6892
0.8914
0.6528
FaceFour
0.9633
0.6479
0.6429
0.4286
UCR
0.9491
0.6474
0,629%
0.6276
Filly Words
0.997
0.74%
0.9688
019855
0.8728
0.7149
0,8409
0.7727
0.4545
0.6212
FunlA
0.6157
0.00ST
02122
0.5414
0.6005
04955
0.4644
FundB
0.5808
0,6164
052222
0.387
0.5489
0.5855
0.6971
Ham
0
041726
0.7556
8.3277
0.02206
6
0.0296
HandOulines
0.5004
0.643
0,7267
0.4409
0.5425
0.6143
0.6266
Hoptics
0.9353
0.115
0.5527
06641
0.5233
0.5833
0.5157
Herring
0.9642
0.964
0.7895
0.6191
0.5855
0.7105
0.7105
InliaeSkare
0
0,5045
0.5477
0.4000
0.503
InsertiinhaatSouud
D46155
0.0044
0:7332
LangoKichenAyggiang
0,8375
0.7696
05433
0.560
0.5788
0
Lightning?
0,9015
(P1140
0.7475
0.6545
0,999
0.0934
Lightning?
1
0:2211
Mallat
0,9888
0,6566
0.8988
0.9687
0.5512
0.0355
0.881
Mest
0.675
0.975
0195
0.700
2000
0,9081
0.9655
0.418
0.8553
0.5571
0.431
0.7573
0.7195
0,4522
0,273M
0.5750
MiddlePhalanCWW
0.875
0.8035
0.5524
09028
08889
09553
0:7222
0,8333
08148
0
09008
0:7222
OliveChi
0,5883
0.558
0.4107
OSULea
0.9955
0,8318
0,7184
0.7577
0.8220
03355
0.6659
Clenee
0,6819
0.6671
04225
03592
033576
013446
014864
Phoneme
0,8898
0.6676
0.6355
036311
0.7531
0,4561
0.5692
Plane
ProtimulFhullmee DatalannOOtiinAgGGGrrr
0.723
0,861
0.65
0.72
0,5299
0.6998
0.5625
0
0-8227
ProximulPhakmmmT TW
0.8333
0.948
09903
0842
0.5833
RefiggeratioDDevies
0,5799
0.6738
D. 17917
0
5268
4665
0-4284
SyreenType
0.5446
0.954
0.7153
0,6137
1.8012
0,192
0.5714
ShapeletSim
0.9278
0.7558
012144
0.5056
ShapesAll
0.9
0.55
0.35
0.95
SmallKK Apppliiancs
0.9538
D.6812
0.2318
0.9117
0.6561
Sony B
0.6633
0.001
0.9005
0.5227
0.7400
Sony ABBoAllBettbttttteee
0.5206
06114
0.5556
0:6126
Stril ighhCurves
0.9118
0.5489
03874
0.8156
0,9028
Strawherry
0.7427
0.673
0.9980 1
0.546
014644
0.4433
SwedishLeaf
0.9689
0.9318
045666
0.5758
0.3919
0.1911
0.4734
Symbols
0.9961
0.745
0.082
0.521
0:3003
0.7958
0:9636
SyntheticCunteel
0.968
67704
066
0.5836
0.36
0.44
ToeSegmentarionn
0,3917
0.2055
TeeSegmentarion2
0.9383
0,7624
11980
0/4348
0.5185
0.8943
Trace
0
0.4211
CodEGG
0.3551
0.5865
0.512
0.5429
0.5184
TonPPuttens
0.9981
0.5677
1886
06994
8.81766
0.5271
0.7253
0.9965
0.6699
099656
0.9991
0.8058
0.92118
0.7575
0.7011
0.6428
0.5054
(155722
0.4696
0.9839
0.6152
0.9744
0.6
L'aaeCisnaee ibrar?
0.9944
0.0899
01964
0.8527
0.9572
0,4222
01915
0.5919
0.531
Wine
0.9477
03024
0.3804
0.6778
0.6246
WordSynonyms
0.9687
0.723
09582
09287
0.7936
0.8107
0.8357
Worms
0.9924
08889
06667
0-2045
033533
0.5795
WormeTweClas
0.9455
0.8
063307
0.6855
0.7551
0.375
0.4432
Yoga
0.7161
0,5726
0.633
0.3919
0.5621
0.5359
0,5079
18

---

Under review as a conference paper at ICLR 2021
D.2 ANOMALY DETECTION PERFORMANCE FOR UNIVARIATE TIME SERIES (CONTAMINATED TRAINING DATASET)
Table A4: AUC scores TimeAutoML when univariate time series training datasets are contami-
nated with 5% and 10% anomaly samples.
Ratio
ECG200
ECGFiveDays
GunPoint
InlyPowerDemand
Medicallmages
MoteStrain
0%
0.9349
0,9719
0.9093
0.8811
0.7693
0.9186
5%
0.9305
0,9697
0.8994
0.8624
0.7598
0.9104
10%
0.9271
0,9624
0.8902
0.8466
0.7455
0.9043
Decline
0.78%
0.95%
1.91%
3.45
2.38
1.43%
10
54
GRLARD
AUC
3.-
LOF
0.3
0.7
01
02
0.1
0.5
266
0.
0.4
0.6
Rate 8
Rate
ECG200
(b) GunPoint
(c) Pealynd
0.00
Timeduto.
0.75
0.70
0.65
0.6
05
(d) Medicallmages
(c) MoteStrain
Figure Al: AUC scores of TimeAutoML and state-of teffthee-r art anomaly detection methods on univari- datasets when irregular sampling rate 9 varies from 0 to 0.7.
19

---

Under review as a conference paper at ICLR 2021
D.3 ANOMALY DETECTION PERFORMANCE FOR MULTIVARIATE TIME SERIES (CONTAMINATED TRAINING DATASET)
Table AS: AUC scores of TimeAutoML when univariate time series training datasets are contami-
nated with 5% and 10% anomaly samples.
Ratio
FingerMovements
LSST
RacketSports
PhonemeSpectra
Heartbeat
EthanolConcentration
0%
0.9643
0.7827
0.9826
0.8685
0.7703
0.8561
5%
0.9554
0.7643
0.9796
0.8623
0.7604
0.8425
10%
0.9388
0.7559
0.9724
0.8601
0.7527
0.8379
Decline
2.55%
2.68%
1.02%
%
1.76%
1.82% %
Timefaoo
Lewm GDT
0.75
DAGUM
TimsturM Latert ODE
0.70-
0.51
GAU-EO
AUC
DAGNM
AUC
CRU
LOF
0.50
0.1
5.0
01
6.2
0.3
0.4
0.5
06
6.7
0.0
63
0.2
03
0.4
65
9.
07
60
5.1
02
0.3
0.4
055
0/
47
Rate 0
Rate
Rate
(a) FingerMovements
(b) LSST
(c) Rackettpoors
Timetutont
TimoknaML
Latent ODE
Laten
BourCAN
DAGIN
55
DAGHH
BestGAN
GRU-ED
GRUAED
AUC
0.7
Lor
LGT
0.5
0.0
0.
0.5
0.6
07
03
(d) PhonemeSpectra
(c) Heartbeat
Figure A2: AUC scores of Ti and state-of-the-art anomaly detection methods on multi- variate datasets when irregular sampling rate 3 varies from 0 to 0.7.
20

---

Under review as a conference paper at ICLR 2021
E APPENDIX E: ILLUSTRATION OF IRREGULAR SAMPLING
We provides an illustrative example to demonstrate how TimeAutoML. remain robust against the irregularities in the sampling rates. Both regularly and irregularly sampled time series (3 0.5) presented in Figure A3 And for the purpose of illustration we assume the normal time series is Sine curve. The anomalous time series is obtained via adding noise to the normal time series over short time interval. The irregularly sampling rate is set as 0.5 in this example.
As evident from the figure, the unusual pattern of the anomalous time series preserves after the irregular sampling. And such unusual pattern appears to be different from the distortion caused by irregular sampling- Due to the special characteristics of TumeAutoML, e-g., embedded LSTM encoder and attention mechanism, it is capable of learning both the short term and long term cor- relations among the training time series and therefore can detect such unusual pattern even in the presence of irregularity in the sampling.
1.0
Normal
Irregular Sampling
-0.5
-1.0
20
40
50
70
10
20
30
40
1.0
Abnormal
1.0
Abnormal
0.5
0.0
0.0
abnormal
abnormal pattern
-0.5
-1.0
-1.0
20
70
Figure A3: Irregular sampling on Sine curves.
21

---

Under review as a conference paper at ICLR 2021
F APPENDIX F: RESULTS OF TIME SERIES CLASSIFICATION
The representation generated by TimeAutoML can be used to carry out time series classification. We combine the representation generated by TimeAutoML with Gradient Boost Decision Tree (GBDT) to carry out time series classification. Followed Franceschi et al. (2019). we compared our method with four best supervised state-of oatte-o-tth-ann art model, HIVE-COTE (Lies et al. 2018). ST (Bostrom & Bagnall (2177) EE (Lines & Bagnall 2015), BOSS (Schäter 2015), which are shown in Table A6 It is worthy mentioning that both of ML and Franceschi et al. (2019) are unsupervised time series representation learning models. Accuracy score is used as the metric to evaluate the
classification performance. Besides, the results of DTW (Lei et al. 2017). which is an unsupervised model, and the results of GBDT on original input have also been posted in Table
For easier comparison, we compute the average ranks of TimeAuto.M and the benchmarks, which are shown in Figure A4 is seen that our method is globally the second-best one (with average rank 2.57), only beaten by HIVE-COTE (2.42), which is a powerful ensemble method using many classi- fiers in hierarchical voting structure. And we observe that, GBDT (6.63) original input has a poor performance, while combined with representation generated by TumoMI can improve the classification performance significantly, which indicating the important role of AutoM..
Table A6: Accuracy scores of TimeAutoML. and the state-of-the-art classification methods UCR time series datasets when time series are regularly sampled (3 = 0). Bold scores represent the best performing method.
Dutaset
Time AutoMI
Franceschi et all
2019
DTW
ST
BOSS
HIVE-COTE
EB
GBDT
AnooWeed
01346
0.757
01834
0.363
0.811
0.629
Reef
0.8
0.7
0.633
0.9
0.8
0.933
0.633
0.633
BeelleFly
0.9
0.7
0.9
0.9
0.95
0.75
0.75
BirdChicken
0.95
0.85
0.8
0.9
Car
0.733
0.817
0.733
0.917
0333
0.867
0.333
0.567
CRF
0,986
9994
0.988
0.908
Coffee
1
0.964
1
0.964
Computers
0.736
0.356
0.708
DiaomSiieeeeduttoo
0
0.993
0:907
0.925
0.911
0.941
0.944
0.941
0.717
0,775
0.728
0.772
0.728
0.728
Phalas Outtinagpu
0.513
0.734
0.77
0.77
0.748
0.763
0.091
0.772
Distal Phalan
0.719
0.676
0.50
0.662
0.676
0.683
0.647
0.662
Earihquakes
0.827
0.748
0,719
0,741
0.748
0.748
0.741
0.748
ECG200
0.93
0.9
0.77
0.83
0.87
0.85
0.88
0.84
BUGyiveDays
0.79
0,708
0,984
0.787
FaceFour
0852
0.875
0.83
0.852
0.955
0.909
0.682
GunPoint
0.991
0 0903
0.991
0.84
Iam
0.771
0.695
0.667
0.685
0.667
0.667
0.571
0.676
Haptics
0.455
0.523
0.461
0.519
0,293
0,461
Herring
0.766
0.578
0.531
0.672
0.547
0.688
0,578
0.672
Pewerond
0.975
0.925
0.95
0.948
0.909
0.063
0.962
0
LungeKiichemAppplaaee
0.899
0.818
0.795
0.859
0.765
0,564
0,811
0.579
Lightning?
0,660
0.918
0.860
0.738
0.336
0.82
0.585
0.705
Lightnigg
0.767
0.795
0.726
0,726
0.685
0.74
0.767
0.616
Meat
1
0.95
0.933
0.85
0.9
0.933
0.933
0.9
0.356
0.314
0,794
0.78
0.784
0.801
0.656
0.656
0.5
0.643
0.545
0.597
0.558
0.532
0.545
0.571
0,552
MoseStrain
0.897
0.871
0,835
0.897
0.879
0.933
0.583
0.796
OliveQui
0.833
0.667
0.9
0,867
0,833
Plane
0.9
0.838
0,888
0.854
0.5 0.805
0.844
0.834
0.859
0.505
0.854
RelrigeraionDeviies
0.579
0.517
0.461
0,581
0.199
0.557
0.437
0.199
ShupeletSim
0.817
0.65
0.956
1
0.817
0.489
Sony AlBORobosSuriael
0.965
0.897
0.725
0.844
0.632
0.765
0.701
0.805
Sony ABOoooosSsssrr
0.963
0.334
0.831
0.934
0.859
0.928
0.878
0.752
Symbols
0.549
0.677
0.974
0.96
0.714
SyntheticControl
0.983
0.63
0.991
0.983
0.967
0.997
0.99
0.883
TueSegmeentttiom]
0.56
0.929
0.57
IosSegmentazion22
0.577
0.335
0,838
0.908
0.962
0.954
0.892
0,577
Trace
0.59
0.8
Twol cu/FCG
0.949
0.997
0.905
0,997
0.981
0.996
0.971
0.736
Wine
0.944
0.87
0.574
0,796
0.741
0.778
0.574
0.537
Woems
0.792
0.714
0.584
0.74
0.558
0.558
0.662
0.468
WoomsTwoCllss
0.44
03188
0.623
0,831
0.311
0.779
0.688
0.545
22

---

Under review as a conference paper at ICLR 2021
1
2
8
HIVE-COTE 2.42
6.63
GBDT
Our+GBDT
2.57
5.78
DTW
Franceschi 2019
3.29
4.62
ST
3.69
3.86
BOSS
Figure A4: The average rank of our method and state-of-the-an models. Our method is an unsu- pervised representation learning method, combined with other classification algorithm can reach an excellent classification performance. We combine GBDT with model in the experiment. A lower value of average rank represents better classification performance.
G APPENDIX G: PARAMETER SENSITIVITY
In our experiment, Å and À are two weighting factors governing the trade-off among three parts in objective in (8). We analyzed the sensitivity of Y₁ and A2 ECG200 dataset (average over 10 trials) on anomaly detection task, which shown in Figure A5 is seen that, our approach is not very sensitive to the choice of λι and Xz. Nevertheless, too large λ friendly to the optimization our model. Based on this observation, we set the X 0.0001 and
small X2 may not be
A2 1 in our experiment.
1.00
0.98
0.96
0.94
AUC
0.92
A1 0.00001
A1 0.0001
0.90
A1 0.001
A1 0.01
0.88
A1 0.1
0.86
A1 1
0.001
0.01
0.1
1
10
A2
Figure A5: The sensitivity analysis about weighting factors X and A2 for each combination of A1 E {0.00001, 0.0001, 0.001, 0.01, 0.1, 1} and Xu E (0.001, 0.01, 0, 10).
We also analyzed the sensitivity of Beta distribution priors and B, in our AutoML framework, which is tested ECG200 dataset (average over 10 trials) on anomaly detection task and the results shown in Figure A6 It is seen that, the choice of Beta distribution priors oi and 3 will not
23

---

Under review as a conference paper at ICLR 2021
have significant impact performance, indicating that the model is not very sensitive to the choice of Beta distribution priors.
1.00
0.98
0.96
0.94
AUC
0.92
ag 5
00 10
0.90
ao = 15
ag = 20
0.88
ao 25
0.86
do 30
5
10
15
20
25
30
Bo
Figure A6: The sensitivity analysis about Beta distribution priors @n and B for each combination of Qg E 15, 10, 15, 20, 25, 30} and Bo {5, 10, 15, 20, 25, 30).
H APPENDIX H: MOTIVATION OF INVOLVING GAUSSIAN MIXTURE MODEL
The idea of TimeAutoML is to map the original time series into the latent space representation and the generated representation can be used to carry out many downstream machine learning tasks, such anomaly detection, clustering, classification and so on.
We employ the GMM model to characterize the latent space representation distribution of the input data for two reasons. First, GMM is flexible and powerful model that has been proved to be capable of approximating any continuous distribution arbitrarily well under mild assumptions. Second, once we obtain a GMM model using the training data, we can calculate the distances between an input time series and the centroids of GMM in the latent space, which are proportional to the sample energy function. Therefore, the GMM model, in combination with sample energy function, help to characterize the level of abnormality of an input time series. The time series that is far away from the centroids of the GMM in the latent space will be deemed as an anomaly.
24


Reviews:
[{"review_id": "aDKFKTlQFDM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I carefully read the paper and it was interesting.\nThe results seem promising; however, the novelty and motivations are not that satisfied.\nDetailed comments are as follows.\n\n1. Challenge 1 (Trial / Error)\n- I think the first challenge (trial/error) is usual and many hyper-parameter tuning algorithms and tools are available. For instance, https://cloud.google.com/ai-platform/training/docs/using-hyperparameter-tuning\n- I am not sure how the proposed method is compared with other AutoML frameworks and hyper-parameter tuning methods?\n- Also, I am not sure what is \"special\" for AutoML for time-series technically.\n- The methodology is the same with previous Thomson sampling and Bayesian optimization. \n\n2. Challenge 2: Irregularly sampled time-series\n- What do you think if we combine time-series imputation and previous time-series representation learnings?\n- In that case, most time-series models can be directly applicable after imputation. \n- Note that there are various imputation and interpolation methods for time-series data.\n- I think this can be an important baseline to be compared with.\n- Also, what is the proposed component for dealing with irregularly sampled data?\n\n3. Challenge 3: Contrastive learning\n- I am not sure why contrastive learning is good for time-series.\n- And why does it guarantee suboptimal performance?\n- If you have some motivations, please provide it in the revised manuscript.\n\n4. Figure 1\n- It is not easy to understand the key ideas in figure 1.\n- There are too many equations but only one sentence in the caption.\n- So, in the revised manuscript, it would be great to improve the readability of this figure with enough caption for the readers.\n\n5. Motivations\n- Equation (3): Can you explain why you select y_i as the representation? It is not well motivated.\n- EM algorithm: Can you explain why EM algorithm is necessary for this?\n- RNNs: Why the RNNs can be used for the encoder and decoders?\n- Many selections in this paper have no underlying motivations even though there are many \"blocks\" in the proposed method.\n\n6. Novelty\n- I am not sure what is the novelty of this paper.\n- As can be seen in equation (8), it seems like the proposed model are some combinations of previous methods.\n- AutoML is definitely not the novel part because the authors utilize Thomson sampling and Bayesian optimization.\n- It would be great if the authors can clarify the novelty of this paper.\n\n7. Lambda 1 and Lambda 2\n- How to optimize those two critical hyper-parameters?\n- It would be good to add more ablation studies to check how much gain does each component in Equation (8) provide. Currently, I can only see the results without L_self.\n\n8. Datasets\n- In the title, abstract, and introduction, the authors highlight the \"multi-variate\" time-series. \n- However, in the experiments, the authors provide the results on mostly univariate time-series dataset.\n- If the proposed model is scalable, it would be great to show the results on various multi-variate and highly irregular time-series datasets. \n- Here, it would be better if you use the dataset which already has missing components.\n\n9. Fair comparison\n- It is unclear how the authors optimize the hyper-parameters of other methods.\n- If the proposed method is well optimized (among various hyper-parameter sets) but the baselines do not, it is not a fair comparison.\n\n10. Computational load\n- AutoML takes much more computational load. \n- It would be good to quantitatively analyze the computational complexity of the proposed method compared with other methods.\n\n11. Benchmarks\n- I think the benchmarks are too limited. \n- The authors should provide various \"time-series\" clustering and anomaly detection algorithms as the benchmarks.\n- For instance, various algorithms in https://blog.statsbot.co/time-series-anomaly-detection-algorithms-1cef5519aef2.\n- Note that K-mean, GMM type of things are not designed for time-series.\n\n----------------------After reading the rebuttals-------------------------------\n\nThanks authors for the detailed response to my review.\nUnfortunately, the responses are not satisfactory for me to increase the score.\nTherefore, I stand on my initial score (4) as my final score.\nThe below are the reasons for this.\n\n1. The authors failed to provide the comparison with other AutoML papers on time-series.\n2. Imputation/Interpolation is a basic data preprocessing step. Also, the additional complexity for the imputation is marginal (especially compared with AutoML). Therefore, excluding the comparisons with imputation is hard to be accepted.\n3. There are a bunch of self-supervised learning methods. It is unclear what is the motivation that the authors utilize \"contrastive learning\" as the self-supervised learning methods. \n4. Also, motivations of many decisions in this paper are still missing. I think I am not the only person that raised this problem.\n5. I asked for \"quantitative\" analyses on computational complexity. However, I cannot find the \"quantitative\" analyses in the revised manuscript and rebuttals for the computational complexity.", "extra_scores": {}, "extra_text": {}}, {"review_id": "JA186rnDH2A", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an autonomous representation learning framework for multivariate time series with irregular sampling rates. Specifically, there are three major components proposed in the framework. 1) An AutoML solution for hyperparameters optimization under Bayesian framework is proposed to automatically seek optimal network structures and parameters. 2) Variational autoencoders based on generative approach and attention mechanism is employed to learn the semantic representation of time series with limitation of irregular sampling. 3) A sample energy function derived from Gaussian mixture model attempts to depict the level of anomaly of sliced time series.\n\nThe proposed unsupervised time series representation learning is novel in terms of AutoML methodology and contrastive learning approach. However, there are some concerns as follows:\n\n(1)\tThe motivation for involving GMM model is not clearly discussed. The proposed sample energy function to denote the level of anomaly is not properly motivated as well.\n\n(2)\tThere is no discussion about batching and computational complexity. Because of irregular sampling, one computational difficulty comes from the observation that times can be different for each time series in a minibatch. Furthermore, the overall AutoML pipeline without human experience suffers high computational complexity and is usually time-consuming in practice.\n\n(3)\tThe experiments are not convincing. First, there is no sensitivity analysis of the hyperparameters employed in the proposed framework. Then, the datasets are not all carefully selected. In this paper, datasets are generated from normal time series datasets with a specially designed irregular sampling policy. This paper lacks experiments on irregularly sampled time series data from real scenarios. For example, PhysioNet [1] (Physionet Challenge 2012 dataset) dataset is a widely used benchmark to handle irregularly sampled time series data in many previous papers. Consequently, the performance evaluation on PhysioNet or similar datasets should be included in this paper.\n\n[1] Silva, Ikaro et al. “Predicting In-Hospital Mortality of ICU Patients: The PhysioNet/Computing in Cardiology Challenge 2012.” Computing in cardiology vol. 39 (2012): 245-248.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "96xIeYoxzX5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes an AutoML framework for multivariate irregularly sampled time series. To achieve this, the proposed framework integrates different modules: data-augmentation self-supervised loss (Equation 7), an anomaly detection loss (Equation 5), and a reconstruction loss. Besides, hyperparameters and model’s configuration is optimized by using AutoML (including Bayesian optimization). The model is evaluated on the well-known time-series datasets UCR and UAE. \n\nComments:\n\n***motivation***  this works combines different techniques including self-supervised learning and hyperparameter optimization. But I cannot clearly find what’s their main contributions in this work since all of these techniques used in this work seems not to be new. I encourage authors to clarify their contributions formally. \n\n***irregularly sampled ts*** since this work is for irregularly sampled time series (see title), this work should consider how to model the irregularly sampled time series. But I cannot find any related mechanism to model or deal with irregularly sampled time series in the model section. Besides, as I know, both of the datasets UCR and UAE are not standard datasets for irregularly sampled time series. And in your experiments, you construct that time series by using an irregularly sampling rate \\beta. This may be problematic since it should be missing data, but not true irregularly sampled time series. There may be two different topics in the time series community. \n \n***representation learning*** since representation learning aims to learn some good representative features that can be easily transferred to other downstream tasks. But in this work, it seems that a single feature is learned from a segment of time series. I wonder whether the representation learned from this framework can be used for other more popular downstream tasks such as classification or prediction. \n\nBesides, since in the objective function, there are clustering loss and anomaly detection loss, I think we cannot say it is an unsupervised representation learning approach. It is more like a clustering model or an anomaly detection model. To demonstrate that the proposed framework can produce a good time-series representation, other downstream tasks such as classification or prediction need to be considered. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an AutoML method for irregular multivariate time series.   The method automates the selection of the configuration as well as the hyperparameter optimization depending on the task. A Bayesian approach handles the network structure search while VAEs + attention is used to learn  representations from irregularly sampled data. There is an additional contribution: anomaly detection via a sample energy function from a GMM on time windows.

While there is some novelty in the proposed approach, mostly in the way in which existing techniques are combined, the paper also has some limitations:
- running the framework over the set of possible models is computationally intensive; in their response, the authors indicate the search space can be constrained, however, doing so would also decrease the performance; in AutoML, added complexity cannot be avoided, but there is no notion of how much longer it takes to find suitable models compared to taking off-the-shelf methods.
- although the paper is geared towards irregularly sampled time series, there are no experiments where the data is naturally irregularly samples; artificially introduced patterns are no substitute for this; (PhysioNet, as suggested by one of the reviewers or MIMIC III both have this type of data and are frequently used in benchmarks)
- AutoML is presented as a general framework, but mostly handles clustering and anomaly detection;  unclear of how useful it would be for forecasting or regression; classification realists are shown in Appendix F against simple baselines (GRU-D is not considered, for instance) and even so AutoML does not achieve state of the art results in half of the cases
Decision: Reject