TimeAutoML: Autonomous Representation Learning for Multivariate Irregularly  Sampled Time Series

1 INTRODUCTION . The past decade has witnessed a rising proliferation in Multivariate Time Series ( MTS ) data , along with a plethora of applications in domains as diverse as IoT data analysis , medical informatics , and network security . Given the huge amount of MTS data , it is crucial to learn their representations effectively so as to facilitate underlying applications such as clustering and anomaly detection . For this purpose , different types of methods have been developed to represent time series data . Traditional time series representation techniques , e.g. , Discrete Fourier Transform ( DCT ) ( Faloutsos et al. , 1994 ) , Discrete Wavelet Transform ( DWT ) ( Chan & Fu , 1999 ) , Piecewise Aggregate Approximation ( PAA ) ( Keogh et al. , 2001 ) , etc. , represent raw time series data based on specific domain knowledge/data properties and hence could be suboptimal for subsequent tasks given the fact that their objectives and feature extraction are decoupled . More recent time series representation approaches , e.g. , Deep Temporal Clustering Representation ( DTCR ) ( Ma et al. , 2019 ) , Self-Organizing Map based Variational Auto Encoder ( SOM-VAE ) ( Fortuin et al. , 2018 ) , etc. , optimize the representation and the underlying task such as clustering in an end-to-end manner . These methods usually assume that time series under investigation are uniformly sampled with a fixed interval . This assumption , however , does not always hold in many applications . For example , within a multimodal IoT system , the sampling rates could vary for different types of sensors . Unsupervised representation learning for irregularly sampled multivariate time series is a challenging task and there are several major hurdles preventing us from building effective models : i ) the design of neural network architecture often employs a trial and error procedure which is time consuming and could cost a substantial amount of labor effort ; ii ) the irregularity in the sampling rates constitutes a major challenge against effective learning of time series representations and render most existing methods not directly applicable ; iii ) traditional unsupervised time series representation learning approach does not consider contrastive loss functions and consequently only can achieve suboptimal performance . To tackle the aforementioned challenges , we propose an autonomous unsupervised representation learning approach for multivariate time series to represent irregularly sampled multivariate time series . TimeAutoML differs from traditional time series representation approaches in three aspects . First , the representation learning pipeline configuration and hyperparameter optimization are carried out automatically . Second , a negative sample generation approach is proposed to generate negative samples for contrastive learning . Finally , an auxiliary classification task is developed to distinguish normal time series from negative samples . In this way , the representation capability of TimeAutoML is greatly enhanced . We conduct extensive experiments on UCR time series datasets and UEA multivariate time series datasets . Our experiments demonstrate that the proposed TimeAutoML outperforms comparison algorithms on both clustering and anomaly detection tasks by a large margin , especially when time series data is irregularly sampled . 2 RELATED WORK . Unsupervised Time Series Representation Learning Time series representation learning plays an essential role in a multitude of downstream analysis such as classification , clustering , anomaly detection . There is a growing interest in unsupervised time series representation learning , partially because no labels are required in the learning process , which suits very well many practical applications . Unsupervised time series representation learning can be broadly divided into two categories , namely 1 ) multi-stage methods and 2 ) end-to-end methods . Multi-stage methods first learn a distance metric from a set of time series , or extract the features from the time series , and then perform downstream machine learning tasks based on the learned or the extracted features . Euclidean distance ( ED ) and Dynamic Time Warping ( DTW ) are the most commonly used traditional time series distance metrics . Although the ED is competitive , it is very sensitive to outliers in the time series . The main drawback of DTW is its heavy computational burden . Traditional time series feature extraction methods include Singular Value Decomposition ( SVD ) , Symbolic Aggregate Approximation ( SAX ) , Discrete Wavelet Transform ( DWT ) ( Chan & Fu , 1999 ) , Piecewise Aggregate Approximation ( PAA ) ( Keogh et al. , 2001 ) , etc . Nevertheless , most of these traditional methods are for regularly sampled time series , so they may not perform well on irregularly sampled time series . In recent years , many new feature extraction methods and distance metrics are proposed to overcome the drawbacks mentioned above . For instance , Paparrizos & Gravano ( 2015 ) ; Petitjean et al . ( 2011 ) combine the proposed distance metrics and K-Means algorithm to achieve clustering . Lei et al . ( 2017 ) first extracts sparse features of time series , which is not sensitive to outliers and irregular sampling rate , and then carries out the K-Means clustering . In contrast , end-to-end approaches learn the representation of the time series in an end-to-end manner without explicit feature extraction or distance learning ( Fortuin et al. , 2018 ; Ma et al. , 2019 ) . However , the aforementioned methods need to manually design the network architecture based on human experience which is time-consuming and costly . Instead , we propose in this paper a representation learning method which optimizes an AutoML pipeline and their hyperparameters in a fully autonomous manner . Furthermore , we consider negative sampling and contrastive learning in the proposed framework to effectively enhance the representation ability of the proposed neural network architecture . Irregularly Sampled Time Series Learning There exist two main groups of works regarding machine learning for irregularly sampled time series data . The first type of methods impute the missing values before conducting the subsequent machine learning tasks ( Shukla & Marlin , 2019 ; Luo et al. , 2018 ; 2019 ; Kim & Chi , 2018 ) . The second type directly learns from the irregularly sampled time series . For instance , Che et al . ( 2018 ) ; Cao et al . ( 2018 ) propose a memory decay mechanism , which replaces the memory cell of RNN by the memory of the previous timestamp multiplied by a learnable decay coefficient when there are no sampling value at this timestamp . Rubanova et al . ( 2019 ) combines RNN with ordinary differential equation to model the dynamic of irregularly sampled time series . Different from the previous works , TimeAutoML makes use of the special characteristics of RNN ( Abid & Zou , 2018 ) and automatically configure a representation learning pipeline to model the temporal dynamics of time series . It is worthy mentioning that there are many types of irregularly sampled time series , which may be caused by sensor failure or sampling time error . And what we put emphasis on analyzing in this paper is a special type of irregularly sampled time series , which have many missing timestamps compared to regularly sampled time series . AutoML Automatic Machine Learning ( AutoML ) aims to automate the time-consuming model development process and has received significant amount of research interests recently . Previous works about AutoML mostly emphasize on the domains of computer vision and natural language processing , including object detection ( Ghiasi et al. , 2019 ; Xu et al. , 2019 ; Chen et al . ) , semantic segmentation ( Weng et al. , 2019 ; Nekrasov et al. , 2019 ; Bae et al. , 2019 ) , translation ( Fan et al. , 2020 ) and sequence labeling ( Chen et al. , 2018a ) . However , AutoML for time series learning is an underappreciated topic so far and the existing works mainly focus on supervised learning tasks , e.g. , time series classification . Ukil & Bandyopadhyay propose an AutoML pipeline for automatic feature extraction and feature selection for time series classification . Van Kuppevelt et al . ( 2020 ) develops an AutoML framework for supervised time series classification , which involves both neural architecture search and hyperparameter optimization . Olsavszky et al . ( 2020 ) proposes a framework called AutoTS , which performs time series forecasting of multiple diseases . Nevertheless , to our best knowledge , no previous work has addressed unsupervised time series learning based on AutoML . Summary of comparisons with related work We next provide a comprehensive comparison between the proposed framework and other state-of-the-art methods , including ( WaRTEm ( Mathew et al. , 2019 ) , DTCR ( Ma et al. , 2019 ) , USRLT ( Franceschi et al. , 2019 ) and BeatGAN ( Zhou et al. , 2019 ) ) , as shown in Table 1 . In particular , we emphasize on a total of seven features in the comparison , including data augmentation , negative sample generation , contrastive learning , selection of autoencoders , similarity metric selection , attention mechanism selection , and automatic hyperparameter search . TimeAutoML is the only method that has all the desired properties . 3 TIMEAUTOML FRAMEWORK . 3.1 PROPOSED AUTOML FRAMEWORK . LetX = { x1 , x2 , · · ·xN } denote a set ofN time series in whichxi ∈ RTi , where Ti is the length of the ith time seriesxi . We aim to build an automated time series representation learning framework to generate task-aware representations that can support a variety of downstream machine learning tasks . In addition , we consider negative sample generation and contrastive self-supervised learning . The contrastive loss function focuses on building time series representations by learning to encode what makes two time series similar or different . The proposed TimeAutoML framework can automatically configure an representation learning pipeline with an array of functional modules , each of these modules is associated with a set of hyperparameters . We assume there are a total of M modules and there areQi options for the ith functional module . Let ki ∈ { 0 , 1 } Qi denote an indicating vector for ith module , with the constraint 1 > ki= ∑Qi j=1 ki , j = 1 ensuring that only a single option is chosen for each module . Let θi , j be the hyperparameters of j th option in ith module , where θCi , j and θ D i , j are respectively the continuous and discrete hyperparameters . Let Θ and K denote the set of variables to optimize , i.e. , Θ = { θi , j , ∀i ∈ [ M ] , j ∈ [ Qi ] } and K = { k1 , . . . , kM } . We further let f ( K , Θ ) denote the corresponding objective function value . Please note that the objective function differs for different tasks . For anomaly detection , we use Area Under the Receiver Operating Curve ( AUC ) as objective function while we use the Normalized Mutual Information ( NMI ) as objective function for clustering . The optimization problem of automatic pipeline configuration is shown below . max K , Θ f ( K , Θ ) subject to { ki ∈ { 0 , 1 } Qi , 1 > ki = 1 , ∀i ∈ [ M ] , θCi , j ∈ Ci , j , θ D i , j ∈ Di , j , ∀i ∈ [ M ] , j ∈ [ Qi ] . ( 1 ) We solve problem ( 1 ) by alternatively leveraging Thompson sampling and Bayesian optimization , which will be discussed as follows .

Summary: This paper proposes an autonomous representation learning framework for multivariate time series with irregular sampling rates. Specifically, there are three major components proposed in the framework. 1) An AutoML solution for hyperparameters optimization under Bayesian framework is proposed to automatically seek optimal network structures and parameters. 2) Variational autoencoders based on generative approach and attention mechanism is employed to learn the semantic representation of time series with limitation of irregular sampling. 3) A sample energy function derived from Gaussian mixture model attempts to depict the level of anomaly of sliced time series.