TimeAutoML: Autonomous Representation Learning for Multivariate Irregularly  Sampled Time Series

1 INTRODUCTION . The past decade has witnessed a rising proliferation in Multivariate Time Series ( MTS ) data , along with a plethora of applications in domains as diverse as IoT data analysis , medical informatics , and network security . Given the huge amount of MTS data , it is crucial to learn their representations effectively so as to facilitate underlying applications such as clustering and anomaly detection . For this purpose , different types of methods have been developed to represent time series data . Traditional time series representation techniques , e.g. , Discrete Fourier Transform ( DCT ) ( Faloutsos et al. , 1994 ) , Discrete Wavelet Transform ( DWT ) ( Chan & Fu , 1999 ) , Piecewise Aggregate Approximation ( PAA ) ( Keogh et al. , 2001 ) , etc. , represent raw time series data based on specific domain knowledge/data properties and hence could be suboptimal for subsequent tasks given the fact that their objectives and feature extraction are decoupled . More recent time series representation approaches , e.g. , Deep Temporal Clustering Representation ( DTCR ) ( Ma et al. , 2019 ) , Self-Organizing Map based Variational Auto Encoder ( SOM-VAE ) ( Fortuin et al. , 2018 ) , etc. , optimize the representation and the underlying task such as clustering in an end-to-end manner . These methods usually assume that time series under investigation are uniformly sampled with a fixed interval . This assumption , however , does not always hold in many applications . For example , within a multimodal IoT system , the sampling rates could vary for different types of sensors . Unsupervised representation learning for irregularly sampled multivariate time series is a challenging task and there are several major hurdles preventing us from building effective models : i ) the design of neural network architecture often employs a trial and error procedure which is time consuming and could cost a substantial amount of labor effort ; ii ) the irregularity in the sampling rates constitutes a major challenge against effective learning of time series representations and render most existing methods not directly applicable ; iii ) traditional unsupervised time series representation learning approach does not consider contrastive loss functions and consequently only can achieve suboptimal performance . To tackle the aforementioned challenges , we propose an autonomous unsupervised representation learning approach for multivariate time series to represent irregularly sampled multivariate time series . TimeAutoML differs from traditional time series representation approaches in three aspects . First , the representation learning pipeline configuration and hyperparameter optimization are carried out automatically . Second , a negative sample generation approach is proposed to generate negative samples for contrastive learning . Finally , an auxiliary classification task is developed to distinguish normal time series from negative samples . In this way , the representation capability of TimeAutoML is greatly enhanced . We conduct extensive experiments on UCR time series datasets and UEA multivariate time series datasets . Our experiments demonstrate that the proposed TimeAutoML outperforms comparison algorithms on both clustering and anomaly detection tasks by a large margin , especially when time series data is irregularly sampled . 2 RELATED WORK . Unsupervised Time Series Representation Learning Time series representation learning plays an essential role in a multitude of downstream analysis such as classification , clustering , anomaly detection . There is a growing interest in unsupervised time series representation learning , partially because no labels are required in the learning process , which suits very well many practical applications . Unsupervised time series representation learning can be broadly divided into two categories , namely 1 ) multi-stage methods and 2 ) end-to-end methods . Multi-stage methods first learn a distance metric from a set of time series , or extract the features from the time series , and then perform downstream machine learning tasks based on the learned or the extracted features . Euclidean distance ( ED ) and Dynamic Time Warping ( DTW ) are the most commonly used traditional time series distance metrics . Although the ED is competitive , it is very sensitive to outliers in the time series . The main drawback of DTW is its heavy computational burden . Traditional time series feature extraction methods include Singular Value Decomposition ( SVD ) , Symbolic Aggregate Approximation ( SAX ) , Discrete Wavelet Transform ( DWT ) ( Chan & Fu , 1999 ) , Piecewise Aggregate Approximation ( PAA ) ( Keogh et al. , 2001 ) , etc . Nevertheless , most of these traditional methods are for regularly sampled time series , so they may not perform well on irregularly sampled time series . In recent years , many new feature extraction methods and distance metrics are proposed to overcome the drawbacks mentioned above . For instance , Paparrizos & Gravano ( 2015 ) ; Petitjean et al . ( 2011 ) combine the proposed distance metrics and K-Means algorithm to achieve clustering . Lei et al . ( 2017 ) first extracts sparse features of time series , which is not sensitive to outliers and irregular sampling rate , and then carries out the K-Means clustering . In contrast , end-to-end approaches learn the representation of the time series in an end-to-end manner without explicit feature extraction or distance learning ( Fortuin et al. , 2018 ; Ma et al. , 2019 ) . However , the aforementioned methods need to manually design the network architecture based on human experience which is time-consuming and costly . Instead , we propose in this paper a representation learning method which optimizes an AutoML pipeline and their hyperparameters in a fully autonomous manner . Furthermore , we consider negative sampling and contrastive learning in the proposed framework to effectively enhance the representation ability of the proposed neural network architecture . Irregularly Sampled Time Series Learning There exist two main groups of works regarding machine learning for irregularly sampled time series data . The first type of methods impute the missing values before conducting the subsequent machine learning tasks ( Shukla & Marlin , 2019 ; Luo et al. , 2018 ; 2019 ; Kim & Chi , 2018 ) . The second type directly learns from the irregularly sampled time series . For instance , Che et al . ( 2018 ) ; Cao et al . ( 2018 ) propose a memory decay mechanism , which replaces the memory cell of RNN by the memory of the previous timestamp multiplied by a learnable decay coefficient when there are no sampling value at this timestamp . Rubanova et al . ( 2019 ) combines RNN with ordinary differential equation to model the dynamic of irregularly sampled time series . Different from the previous works , TimeAutoML makes use of the special characteristics of RNN ( Abid & Zou , 2018 ) and automatically configure a representation learning pipeline to model the temporal dynamics of time series . It is worthy mentioning that there are many types of irregularly sampled time series , which may be caused by sensor failure or sampling time error . And what we put emphasis on analyzing in this paper is a special type of irregularly sampled time series , which have many missing timestamps compared to regularly sampled time series . AutoML Automatic Machine Learning ( AutoML ) aims to automate the time-consuming model development process and has received significant amount of research interests recently . Previous works about AutoML mostly emphasize on the domains of computer vision and natural language processing , including object detection ( Ghiasi et al. , 2019 ; Xu et al. , 2019 ; Chen et al . ) , semantic segmentation ( Weng et al. , 2019 ; Nekrasov et al. , 2019 ; Bae et al. , 2019 ) , translation ( Fan et al. , 2020 ) and sequence labeling ( Chen et al. , 2018a ) . However , AutoML for time series learning is an underappreciated topic so far and the existing works mainly focus on supervised learning tasks , e.g. , time series classification . Ukil & Bandyopadhyay propose an AutoML pipeline for automatic feature extraction and feature selection for time series classification . Van Kuppevelt et al . ( 2020 ) develops an AutoML framework for supervised time series classification , which involves both neural architecture search and hyperparameter optimization . Olsavszky et al . ( 2020 ) proposes a framework called AutoTS , which performs time series forecasting of multiple diseases . Nevertheless , to our best knowledge , no previous work has addressed unsupervised time series learning based on AutoML . Summary of comparisons with related work We next provide a comprehensive comparison between the proposed framework and other state-of-the-art methods , including ( WaRTEm ( Mathew et al. , 2019 ) , DTCR ( Ma et al. , 2019 ) , USRLT ( Franceschi et al. , 2019 ) and BeatGAN ( Zhou et al. , 2019 ) ) , as shown in Table 1 . In particular , we emphasize on a total of seven features in the comparison , including data augmentation , negative sample generation , contrastive learning , selection of autoencoders , similarity metric selection , attention mechanism selection , and automatic hyperparameter search . TimeAutoML is the only method that has all the desired properties . 3 TIMEAUTOML FRAMEWORK . 3.1 PROPOSED AUTOML FRAMEWORK . LetX = { x1 , x2 , · · ·xN } denote a set ofN time series in whichxi ∈ RTi , where Ti is the length of the ith time seriesxi . We aim to build an automated time series representation learning framework to generate task-aware representations that can support a variety of downstream machine learning tasks . In addition , we consider negative sample generation and contrastive self-supervised learning . The contrastive loss function focuses on building time series representations by learning to encode what makes two time series similar or different . The proposed TimeAutoML framework can automatically configure an representation learning pipeline with an array of functional modules , each of these modules is associated with a set of hyperparameters . We assume there are a total of M modules and there areQi options for the ith functional module . Let ki ∈ { 0 , 1 } Qi denote an indicating vector for ith module , with the constraint 1 > ki= ∑Qi j=1 ki , j = 1 ensuring that only a single option is chosen for each module . Let θi , j be the hyperparameters of j th option in ith module , where θCi , j and θ D i , j are respectively the continuous and discrete hyperparameters . Let Θ and K denote the set of variables to optimize , i.e. , Θ = { θi , j , ∀i ∈ [ M ] , j ∈ [ Qi ] } and K = { k1 , . . . , kM } . We further let f ( K , Θ ) denote the corresponding objective function value . Please note that the objective function differs for different tasks . For anomaly detection , we use Area Under the Receiver Operating Curve ( AUC ) as objective function while we use the Normalized Mutual Information ( NMI ) as objective function for clustering . The optimization problem of automatic pipeline configuration is shown below . max K , Θ f ( K , Θ ) subject to { ki ∈ { 0 , 1 } Qi , 1 > ki = 1 , ∀i ∈ [ M ] , θCi , j ∈ Ci , j , θ D i , j ∈ Di , j , ∀i ∈ [ M ] , j ∈ [ Qi ] . ( 1 ) We solve problem ( 1 ) by alternatively leveraging Thompson sampling and Bayesian optimization , which will be discussed as follows .

Summary: This work proposes an AutoML framework for multivariate irregularly sampled time series. To achieve this, the proposed framework integrates different modules: data-augmentation self-supervised loss (Equation 7), an anomaly detection loss (Equation 5), and a reconstruction loss. Besides, hyperparameters and model’s configuration is optimized by using AutoML (including Bayesian optimization). The model is evaluated on the well-known time-series datasets UCR and UAE.