Self-Supervised Time Series Representation Learning by Inter-Intra Relational Reasoning

1 INTRODUCTION . Time series data is ubiquitous and there has been significant progress for time series analysis ( Das , 1994 ) in machine learning , signal processing , and other related areas , with many real-world applications such as healthcare ( Stevner et al. , 2019 ) , industrial diagnosis ( Kang et al. , 2015 ) , and financial forecasting ( Sen et al. , 2019 ) . Deep learning models have emerged as successful models for time series analysis ( Hochreiter & Schmidhuber , 1997 ; Graves et al. , 2013 ; Shukla & Marlin , 2019 ; Fortuin et al. , 2019 ; Oreshkin et al. , 2020 ) . Despite their fair share of success , the existing deep supervised models are not suitable for high-dimensional time series data with a limited amount of training samples as those data-driven approaches rely on finding ground truth for supervision , where data labeling is a labor-intensive and time-consuming process , and sometimes impossible for time series data . One solution is to learn useful representations from unlabeled data , which can substantially reduce dependence on costly manual annotation . Self-supervised learning aims to capture the most informative properties from the underlying structure of unlabeled data through the self-generated supervisory signal to learn generalized representations . Recently , self-supervised learning has attracted more and more attention in computer vision by designing different pretext tasks on image data such as solving jigsaw puzzles ( Noroozi & Favaro , 2016 ) , inpainting ( Pathak et al. , 2016 ) , rotation prediction ( Gidaris et al. , 2018 ) , and contrastive learning of visual representations ( Chen et al. , 2020 ) , and on video data such as object tracking ( Wang & Gupta , 2015 ) , and pace prediction ( Wang et al. , 2020 ) . Although some video-based ap- 1Anonymous repository link . proaches attempt to capture temporal information in the designed pretext task , time series is far different structural data compared with video . More recently , in the time series analysis domain , some metric learning based self-supervised methods such as triplet loss ( Franceschi et al. , 2019 ) and contrastive loss ( Schneider et al. , 2019 ; Saeed et al. , 2020 ) , or multi-task learning based self-supervised methods that predict different handcrafted features ( Pascual et al. , 2019a ; Ravanelli et al. , 2020 ) and different signal transformations ( Saeed et al. , 2019 ; Sarkar & Etemad , 2020 ) have emerged . However , few of those works consider the intra-temporal structure of time series . Therefore , how to design an efficient pretext task in a self-supervised manner for time series representation learning is still an open problem . In this work , we present SelfTime : a general self-supervised time series representation learning framework . Inspired by relational discovery during self-supervised human learning , which attempts to discover new knowledge by reasoning the relation among entities ( Goldwater et al. , 2018 ; Patacchiola & Storkey , 2020 ) , we explore the inter-sample relation reasoning and intra-temporal relation reasoning of time series to capture the underlying structure pattern of the unlabeled time series data . Specifically , as shown in Figure 1 , for inter-sample relation reasoning , given an anchor sample , we generate from its transformation counterpart and another individual sample as the positive and negative samples respectively . For intra-temporal relation reasoning , we firstly generate an anchor piece , then , several reference pieces are sampled to construct different scales of temporal relation between the anchor piece and the reference piece , where relation scales are determined based on the temporal distance . Note that in Figure 1 , we only show an example of 3-scale temporal relations including short-term , middle-term , and long-term relation for an illustration , whereas in different scenarios , there could be different temporal relation scale candidates . Based on the sampled relation , a shared feature extraction backbone combined with two separate relation reasoning heads are employed to quantify the relationships between the sample pairs or the time piece pairs for inter-sample relation reasoning or intra-temporal relation reasoning , respectively . Finally , the useful representations of time series are extracted from the backbone under the supervision of relation reasoning heads on the unlabeled data . Overall , SelfTime is simple yet effective by conducting the designed pretext tasks directly on the original input signals . Our main contributions are three-fold : ( 1 ) we present a general self-supervised time series representation learning framework by investigating different levels of relations of time series data including inter-sample relation and intra-temporal relation . ( 2 ) We design a simple and effective intra-temporal relation sampling strategy to capture the underlying temporal patterns of time series . ( 3 ) We conduct extensive experiments on different categories of real-world time series data , and systematically study the impact of different data augmentation strategies and temporal relation sampling strategies on self-supervised learning of time series . By comparing with multiple state-of-the-art baselines , experimental results show that SelfTime builds new state-of-the-art on self-supervised time series representation learning . 2 RELATED WORK . Time Series Modeling . In the last decades , time series modeling has been paid close attention with numerous efficient methods , including distance-based methods , feature-based methods , ensemblebased methods , and deep learning based methods . Distance-based methods ( Berndt & Clifford , 1994 ; Górecki & Łuczak , 2014 ) try to measure the similarity between time series using Euclidean distance or Dynamic Time Warping distance , and then conduct classification based on 1-NN classifiers . Feature-based methods aim to extract useful feature for time series representation . Two typical types including bag-of-feature based methods ( Baydogan et al. , 2013 ; Schäfer , 2015 ) and shapelet based methods ( Ye & Keogh , 2009 ; Hills et al. , 2014 ) . Ensemble-based methods ( Lines & Bagnall , 2015 ; Bagnall et al. , 2015 ) aims at combining multiple classifiers for higher classification performance . More recently , deep learning based methods ( Karim et al. , 2017 ; Ma et al. , 2019 ; Cheng et al. , 2020 ) conduct classification by cascading the feature extractor and classifier based on MLP , RNN , and CNN in an end-to-end manner . Our approach focuses instead on self-supervised representation learning of time series on unlabeled data , exploiting inter-sample relation and intra-temporal relation of time series to guide the generation of useful feature . Relational Reasoning . Reasoning the relations between entities and their properties makes significant sense to generally intelligent behavior ( Kemp & Tenenbaum , 2008 ) . In the past decades , there has been an extensive researches about relational reasoning and its applications including knowledge base ( Socher et al. , 2013 ) , question answering ( Johnson et al. , 2017 ; Santoro et al. , 2017 ) , video action recognition ( Zhou et al. , 2018 ) , reinforcement learning ( Zambaldi et al. , 2019 ) , and graph representation ( Battaglia et al. , 2018 ) , which perform relational reasoning directly on the constructed sets or graphs that explicitly represent the target entities and their relations . Different from those previous works that attempt to learn a relation reasoning head for a special task , inter-sample relation reasoning based on unlabeled image data is employed in ( Patacchiola & Storkey , 2020 ) to learn useful visual representation in the underlying backbone . Inspired by this , in our work , we focus on time series data by exploring both inter-sample and intra-temporal relation for time series representation in a self-supervised scenario . Self-supervised Learning . Self-supervised learning has attracted lots of attention recently in different domains including computer vision , audio/speech processing , and time series analysis . For image data , the pretext tasks including solving jigsaw puzzles ( Noroozi & Favaro , 2016 ) , rotation prediction ( Gidaris et al. , 2018 ) , and visual contrastive learning ( Chen et al. , 2020 ) are designed for self-supervised visual representation . For video data , the pretext tasks such as frame order validation ( Misra et al. , 2016 ; Wei et al. , 2018 ) , and video pace prediction ( Wang et al. , 2020 ) are designed which considering additional temporal signal of video . Different from video signal that includes plenty of raw feature in both spatial and temporal dimension , time series is far different structural data with less raw features at each time point . For time series data such as audio and ECG , the metric learning based methods such as triplet loss ( Franceschi et al. , 2019 ) and contrastive loss ( Schneider et al. , 2019 ; Saeed et al. , 2020 ) , or multi-task learning based methods that predict different handcrafted features such as MFCCs , prosody , and waveform ( Pascual et al. , 2019a ; Ravanelli et al. , 2020 ) , and different transformations of raw signal ( Sarkar & Etemad , 2020 ; Saeed et al. , 2019 ) have emerged recently . However , few of those works consider the intra-temporal structure of time series . Therefore , how to design an efficient self-supervised pretext task to capture the underlying structure of time series is still an open problem . 3 METHOD . Given an unlabeled time series set T = { tn } Nn=1 , where each time series tn = ( tn,1 , ... tn , T ) T contains T ordered real values . We aim to learn a useful representation zn = fθ ( tn ) from the backbone encoder fθ ( · ) where θ is the learnable weights of the neural networks . The architecture of the proposed SelfTime is shown in Figure 2 , which consists of an inter-sample relational reasoning branch and an intra-temporal relational reasoning branch . Firstly , taking the original time series signals and their sampled time pieces as the inputs , a shared backbone encoder fθ ( · ) extracts time series feature and time piece feature to aggregate the inter-sample relation feature and intra-temporal relation feature respectively , and then feeds them to two separate relation reasoning heads rµ ( · ) and rϕ ( · ) to reason the final relation score of inter-sample relation and intra-temporal relation . 3.1 INTER-SAMPLE RELATION REASONING . Formally , given any two different time series samples tm and tn from T , we randomly generate two sets of K augmentationsA ( tm ) = { t ( i ) m } Ki=1 andA ( tn ) = { t ( i ) n } Ki=1 , where t ( i ) m and t ( i ) n are the i-th augmentations of tm and tn respectively . Then , we construct two types of relation pairs : positive relation pairs and negative relation pairs . A positive relation pair is ( t ( i ) m , t ( j ) m ) sampled from the same augmentation set A ( tm ) , while a negative relation pair is ( t ( i ) m , t ( j ) n ) sampled from different augmentation sets A ( tm ) and A ( tn ) . Based on the sampled relation pairs , we use the backbone encoder fθ to learn the relation representation as follows : Firstly , we extract sample representations z ( i ) m = fθ ( t ( i ) m ) , z ( j ) m = fθ ( t ( j ) m ) , and z ( j ) n = fθ ( t ( j ) n ) . Then , we construct the positive relation representation [ z ( i ) m , z ( j ) m ] , and the negative relation representation [ z ( i ) m , z ( j ) n ] , where [ · , · ] denotes the vector concatenation operation . Next , the inter-sample relation reasoning head rµ ( · ) takes the generated relation representation as input to reason the final relation score h ( i , j ) 2m−1 = rµ ( [ z ( i ) m , z ( j ) m ] ) for positive relation and h ( i , j ) 2m = rµ ( [ z ( i ) m , z ( j ) n ] ) for negative relation , respectively . Finally , the inter-sample relation reasoning task is formulated as a binary classification task and the model is trained with binary cross-entropy loss Linter as follows : Linter = − 2N∑ n=1 K∑ i=1 K∑ j=1 ( y ( i , j ) n · log ( h ( i , j ) n ) + ( 1− y ( i , j ) n ) · log ( 1− h ( i , j ) n ) ) ( 1 ) where y ( i , j ) n = 1 for the positive relation and y ( i , j ) n = 0 for the negative relation .

Summary: This paper presents a general Self-supervised Time Series representation learning framework. It explores the inter-sample relation reasoning and intra-temporal relation reasoning of time series to capture the underlying structure pattern of the unlabeled time series data.  The proposed method achieves new state-of-the-art results and outperforms existing methods by a significant margin on multiple real-world time-series datasets for the classification tasks.