Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding

1 INTRODUCTION . Real-world time-series data is high dimensional , complex , and has unique properties that bring about many challenges for data modeling ( Yang & Wu , 2006 ) . In addition , these signals are often sparsely labeled , making it even more challenging for supervised learning tasks . Unsupervised representation learning can extract informative low-dimensional representations from raw time series by leveraging the data ’ s inherent structure , without the need for explicit supervision . These representations are more generalizable and robust , as they are less specialized for solving a single supervised task . Unsupervised representation learning is well studied in domains such as vision ( Donahue & Simonyan , 2019 ; Denton et al. , 2017 ; Radford et al. , 2015 ) and natural language processing ( Radford et al. , 2017 ; Young et al. , 2018 ; Mikolov et al. , 2013 ) , but has been underexplored in the literature for time series settings . Frameworks designed for time series need to be efficient and scalable because signals encountered in practice can be long , high dimensional , and high frequency . Moreover , it should account for and be able to model dynamic changes that occur within samples , i.e. , non-stationarity of signals . The ability to model the dynamic nature of time series data is especially valuable in medicine . Health care data is often organized as a time series , with multiple data types , collected from various sources at different sampling frequencies , and riddled with artifacts and missing values . Throughout their stay at the hospital or within the disease progression period , patients transition gradually between distinct ∗http : //www.cs.toronto.edu/~stonekaboni/ clinical states , with periods of relative stability , improvement , or unexpected deterioration , requiring escalation of care that alters the patient ’ s trajectory . A particular challenge in medical time-series data is the lack of well-defined or available labels that are needed for identifying the underlying clinical state of an individual or for training models aimed at extracting low-dimensional representations of these states . For instance , in the context of critical-care , a patient ’ s stay in the critical care unit ( CCU ) is captured continuously via streaming physiological signals by the bedside monitor . Obtaining labels for the patient ’ s state for extended periods of these signals is practically impossible as the underlying physiological state can be unknown even to the clinicians . This further motivates the use of unsupervised representation learning in these contexts . Learning rich representations can be crucial in facilitating the tracking of disease progression , predicting the future trajectories of the patients , and tailoring treatments to these underlying states . In this paper , we propose a self-supervised framework for learning representations for complex multivariate non-stationary time series . This approach , called Temporal Neighborhood Coding ( TNC ) , is designed for temporal settings where the latent distribution of the signals changes over time , and it aims to capture the progression of the underlying temporal dynamics . TNC is efficient , easily scalable to high dimensions , and can be used in different time series settings . We assess the quality of the learned representations on multiple datasets and show that the representations are general and transferable to many downstream tasks such as classification and clustering . We further demonstrate that our method outperforms existing approaches for unsupervised representation learning , and it even performs closely to supervised techniques in classification tasks . The contributions of this work are three-fold : 1 . We present a novel neighborhood-based unsupervised learning framework for non-stationary multivariate time series data . 2 . We introduce the concept of a temporal neighborhood with stationary properties as the distribution of similar windows in time . The neighborhood boundaries are determined automatically using the properties of the signal and statistical testing . 3 . We incorporate concepts from Positive Unlabeled Learning , specifically , sample weight adjustment , to account for potential bias introduced in sampling negative examples for the contrastive loss . 2 METHOD . We introduce a framework for learning representations that encode the underlying state of a multivariate , non-stationary time series . Our self-supervised approach , TNC , takes advantage of the local smoothness of the generative process of signals to learn generalizable representations for windows of time series . This is done by ensuring that in the representation space , the distribution of signals proximal in time is distinguishable from the distribution of signals far away , i.e. , proximity in time is identifiable in the encoding space . We represent our multivariate time series signals as X ∈ RD×T , where D is the number of features and T is the number of measurements over time . X [ t− δ2 , t+ δ2 ] represents a window of time series of length δ , centered around time t , that includes measurements of all features taken in the interval [ t− δ2 , t+ δ 2 ] . Throughout the paper , we refer to this window as Wt for notational simplicity . Our goal is to learn the underlying representation of Wt , and by sliding this window over time , we can obtain the trajectory of the underlying states of the signal . We define the temporal neighborhood ( Nt ) of a window Wt as the set of all windows with centroids t∗ , sampled from a normal distribution t∗ ∼ N ( t , η · δ ) . Where N is a Gaussian centered at t , δ is the size of the window , and η is the parameter that defines the range of the neighborhood . Relying on the local smoothness of a signal ’ s generative process , the neighborhood distribution is characterized as a Gaussian to model the gradual transition in temporal data , and intuitively , it approximates the distribution of samples that are similar to Wt . The η parameter determines the neighborhood range and depends on the signal characteristics and how gradual the time series ’ s statistical properties change over time . This can be set by domain experts based on prior knowledge of the signal behavior , or for more robust estimation , it can be determined by analyzing the stationarity properties of the signal for every Wt . Since the neighborhood represents similar samples , the range should identify the approximate time span within which the signal remains stationary , and the generative process does not change . For this purpose , we use the Augmented Dickey-Fuller ( ADF ) statistical test to determine this region for every window . Proper estimation of the neighborhood range is an integral part of the TNC framework . If η is too small , many samples from within a neighborhood will overlap , and therefore the encoder would only learn to encode the overlapping information . On the other hand , if η is too big , the neighborhood would span over multiple underlying states , and therefore the encoder would fail to distinguish the variation among these states . Using the ADF test , we can automatically adjust the neighborhood for every window based on the signal behavior . More details on this test and how it is used to estimate η is described in section 2 . Now , assuming windows within a neighborhood possess similar properties , signals outside of this neighborhood , denoted as N̄t , are considered non-neighboring windows . Samples from N̄t are likely to be different from Wt , and can be considered as negative samples in a context of a contrastive learning framework . However , this assumption can suffer from the problem of sampling bias , common in most contrastive learning approaches ( Chuang et al. , 2020 ; Saunshi et al. , 2019 ) . This bias occurs because randomly drawing negative examples from the data distribution may result in negative samples that are actually similar to the reference . This can significantly impact the learning framework ’ s performance , but little work has been done on addressing this issue ( Chuang et al. , 2020 ) . In our context , this can happen when there are windows from N̄t that are far away from Wt , but have the same underlying state . To alleviate this bias in the TNC framework , we consider samples from N̄t as unlabeled samples , as opposed to negative , and use ideas from Positive-Unlabeled ( PU ) learning to accurately measure the loss function . In reality , even though samples within a neighborhood are all similar , we can not make the assumption that samples outside this region are necessarily different . For instance , in the presence of long term seasonalities , signals can exhibit similar properties at distant times . In a healthcare context , this can be like a stable patient that undergoes a critical condition , but returns back to a stable state afterwards . In PU learning , a classifier is learned using labeled data drawn from the positive class ( P ) and unlabeled data ( U ) that is a mixture of positive and negative samples with a positive class prior π ( Du Plessis et al. , 2014 ; Kiryo et al. , 2017 ; Du Plessis & Sugiyama , 2014 ) . Existing PU learning methods fall under two categories based on how they handle the unlabeled data : 1 ) methods that identify negative samples from the unlabeled cohort ( Li & Liu , 2003 ) ; 2 ) methods that treat the unlabeled data as negative samples with smaller weights ( Lee & Liu , 2003 ; Elkan & Noto , 2008 ) . In the second category , unlabeled samples should be properly weighted in the loss term in order to train an unbiased classifier . Elkan & Noto ( 2008 ) introduces a simple and efficient way of approximating the expectation of a loss function by assigning individual weights w to training examples from the unlabeled cohort . This means each sample from the neighborhood is treated as a positive example with unit weight , while each sample from N̄ is treated as a combination of a positive example with weight w and a negative example with complementary weight 1− w. In the original paper ( Elkan & Noto , 2008 ) , the weight is defined as the probability for a sample from the unlabeled set to be a positive sample , i.e . w = p ( y = 1|x ) for x ∈ U . In the TNC framework , this weight represents the probability of having samples similar to Wt in N̄ . By incorporating weight adjustment into the TNC loss ( Equation 1 ) , we account for possible positive samples that occur in the non-neighboring distribution . w can be approximated using prior knowledge of the underlying state distribution or tuned as a hyperparameter . Appendix A.6 explains how the weight parameter is selected for our different experiment setups and also demonstrates the impact of weight adjustment on performance for downstream tasks . After defining the neighborhood distribution , we train an objective function that encourages a distinction between the representation of samples of the same neighborhood from the outside samples . An ideal encoder preserves the neighborhood properties in the encoding space . Therefore representations Zl = Enc ( Wl ) of samples from a neighborhood Wl ∈ Nt , can be distinguished from representation Zk = Enc ( Wk ) of samples from outside the neighborhood Wk ∈ N̄t . TNC is composed of two main components : 1 . An Encoder Enc ( Wt ) that maps Wt ∈ RD×δ to a representation Zt ∈ RM , in a lower dimensional space ( M D × δ ) , where D × δ is the total number of measurements in Wt . 2 . A Discriminator D ( Zt , Z ) that approximates the probability of Z being the representation of a window in Nt . More specifically , it receives two samples from the encoding space and predicts the probability of those samples belonging to the same temporal neighborhood . Published as a conference paper at ICLR 2021 Enc Enc Representation D at a Enc Enc Representation D at a Weighting ( a ) Neighborhood samples Enc Enc Representation D at a Enc Enc Representation D at a Weighting ( b ) Non-neighboring samples Figure 1 : Overview of the TNC framework components . For each sample window Wt ( indicated with the dashed black box ) , we first define the neighborhood distribution . The encoder learns the distribution of windows sampled from Nt and N̄t , in the representation space . Then samples from these distributions are fed into the discriminator alongside Zt , to predict the probability of the windows being in the same neighborhood . TNC is a general framework ; therefore , it is agnostic to the nature of the time series and the architecture of the encoder . The encoder can be any parametric model that is well-suited to the signal properties ( Oord et al. , 2016 ; Bai et al. , 2018 ; Fawaz et al. , 2019 ) . For the DiscriminatorD ( Zt , Z ) we use a simple multi-headed binary classifier that outputs 1 if Z and Zt are representations of neighbors in time , and 0 otherwise . In the experiment section , we describe the architectural details of the models used for our experiments in more depth . Figure 1 provides a summary overview of the TNC framework . We formalize the objective function of our unsupervised learning framework in Equation 1 . In essence , we would like the probability likelihood estimation of the Discriminator to be accurate , i.e. , close to 1 for the representation of neighboring samples and close to 0 for windows far apart . Samples from the non-neighboring region ( N̄ ) are weight-adjusted using the w parameters to account for positive samples in this distribution . L = −EWt∼X [ EWl∼Nt [ logD ( Zt , Zl ) ︸ ︷︷ ︸ D ( Enc ( Wt ) , Enc ( Wl ) ) ] +EWk∼N̄t [ ( 1−wt ) ×log ( 1−D ( Zt , Zk ) ) ︸ ︷︷ ︸ D ( Enc ( Wt ) , Enc ( Wk ) ) +wt×logD ( Zt , Zk ) ] ] ( 1 ) We train the encoder and the discriminator hand in hand by optimizing for this objective . Note that the Discriminator is only part of training and will not be used during inference . Similar to the encoder , it can be approximated using any parametric model . However , the more complex the Discriminator , the harder it becomes to interpret the latent space ’ s decision boundaries since it allows similarities to be mapped on complex nonlinear relationships . Defining the neighborhood parameter using the ADF test : As mentioned earlier , the neighborhood range can be specified using the characteristics of the data . In non-stationary time series , the generative process of the signals changes over time . We define the temporal neighborhood around every window as the region where the signal is relatively stationary . Since a signal may remain in an underlying state for an unknown amount of time , each window ’ s neighborhood range may vary in size and must be adjusted to signal behavior . To that end , we use the Augmented Dickey-Fuller ( ADF ) statistical test to derive the neighborhood range η . The ADF test belongs to a category of tests called `` Unit Root Test '' , and is a method for testing the stationarity of a time series . For every Wt , we want to find the neighborhood range around that window that indicates a stationary region . To determine this , we start from η = 1 and gradually increase the neighborhood size η , measuring the p-value from the test at every step . Once p-value is above a threshold ( in our setting 0.01 ) , it means that it fails to reject the null hypothesis and suggests that within this neighborhood region , the signal is no longer stationary . This way , we find the widest neighborhood within which the signal remains relatively stationary . Note that the window size δ is constant throughout the experiment , and during ADF testing , we only adjust the neighborhood ’ s width .

Summary: The paper proposes an unsupervised representation (embedding) learning method for time-series. While unsupervised representation learning has been extensively studied and shown good performance in fields like NLP and vision, it is relatively new to the time-series community. This paper, in contrast to recent work (CPC and Triplet-Loss), has the following differences: