Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding

1 INTRODUCTION . Real-world time-series data is high dimensional , complex , and has unique properties that bring about many challenges for data modeling ( Yang & Wu , 2006 ) . In addition , these signals are often sparsely labeled , making it even more challenging for supervised learning tasks . Unsupervised representation learning can extract informative low-dimensional representations from raw time series by leveraging the data ’ s inherent structure , without the need for explicit supervision . These representations are more generalizable and robust , as they are less specialized for solving a single supervised task . Unsupervised representation learning is well studied in domains such as vision ( Donahue & Simonyan , 2019 ; Denton et al. , 2017 ; Radford et al. , 2015 ) and natural language processing ( Radford et al. , 2017 ; Young et al. , 2018 ; Mikolov et al. , 2013 ) , but has been underexplored in the literature for time series settings . Frameworks designed for time series need to be efficient and scalable because signals encountered in practice can be long , high dimensional , and high frequency . Moreover , it should account for and be able to model dynamic changes that occur within samples , i.e. , non-stationarity of signals . The ability to model the dynamic nature of time series data is especially valuable in medicine . Health care data is often organized as a time series , with multiple data types , collected from various sources at different sampling frequencies , and riddled with artifacts and missing values . Throughout their stay at the hospital or within the disease progression period , patients transition gradually between distinct ∗http : //www.cs.toronto.edu/~stonekaboni/ clinical states , with periods of relative stability , improvement , or unexpected deterioration , requiring escalation of care that alters the patient ’ s trajectory . A particular challenge in medical time-series data is the lack of well-defined or available labels that are needed for identifying the underlying clinical state of an individual or for training models aimed at extracting low-dimensional representations of these states . For instance , in the context of critical-care , a patient ’ s stay in the critical care unit ( CCU ) is captured continuously via streaming physiological signals by the bedside monitor . Obtaining labels for the patient ’ s state for extended periods of these signals is practically impossible as the underlying physiological state can be unknown even to the clinicians . This further motivates the use of unsupervised representation learning in these contexts . Learning rich representations can be crucial in facilitating the tracking of disease progression , predicting the future trajectories of the patients , and tailoring treatments to these underlying states . In this paper , we propose a self-supervised framework for learning representations for complex multivariate non-stationary time series . This approach , called Temporal Neighborhood Coding ( TNC ) , is designed for temporal settings where the latent distribution of the signals changes over time , and it aims to capture the progression of the underlying temporal dynamics . TNC is efficient , easily scalable to high dimensions , and can be used in different time series settings . We assess the quality of the learned representations on multiple datasets and show that the representations are general and transferable to many downstream tasks such as classification and clustering . We further demonstrate that our method outperforms existing approaches for unsupervised representation learning , and it even performs closely to supervised techniques in classification tasks . The contributions of this work are three-fold : 1 . We present a novel neighborhood-based unsupervised learning framework for non-stationary multivariate time series data . 2 . We introduce the concept of a temporal neighborhood with stationary properties as the distribution of similar windows in time . The neighborhood boundaries are determined automatically using the properties of the signal and statistical testing . 3 . We incorporate concepts from Positive Unlabeled Learning , specifically , sample weight adjustment , to account for potential bias introduced in sampling negative examples for the contrastive loss . 2 METHOD . We introduce a framework for learning representations that encode the underlying state of a multivariate , non-stationary time series . Our self-supervised approach , TNC , takes advantage of the local smoothness of the generative process of signals to learn generalizable representations for windows of time series . This is done by ensuring that in the representation space , the distribution of signals proximal in time is distinguishable from the distribution of signals far away , i.e. , proximity in time is identifiable in the encoding space . We represent our multivariate time series signals as X ∈ RD×T , where D is the number of features and T is the number of measurements over time . X [ t− δ2 , t+ δ2 ] represents a window of time series of length δ , centered around time t , that includes measurements of all features taken in the interval [ t− δ2 , t+ δ 2 ] . Throughout the paper , we refer to this window as Wt for notational simplicity . Our goal is to learn the underlying representation of Wt , and by sliding this window over time , we can obtain the trajectory of the underlying states of the signal . We define the temporal neighborhood ( Nt ) of a window Wt as the set of all windows with centroids t∗ , sampled from a normal distribution t∗ ∼ N ( t , η · δ ) . Where N is a Gaussian centered at t , δ is the size of the window , and η is the parameter that defines the range of the neighborhood . Relying on the local smoothness of a signal ’ s generative process , the neighborhood distribution is characterized as a Gaussian to model the gradual transition in temporal data , and intuitively , it approximates the distribution of samples that are similar to Wt . The η parameter determines the neighborhood range and depends on the signal characteristics and how gradual the time series ’ s statistical properties change over time . This can be set by domain experts based on prior knowledge of the signal behavior , or for more robust estimation , it can be determined by analyzing the stationarity properties of the signal for every Wt . Since the neighborhood represents similar samples , the range should identify the approximate time span within which the signal remains stationary , and the generative process does not change . For this purpose , we use the Augmented Dickey-Fuller ( ADF ) statistical test to determine this region for every window . Proper estimation of the neighborhood range is an integral part of the TNC framework . If η is too small , many samples from within a neighborhood will overlap , and therefore the encoder would only learn to encode the overlapping information . On the other hand , if η is too big , the neighborhood would span over multiple underlying states , and therefore the encoder would fail to distinguish the variation among these states . Using the ADF test , we can automatically adjust the neighborhood for every window based on the signal behavior . More details on this test and how it is used to estimate η is described in section 2 . Now , assuming windows within a neighborhood possess similar properties , signals outside of this neighborhood , denoted as N̄t , are considered non-neighboring windows . Samples from N̄t are likely to be different from Wt , and can be considered as negative samples in a context of a contrastive learning framework . However , this assumption can suffer from the problem of sampling bias , common in most contrastive learning approaches ( Chuang et al. , 2020 ; Saunshi et al. , 2019 ) . This bias occurs because randomly drawing negative examples from the data distribution may result in negative samples that are actually similar to the reference . This can significantly impact the learning framework ’ s performance , but little work has been done on addressing this issue ( Chuang et al. , 2020 ) . In our context , this can happen when there are windows from N̄t that are far away from Wt , but have the same underlying state . To alleviate this bias in the TNC framework , we consider samples from N̄t as unlabeled samples , as opposed to negative , and use ideas from Positive-Unlabeled ( PU ) learning to accurately measure the loss function . In reality , even though samples within a neighborhood are all similar , we can not make the assumption that samples outside this region are necessarily different . For instance , in the presence of long term seasonalities , signals can exhibit similar properties at distant times . In a healthcare context , this can be like a stable patient that undergoes a critical condition , but returns back to a stable state afterwards . In PU learning , a classifier is learned using labeled data drawn from the positive class ( P ) and unlabeled data ( U ) that is a mixture of positive and negative samples with a positive class prior π ( Du Plessis et al. , 2014 ; Kiryo et al. , 2017 ; Du Plessis & Sugiyama , 2014 ) . Existing PU learning methods fall under two categories based on how they handle the unlabeled data : 1 ) methods that identify negative samples from the unlabeled cohort ( Li & Liu , 2003 ) ; 2 ) methods that treat the unlabeled data as negative samples with smaller weights ( Lee & Liu , 2003 ; Elkan & Noto , 2008 ) . In the second category , unlabeled samples should be properly weighted in the loss term in order to train an unbiased classifier . Elkan & Noto ( 2008 ) introduces a simple and efficient way of approximating the expectation of a loss function by assigning individual weights w to training examples from the unlabeled cohort . This means each sample from the neighborhood is treated as a positive example with unit weight , while each sample from N̄ is treated as a combination of a positive example with weight w and a negative example with complementary weight 1− w. In the original paper ( Elkan & Noto , 2008 ) , the weight is defined as the probability for a sample from the unlabeled set to be a positive sample , i.e . w = p ( y = 1|x ) for x ∈ U . In the TNC framework , this weight represents the probability of having samples similar to Wt in N̄ . By incorporating weight adjustment into the TNC loss ( Equation 1 ) , we account for possible positive samples that occur in the non-neighboring distribution . w can be approximated using prior knowledge of the underlying state distribution or tuned as a hyperparameter . Appendix A.6 explains how the weight parameter is selected for our different experiment setups and also demonstrates the impact of weight adjustment on performance for downstream tasks . After defining the neighborhood distribution , we train an objective function that encourages a distinction between the representation of samples of the same neighborhood from the outside samples . An ideal encoder preserves the neighborhood properties in the encoding space . Therefore representations Zl = Enc ( Wl ) of samples from a neighborhood Wl ∈ Nt , can be distinguished from representation Zk = Enc ( Wk ) of samples from outside the neighborhood Wk ∈ N̄t . TNC is composed of two main components : 1 . An Encoder Enc ( Wt ) that maps Wt ∈ RD×δ to a representation Zt ∈ RM , in a lower dimensional space ( M D × δ ) , where D × δ is the total number of measurements in Wt . 2 . A Discriminator D ( Zt , Z ) that approximates the probability of Z being the representation of a window in Nt . More specifically , it receives two samples from the encoding space and predicts the probability of those samples belonging to the same temporal neighborhood . Published as a conference paper at ICLR 2021 Enc Enc Representation D at a Enc Enc Representation D at a Weighting ( a ) Neighborhood samples Enc Enc Representation D at a Enc Enc Representation D at a Weighting ( b ) Non-neighboring samples Figure 1 : Overview of the TNC framework components . For each sample window Wt ( indicated with the dashed black box ) , we first define the neighborhood distribution . The encoder learns the distribution of windows sampled from Nt and N̄t , in the representation space . Then samples from these distributions are fed into the discriminator alongside Zt , to predict the probability of the windows being in the same neighborhood . TNC is a general framework ; therefore , it is agnostic to the nature of the time series and the architecture of the encoder . The encoder can be any parametric model that is well-suited to the signal properties ( Oord et al. , 2016 ; Bai et al. , 2018 ; Fawaz et al. , 2019 ) . For the DiscriminatorD ( Zt , Z ) we use a simple multi-headed binary classifier that outputs 1 if Z and Zt are representations of neighbors in time , and 0 otherwise . In the experiment section , we describe the architectural details of the models used for our experiments in more depth . Figure 1 provides a summary overview of the TNC framework . We formalize the objective function of our unsupervised learning framework in Equation 1 . In essence , we would like the probability likelihood estimation of the Discriminator to be accurate , i.e. , close to 1 for the representation of neighboring samples and close to 0 for windows far apart . Samples from the non-neighboring region ( N̄ ) are weight-adjusted using the w parameters to account for positive samples in this distribution . L = −EWt∼X [ EWl∼Nt [ logD ( Zt , Zl ) ︸ ︷︷ ︸ D ( Enc ( Wt ) , Enc ( Wl ) ) ] +EWk∼N̄t [ ( 1−wt ) ×log ( 1−D ( Zt , Zk ) ) ︸ ︷︷ ︸ D ( Enc ( Wt ) , Enc ( Wk ) ) +wt×logD ( Zt , Zk ) ] ] ( 1 ) We train the encoder and the discriminator hand in hand by optimizing for this objective . Note that the Discriminator is only part of training and will not be used during inference . Similar to the encoder , it can be approximated using any parametric model . However , the more complex the Discriminator , the harder it becomes to interpret the latent space ’ s decision boundaries since it allows similarities to be mapped on complex nonlinear relationships . Defining the neighborhood parameter using the ADF test : As mentioned earlier , the neighborhood range can be specified using the characteristics of the data . In non-stationary time series , the generative process of the signals changes over time . We define the temporal neighborhood around every window as the region where the signal is relatively stationary . Since a signal may remain in an underlying state for an unknown amount of time , each window ’ s neighborhood range may vary in size and must be adjusted to signal behavior . To that end , we use the Augmented Dickey-Fuller ( ADF ) statistical test to derive the neighborhood range η . The ADF test belongs to a category of tests called `` Unit Root Test '' , and is a method for testing the stationarity of a time series . For every Wt , we want to find the neighborhood range around that window that indicates a stationary region . To determine this , we start from η = 1 and gradually increase the neighborhood size η , measuring the p-value from the test at every step . Once p-value is above a threshold ( in our setting 0.01 ) , it means that it fails to reject the null hypothesis and suggests that within this neighborhood region , the signal is no longer stationary . This way , we find the widest neighborhood within which the signal remains relatively stationary . Note that the window size δ is constant throughout the experiment , and during ADF testing , we only adjust the neighborhood ’ s width .

Summary: The authors propose a novel unsupervised encoding scheme for time series. Utilizing a statistical test for non-stationarity, the authors derive a Temporal Neighborhood Coding (TNC) scheme and combine it with ideas from Positive-Unlabeled (PU) learning to learn informative hidden representations of time series windows. The representations are evaluated in terms of how well they can be clustered and how much they influence classification performance on three data sets. The supreme performance was demonstrated when comparing to the state of the art methods and a $k$NN (for classification) baseline. Furthermore, the authors illustrate how the learnt representations remain interpretable as long as the encoding network is reasonably small.