DIVERSIFY to Generalize: Learning Generalized Representations for Time Series Classification

1 INTRODUCTION . Time series classification is one of the most challenging problems in machine learning and statistics community . Example applications include sensor-based human activity recognition , Parkinson ’ s disease diagnosis , and electronic power consumption ( Fawaz et al. , 2019 ) . One important nature of time series is non-stationary property , which means that its statistical features are changing over time . For years , there have been tremendous efforts to tackle the time series classification problem , such as hidden Markov models ( Fulcher & Jones , 2014 ) , RNN-based methods ( Hüsken & Stagge , 2003 ) , and Transformer-based approaches ( Li et al. , 2019 ) . In this paper , we are specifically interested in modeling time series from the distribution perspective . More precisely , we aim to learn representations for time series that can generalize to unseen distributions . Note that this scenario has been extensively studied in existing literature of domain generalization ( Muandet et al. , 2013 ; Wang et al. , 2021a ) and out-of-distribution generalization ( Krueger et al. , 2021 ) , where researchers are keen to bridge the gap between known and unknown distributions , thus generalize well . While most of the efforts are done in image classification , few of them focus on the time series domain , which is more challenging . Although time series share a similar goal as image data in domain generalization , it naturally brings more challenges due to its nonstationary property : the distribution keeps changing over time , which contains diverse distribution information that should be harnessed well for better generalization . We show an illustrative example in Figure 1 . Domain generalization in image classification often involves several domains and the domain information is known ( subfigure ( a ) ) . Thus , we can leverage such domain information to build generalization models . However , in Figure 1 ( b ) , we see that in time series data , although the distribution is changing dynamically over time , its domain information is not available . This will dramatically impede the modeling of existing domain generalization algorithms as they typically assume access to domain information ( subfigure ( c ) ) . In order to learn a generalized time series model , we propose DIVERSIFY , a domain generalization algorithm to characterize the latent distributions inside the time series data . Concretely speaking , our method consists of a min-max adversarial game that : on one hand , it learns to segment the time series data into several latent sub-domains by maximizing the segment-wise distribution gap to preserve diversities , i.e. , the worst-case distribution scenario ; on the other hand , it learns domain-invariant representations by reducing the distribution divergence for the worst-case scenario . Such diversification naturally exists in a non-stationary dataset where the data from multiple people naturally follow several latent distributions . Moreover , it is also surprising to find that even the data of one person still has such diversification : it can also be split into several latent distributions . Obviously , DIVERSIFY can effectively characterize the latent distributions ( Figure 1 ( d ) ) . To summarize , our contributions are three-fold : • Novel problem : For deep learning-based time series classification , we identify the generalized representation problem , which is challenging than traditional image classification problem due to the existence of unidentified latent distributions . • New methodology : We propose DIVERSIFY , a theoretically-motivated solution to solve the generalized representation learning problem to identify the latent distributions . • Good performance : Our approach is extensively evaluated in three types of tasks : gesture recognition , speech command recognition , and sensor-based activity recognition . By qualitative and quantitative analysis , we demonstrate the superiority of DIVERSIFY on several challenging scenarios : on difficult tasks , significantly diverse datasets , and limited data . 2 METHODOLOGY . 2.1 PROBLEM FORMULATION . We are given a time-series dataset Dtr = { ( xi , yi ) } Ni=1 as the training dataset , where N is the number of samples , xi ∈ X ⊂ Rp is the p-dimensional instance ( sliding window ) and yi ∈ Y = { 1 , . . . , C } is its label . We use Ptr ( x , y ) on X × Y to denote the joint distribution of the training dataset . Our goal is to learn a generalized model fromDtr to predict well on an unseen target dataset , Dte , which is inaccessible in training . Like Dtr , time series in Dte also be split into short series . In our problem , the training and test datasets have the same input and output spaces but different distributions , i.e. , X tr = X te , Yte = Yte , but Ptr ( x , y ) 6= Pte ( x , y ) . We aim to train a model h from Dtr to minimize the risk on Dte : minh E ( x , y ) ∼Pte [ h ( x ) 6= y ] . Note that due to the non-stationary property , the training dataset may be composed of several unknown latent distributions , instead of one fixed distribution , i.e. , Ptr ( x , y ) = ∑K i=1 πiPi ( x , y ) , where Pi ( x , y ) is the distribution of the i-th sub-domains of the training data and πi is its weight . K is the number of sub-domains that is unknown , and ∑K i=1 πi = 1 . 2.2 MOTIVATION . The labeled time series data can be composed of several latent distributions ( domains ) that are challenging to characterize , even if the dataset is fully labeled . For instance , data collected by sensors of three persons may belong to two different distributions when considering their similarities . Moreover , even for data from one single person , different segments of one sequence may follow different distributions . In a nutshell , in reality , there often exist several sub-domains in one time series dataset . To ensure good generalization performance on the test dataset , it is important to learn distributioninvariant , or domain-invariant representations from the training dataset by characterizing its latent distributions . These latent distributions may contain both benign and malignant knowledge that influences generalization on the target dataset . In Figure 1 ( c ) , we assume the source domain contains two sub-domains ( circle and plus points ) . Directly learning from the entire source domain by treating it as one distribution may generate the black margin . Although green star data points and red star points can be classified easily , red star data points are misclassified to the green class when predicting on the out-of-distribution domain ( star points ) with the learned model . While other methods fail , our method can characterize the latent distributions , which will be introduced later . 2.3 DIVERSIFY . In this paper , we propose DIVERSIFY to learn generalized representations for time series classification . The core of DIVERSIFY is to characterize the latent distributions in a time series dataset and then to minimize the distribution divergence between each two . To characterize the diverse information for better generalization , DIVERSIFY utilizes an iterative process : it first obtains the worst-case distribution scenario from a given dataset , then bridges the distribution gaps between each pair of latent distributions . Why the worst-case scenario ? We argue that the worst-case scenario will maximally preserve the diverse information of each latent distribution , thus benefiting generalization . Figure 2 describes the main procedures of DIVERSIFY : 1 . Pre-processing : this step adopts the sliding window to split the entire training dataset into fixed-size windows . We believe that the data from one window is the smallest domain unit . 2 . Fine-grained feature update : this step updates the feature extractor using the proposed pseudo domain-class labels as the supervision . 3 . Latent distribution characterization : this step aims to identify the domain label for each instance to acquire the latent distribution information . It tries to maximize the different distribution gaps to enlarge diversity . 4 . Domain-invariant representation learning : this step utilizes pseudo domain labels from the last step to learn domain-invariant representations and train a generalizable model . Note that steps 2 ∼ 4 are iterative as shown in Figure 2 . Now we elaborate their details . Fine-grained Feature Update Before we characterize the latent distributions , we perform finegrained feature update to obtain useful representations . As shown in Figure 2 ( blue ) , in order to fully utilize the knowledge contained in the domains and classes , we propose a new concept : pseudo domain-class label , which serves as the supervision for feature extractor . This will ensure that the feature extractor can learn fine-grained information which benefits the later steps . At the first iteration , there is no domain label d′ and we simply initialize d′ = 0 for all samples . We treat per class per domain as a new class , and set the label to s ∈ { 1 , 2 , · · · , S } . We have S = K×C where K is the pre-defined number of latent distributions that can be tuned in experiments . We perform the following pseudo domain-class label assignment to get discrete values for supervision : s = d′ × C + y . ( Response to reviewer kiW8 : this equation is put inline . ) Let h ( 2 ) f , h ( 2 ) b , h ( 2 ) c be feature extractor , bottleneck , and classifier , respectively ( we use superscript to denote step number ) , then , the supervised loss is computed using the cross-entropy loss ` : Lsuper = E ( x , y ) ∼Ptr ` ( h ( 2 ) c ( h ( 2 ) b ( h ( 2 ) f ( x ) ) ) , s ) . ( 1 ) Latent Distribution Characterization This step characterizes the latent distributions contained in one dataset . As shown in Figure 2 ( green ) , we employ an adversarial training strategy to disentangle the domain labels from the class labels . However , there are no actual domain labels provided , which hinders such disentanglement . Inspired by ( Liang et al. , 2020 ) , we adopt a self-supervised pseudolabeling strategy to obtain domain labels . Firstly , we attain the centroid for each domain with class-invariant features : µ̃k = ∑ xi∈X tr δk ( h ( 3 ) c ( h ( 3 ) b ( h ( 3 ) f ( xi ) ) ) ) h ( 3 ) b ( h ( 3 ) f ( xi ) ) ∑ xi∈X tr δk ( h ( 3 ) c ( h ( 3 ) b ( h ( 3 ) f ( xi ) ) ) ) , ( 2 ) where h ( 3 ) f , h ( 3 ) b , h ( 3 ) c are feature extractor , bottleneck , and classifier , respectively . µ̃k is the initial centroid of the kth latent sub-domains while δk is the kth element of the logit soft-max output . Then , we obtain the pseudo domain labels via the nearest centroid classifier using a distance function D : d̃′i = argmin k D ( h ( 3 ) b ( h ( 3 ) f ( xi ) ) , µ̃k ) . ( 3 ) Then , we compute the centroids based on the new pseudo labels and obtain the updated pseudo domain labels : µk = ∑ xi∈X tr I ( d̃ ′ i = k ) h ( 3 ) b ( h ( 3 ) f ( x ) ) ∑ xi∈X tr I ( d̃ ′ i = k ) , d′i = argmin k D ( h ( 3 ) b ( h ( 3 ) f ( xi ) ) , µk ) , ( 4 ) where I ( a ) = 1 when a is true , otherwise 0 . After we obtain d′ , we can compute the goal of step 2 : Lself + Lcls = E ( x , y ) ∼Ptr ` ( h ( 3 ) c ( h ( 3 ) b ( h ( 3 ) f ( x ) ) ) , d ′ ) + ` ( h ( 3 ) adv ( Rλ1 ( h ( 3 ) b ( h ( 3 ) f ( x ) ) ) ) , y ) , ( 5 ) where h ( 3 ) adv is the discriminator for step 3 that contains several linear layers and one classification layer . Rλ1 is the gradient reverse layer with hyperparameter λ1 ( Ganin et al. , 2016 ) . After this step ends , we can obtain pseudo domain label d′ for x. Domain-invariant Representation Learning After obtaining the latent distributions , we learn domain-invariant representations for generalization . In fact , this step ( Figure 2 purple ) is simple : we borrow the idea from DANN ( Ganin et al. , 2016 ) and directly use adversarial training to update the classification loss Lcls and domain classifier loss Ldom using GRL : Lcls + Ldom = E ( x , y ) ∼Ptr ` ( h ( 4 ) c ( h ( 4 ) b ( h ( 4 ) f ( x ) ) ) , y ) + ` ( h ( 4 ) adv ( Rλ2 ( h ( 4 ) b ( h ( 4 ) f ( x ) ) ) ) , d ′ ) , ( 6 ) where ` is the cross-entropy loss and Rλ2 is the gradient reverse layer with hyperparameter λ2 ( Ganin et al. , 2016 ) . More details of GRL and adv . training are presented in appendix A.1 . We repeat the above three steps until convergence or max epochs . The final model is selected via a validation dataset split from the source domain ( Gulrajani & Lopez-Paz , 2021 ) . As for inference , we predict the labels with the modules from the last step .

Summary: This work proposes a framework to learn a domain-invariant representation of time series. Using the idea of domain generalization, this paper removes the influence of varying domains/distributions across patients/datasets through pseudo domain classification. Although this kind of model has been rather studied in images, but not a lot in time series.