Information-Aware Time Series Meta-Contrastive Learning

1 INTRODUCTION . Time series data in the real world is high dimensional , unstructured , and complex with unique properties , leading to challenges for data modeling ( Yang & Wu , 2006 ) . In addition , without human recognizable patterns , it is much harder to label time series data than images and languages in realworld applications . These labeling limitations hinder deep learning methods , which typically require a huge amount of labeled data for training , been applied on time series data ( Eldele et al. , 2021 ) . Representation learning learns a fixed-dimension embedding from the original time series that keeps their inherent features . Comparing to the raw time series data , these representations are with better transferability and generalization capacity . To deal with labeling limitations , contrastive learning methods have been widely adopted in various domains for their soaring performance on representation learning , including vision , language , and graph-structured data ( Chen et al. , 2020 ; Xie et al. , 2019 ; You et al. , 2020 ) . In a nutshell , contrastive learning methods typically train an encoder to map instances to an embedding space where dissimilar ( negative ) instances are easily distinguishable from similar ( positive ) ones and model predictions to be invariant to small noise applied to either input examples or hidden states . Despite being effective and prevalent , contrastive learning has been less explored in the time series domain ( Eldele et al. , 2021 ; Franceschi et al. , 2019 ; Fan et al. , 2020 ; Tonekaboni et al. , 2021 ) . Existing contrastive learning approaches often involve a specific data augmentation strategy that creates novel and realistic-looking training data without changing its label to construct positive alternatives for any input sample . Their success relies on carefully designed rules of thumb guided by domain expertise . Routinely used data augmentations for contrastive learning are mainly designed for image and language data , such as color distortion , flip , word replacement , and back-translation ( Chen et al. , 2020 ; Luo et al. , 2021 ) . These augmentation techniques generally do not apply to time series data . Recently , some researchers propose augmentations for time series to enhance the size and quality of the training data ( Wen et al. , 2021 ) . For example , Eldele et al . ( 2021 ) and Fan et al . ( 2020 ) propose to adopt jittering , scaling , and permutation strategies to generate augmented instances . Franceschi et al . ( 2019 ) extracts subsequences for data augmentation . In spite of the current progress , existing methods have two main limitations . First , unlike images with human recognizable features , time series data are often associated with inexplicable underlying patterns . Strong augmentation such as permutation may ruin such patterns and consequently , the model will mistake the negative handcrafts for positive ones . While weak augmentation methods such as jittering may generate augmented instances that are too similar to the raw inputs to be informative enough for contrastive learning . On the other hand , time series datasets from different domains may have diverse nature . Adapting a universal data augmentation method , such as subsequence in Xie et al . ( 2019 ) , for all datasets and tasks leads to sub-optimal performances . Other works follow empirical rules to select suitable augmentations from expensive trial-and-error . Akin to hand-crafting features , hand-picking choices of data augmentations are undesirable from the learning perspective . The diversity and heterogeneity of real-life time series data further hinder these methods away from wide applicability . To address the challenges , we first introduce the criteria for selecting good data augmentations in contrastive learning . Data augmentation benefits generalizable , transferable , and robust representation learning by correctly extrapolating the input training space to a larger region ( Wilk et al. , 2018 ) . The positive instances enclose a discriminative zone in which all the data points should be similar to the original instance . The desired data augmentations for contrastive representation learning should have both high fidelity and high variety . High fidelity encourages the augmented data to maintain the semantic identity that is invariant to transformations ( Wilk et al. , 2018 ) . For example , if the downstream task is classification , then the generated augmentations of inputs should be class-preserving . Meanwhile , generating augmented samples with high variety benefits representation learning by increasing the generalization capacity ( Chen et al. , 2020 ) . From the motivation , we theoretically analyze the information flows in data augmentations based upon information theory and derive the criteria for selecting desired time series augmentations . Due to the inexplicability in practical time series data , we assume that the semantic identity is presented by the target in the downstream task . Thus , high fidelity can be achieved by maximizing the mutual information between the downstream label and the augmented data . A one-hot pseudo label is assigned to each instance in the unsupervised setting when downstream labels are unavailable . These pseudo labels encourage augmentations of different instances to be distinguishable from each other . We show that data augmentations preserving these pseudo labels can add new information without decreasing the fidelity . Concurrently , we maximize the entropy of augmented data conditional on the original instances to increase the variety of data augmentations . Based on the derived criteria , we propose an adaptive data augmentation method , InfoTS ( as shown in Figure 1 ) , by employing a meta-learning mechanism to avoid ad-hoc choices or painstakingly trial-and-error tuning . Specifically , we utilize a meta-network to learn the augmentation prior in tandem with contrastive learning . The meta-learner automatically selects optimal augmentations from candidate augmentations to generate feasible positive samples . Along with random sampled negative instances , augmented instances are then fed into a time series encoder to learn representations in a contrastive manner . With a reparameterization trick , the meta-network can be efficiently optimized with back-propagation based upon the proposed criteria . Therefore , the meta-network can automatically select data augmentations in a per dataset and per learning task manner without resorting to expert knowledge or tedious downstream validation . Our main contributions include : • We propose criteria to guide the selection of data augmentations for contrastive time series representation learning without prefabricated knowledge . • We propose a meta-learning based method to automatically select feasible data augmentations for different time series datasets , which can be efficiently optimized with backpropagation . • We empirically verify the effectiveness of the proposed criteria to find optimal data augmentations . Extensive experiments demonstrate that InfoTS can achieve highly competitive performance with up to 11.4 % reduction in MSE on forecasting task and up to 2.8 % relative improvement in accuracy on classification task over the leading baselines . 2 METHODOLOGY . 2.1 NOTATIONS AND PROBLEM DEFINITION . A time series instance x has dimension T × F , where T is the length of sequence and F is the dimension of features . Given a set of time series instances X , we aim to learn an encoder fθ ( x ) that maps each instance x to a fixed-length vector z ∈ RD , where θ is the learnable parameters of the encoder network and D is the dimension of representation vectors . In semi-supervised settings , each instance x in the labelled set XL ⊆ X is associated with a label y for the downstream task . Specially , XL = X holds in the fully supervised setting . In the work , we use the Sans-serif style lowercase letters , such as x , to denote random time series variables and italic lowercase letters , such as x , for sampled instances . 2.2 WHAT ARE GOOD AUGMENTATIONS FOR CONTRASTIVE LEARNING ? . The goal of data augmentation for contrastive learning is to create realistically rational instances that maintain semantics through different transformation approaches . Unlike instances in vision and language domains , the underlying semantics of time series data is not recognizable to human , making it hard , if not impossible , to include human knowledge to data augmentation for time series data . For example , rotating an image will not change its content or the label . While permuting a time series instance may ruin its signal patterns and generates a meaningless time series instance . In addition , the tremendous heterogeneity of real-life time series datasets further makes selections based on trial-and-errors impractical . Although multiple data augmentation methods have been proposed for time series data , there is less discussion on what is a good augmentation that is meaningful for a given learning task and dataset without prefabricated human priors . From our perspective , ideal data augmentations for contrastive representation should keep high fidelity , high variety , and adaptive to different datasets . The illustration and examples are shown in Figure 2 . High Fidelity . Augmentations with high fidelity maintain the semantic identity that is invariant to transformations . Considering the inexplicability in practical time series data , it is challenging to visually check the fidelity of augmentations . Thus , we assume that the semantic identity of a time series instance is presented by its label in the downstream task , which might be either available or unavailable during the training period . Here , we start our analysis from the supervised case and will extend it to the unsupervised case later . Based on the information theory ( Tishby et al. , 2000 ) , we define the objective that keeps high fidelity as the large mutual information ( MI ) between augmentation v and the label y , i.e. , MI ( v ; y ) . We consider augmentation v as a probabilistic function of x and a random variable , that v = g ( x ; ) . From the definition of mutual information , we have MI ( v ; y ) = H ( y ) −H ( y|v ) , where H ( y ) is the ( Shannon ) entropy of y and H ( y|v ) is the entropy of y conditioned on augmentation v. Since H ( y ) is irrelevant to data augmentations , the objective is equivalent to minimizing the conditional entropy H ( y|v ) . Considering the efficient optimization , we follow Ying et al . ( 2019 ) and Luo et al . ( 2020 ) to approximate it with cross-entropy between y and ŷ , where ŷ is the prediction with augmentation v as the input and calculated via v = g ( x ; ) z = fθ ( v ) ŷ = hw ( z ) , ( 1 ) where z is the representation and hw ( · ) is a prediction projector parameterized by w. The prediction projector is optimized by the classification objective . Then , the objective of high fidelity for supervised or semi-supervised cases is to minimize CE ( y ; ŷ ) = − C∑ c=1 P ( y = c ) logP ( ŷ = c ) , ( 2 ) where C is the number of labels . In the unsupervised settings where y is unavailable , one-hot encoding ys ∈ R|X| is utilized as the pseudo label to replace y in Eq . ( 2 ) . The motivation is that augmented instances are still distinguishable from other instances with the classifier . We theoretically show that augmentations that preserving pseudo labels have the following properties . Property 1 ( Preserving Fidelity ) . If augmentation v preserves the one-hot encoding pseudo label , the mutual information between v and the downstream task label y ( although not visible to training ) is equivalent to that between raw input x and y , i.e. , MI ( v ; y ) = MI ( x ; y ) . Property 2 ( Adding New Information ) . By preserving the one-hot encoding pseudo label , augmentation v contains new information comparing to the raw input x , i.e. , H ( v ) ≥ H ( x ) . Detailed proofs are shown in the Appendix A . These properties show that in the unsupervised setting , preserving the one-hot encoding pseudo label guarantees that the generated augmentations will not decrease the fidelity , regardless of the downstream tasks and variances inherent in the augmentations . Concurrently , it may introduce new information for contrastive learning . Since the number of labels is equal to the number of instances in dataset X in an unsupervised case , direct optimization of Eq . ( 2 ) is inefficient and unscalable . Thus , we further relax it by approximating y with the batch-wise one-hot encoding yB , which decreases the number of labels C from the dataset size to the batch size . High Variety . Sufficient variances in augmentations improve the generalization capacity of contrastive learning models . In the information theory , the uncertainty inherent in the random variable ’ s possible outcomes is described by its entropy . Considering that augmented instances are generated based on the raw input x , we maximize the entropy of v conditioned on x , H ( v|x ) , to maintain a high variety of augmentations . From the definition of conditional entropy , we have H ( v|x ) = H ( v ) −MI ( v ; x ) . ( 3 ) We dismiss the first part since the unconstrained entropy of v can be dominated by meaningless noise . Considering the continuity of both v and x , we adopt a mutual information neural estimator , InfoNCE ( Oord et al. , 2018 ; Tian et al. , 2020 ) to approximately compute the mutual information for its practical effectiveness . Other MI estimators , such as Jensen-Shannon ( JSD ) estimator ( Nowozin et al. , 2016 ) , normalized temperature-scaled cross-entropy ( NT-Xent ) ( Chen et al. , 2020 ) , and leaveone-out ( Poole et al. , 2019 ) can also conveniently be the plug-and-play component in our framework . Then , the objective to encourage high variety is to maximize the InfoNCE between v and x : InfoNCE ( v ; x ) = −Ex [ log exp ( sim ( zx , zv ) ) ∑ x′∈X exp ( sim ( zx , zv′ ) ) ] , ( 4 ) where v′ is an augmented instance of input instance x′ . zx , zv , and zv′ are representations of instance x , v , and v′ respectively . sim ( z1 , z2 ) = zT1 z2 is the inner product of vectors z1 and z2 . Criteria . Combining the information aware definition of both high fidelity and variety , we propose the criteria for selecting good augmentations without prior knowledge , min v −InfoNCE ( v ; x ) + βCE ( y ; hw ( fθ ( v ) ) ) , ( 5 ) where β is a hyper-parameter to achieve the trade-off between fidelity and variety . Note that in the unsupervised settings , y is replaced by one-hot encoding pseudo label .. Relation to Information Bottleneck . Although the formation is similar to information bottleneck in data compression , minp ( e|x ) MI ( x ; e ) − βMI ( e ; y ) , our criteria are different in the following aspects . First , e in the information bottleneck is a representation of input x , while v in Eq . ( 5 ) represents the augmented instances . Second , information bottleneck aims to keep minimal and sufficient information for data compression , while our criteria are designed for data augmentations in contrastive learning . Third , in information bottleneck , the compressed representation e is a deterministic function of input x with no variances . MI ( e ; y ) and MI ( e ; x ) are constraint by MI ( x ; y ) and H ( x ) that MI ( e ; y ) ≤ MI ( x ; y ) and MI ( e ; x ) = H ( e ) , where H ( e ) is the entropy of e. In our criteria , v is a probabilistic function of input x . As a result , the variances of v makes the augmentation space much larger than the compression representation space in information bottleneck . Relation to InfoMin . InfoMin is designed based on the information bottleneck that good views should keep minimal and sufficient information from the original input ( Tian et al. , 2020 ) . Similar to the information bottleneck , InfoMin assumes that augmented views are functions of the input , which heavily constrains the variance of data augmentations . Besides , high fidelity property is dismissed in the unsupervised setting . It works for image datasets due to the availability of human knowledge . However , it may fail to generate reasonable augmentations for time series data .

Summary: This paper describes an information-aware approach to representation learning for time series. The formulation focuses on how to obtain effective data augmentations and addresses the underlying problem from information-theoretic viewpoints, leading to the two optimization criteria, namely, high fidelity and high variety. The experimental results on several time series datasets for forecasting and classification show improvements over the methods in comparison. Detailed comments are listed below.