Information-Aware Time Series Meta-Contrastive Learning

1 INTRODUCTION . Time series data in the real world is high dimensional , unstructured , and complex with unique properties , leading to challenges for data modeling ( Yang & Wu , 2006 ) . In addition , without human recognizable patterns , it is much harder to label time series data than images and languages in realworld applications . These labeling limitations hinder deep learning methods , which typically require a huge amount of labeled data for training , been applied on time series data ( Eldele et al. , 2021 ) . Representation learning learns a fixed-dimension embedding from the original time series that keeps their inherent features . Comparing to the raw time series data , these representations are with better transferability and generalization capacity . To deal with labeling limitations , contrastive learning methods have been widely adopted in various domains for their soaring performance on representation learning , including vision , language , and graph-structured data ( Chen et al. , 2020 ; Xie et al. , 2019 ; You et al. , 2020 ) . In a nutshell , contrastive learning methods typically train an encoder to map instances to an embedding space where dissimilar ( negative ) instances are easily distinguishable from similar ( positive ) ones and model predictions to be invariant to small noise applied to either input examples or hidden states . Despite being effective and prevalent , contrastive learning has been less explored in the time series domain ( Eldele et al. , 2021 ; Franceschi et al. , 2019 ; Fan et al. , 2020 ; Tonekaboni et al. , 2021 ) . Existing contrastive learning approaches often involve a specific data augmentation strategy that creates novel and realistic-looking training data without changing its label to construct positive alternatives for any input sample . Their success relies on carefully designed rules of thumb guided by domain expertise . Routinely used data augmentations for contrastive learning are mainly designed for image and language data , such as color distortion , flip , word replacement , and back-translation ( Chen et al. , 2020 ; Luo et al. , 2021 ) . These augmentation techniques generally do not apply to time series data . Recently , some researchers propose augmentations for time series to enhance the size and quality of the training data ( Wen et al. , 2021 ) . For example , Eldele et al . ( 2021 ) and Fan et al . ( 2020 ) propose to adopt jittering , scaling , and permutation strategies to generate augmented instances . Franceschi et al . ( 2019 ) extracts subsequences for data augmentation . In spite of the current progress , existing methods have two main limitations . First , unlike images with human recognizable features , time series data are often associated with inexplicable underlying patterns . Strong augmentation such as permutation may ruin such patterns and consequently , the model will mistake the negative handcrafts for positive ones . While weak augmentation methods such as jittering may generate augmented instances that are too similar to the raw inputs to be informative enough for contrastive learning . On the other hand , time series datasets from different domains may have diverse nature . Adapting a universal data augmentation method , such as subsequence in Xie et al . ( 2019 ) , for all datasets and tasks leads to sub-optimal performances . Other works follow empirical rules to select suitable augmentations from expensive trial-and-error . Akin to hand-crafting features , hand-picking choices of data augmentations are undesirable from the learning perspective . The diversity and heterogeneity of real-life time series data further hinder these methods away from wide applicability . To address the challenges , we first introduce the criteria for selecting good data augmentations in contrastive learning . Data augmentation benefits generalizable , transferable , and robust representation learning by correctly extrapolating the input training space to a larger region ( Wilk et al. , 2018 ) . The positive instances enclose a discriminative zone in which all the data points should be similar to the original instance . The desired data augmentations for contrastive representation learning should have both high fidelity and high variety . High fidelity encourages the augmented data to maintain the semantic identity that is invariant to transformations ( Wilk et al. , 2018 ) . For example , if the downstream task is classification , then the generated augmentations of inputs should be class-preserving . Meanwhile , generating augmented samples with high variety benefits representation learning by increasing the generalization capacity ( Chen et al. , 2020 ) . From the motivation , we theoretically analyze the information flows in data augmentations based upon information theory and derive the criteria for selecting desired time series augmentations . Due to the inexplicability in practical time series data , we assume that the semantic identity is presented by the target in the downstream task . Thus , high fidelity can be achieved by maximizing the mutual information between the downstream label and the augmented data . A one-hot pseudo label is assigned to each instance in the unsupervised setting when downstream labels are unavailable . These pseudo labels encourage augmentations of different instances to be distinguishable from each other . We show that data augmentations preserving these pseudo labels can add new information without decreasing the fidelity . Concurrently , we maximize the entropy of augmented data conditional on the original instances to increase the variety of data augmentations . Based on the derived criteria , we propose an adaptive data augmentation method , InfoTS ( as shown in Figure 1 ) , by employing a meta-learning mechanism to avoid ad-hoc choices or painstakingly trial-and-error tuning . Specifically , we utilize a meta-network to learn the augmentation prior in tandem with contrastive learning . The meta-learner automatically selects optimal augmentations from candidate augmentations to generate feasible positive samples . Along with random sampled negative instances , augmented instances are then fed into a time series encoder to learn representations in a contrastive manner . With a reparameterization trick , the meta-network can be efficiently optimized with back-propagation based upon the proposed criteria . Therefore , the meta-network can automatically select data augmentations in a per dataset and per learning task manner without resorting to expert knowledge or tedious downstream validation . Our main contributions include : • We propose criteria to guide the selection of data augmentations for contrastive time series representation learning without prefabricated knowledge . • We propose a meta-learning based method to automatically select feasible data augmentations for different time series datasets , which can be efficiently optimized with backpropagation . • We empirically verify the effectiveness of the proposed criteria to find optimal data augmentations . Extensive experiments demonstrate that InfoTS can achieve highly competitive performance with up to 11.4 % reduction in MSE on forecasting task and up to 2.8 % relative improvement in accuracy on classification task over the leading baselines . 2 METHODOLOGY . 2.1 NOTATIONS AND PROBLEM DEFINITION . A time series instance x has dimension T × F , where T is the length of sequence and F is the dimension of features . Given a set of time series instances X , we aim to learn an encoder fθ ( x ) that maps each instance x to a fixed-length vector z ∈ RD , where θ is the learnable parameters of the encoder network and D is the dimension of representation vectors . In semi-supervised settings , each instance x in the labelled set XL ⊆ X is associated with a label y for the downstream task . Specially , XL = X holds in the fully supervised setting . In the work , we use the Sans-serif style lowercase letters , such as x , to denote random time series variables and italic lowercase letters , such as x , for sampled instances . 2.2 WHAT ARE GOOD AUGMENTATIONS FOR CONTRASTIVE LEARNING ? . The goal of data augmentation for contrastive learning is to create realistically rational instances that maintain semantics through different transformation approaches . Unlike instances in vision and language domains , the underlying semantics of time series data is not recognizable to human , making it hard , if not impossible , to include human knowledge to data augmentation for time series data . For example , rotating an image will not change its content or the label . While permuting a time series instance may ruin its signal patterns and generates a meaningless time series instance . In addition , the tremendous heterogeneity of real-life time series datasets further makes selections based on trial-and-errors impractical . Although multiple data augmentation methods have been proposed for time series data , there is less discussion on what is a good augmentation that is meaningful for a given learning task and dataset without prefabricated human priors . From our perspective , ideal data augmentations for contrastive representation should keep high fidelity , high variety , and adaptive to different datasets . The illustration and examples are shown in Figure 2 . High Fidelity . Augmentations with high fidelity maintain the semantic identity that is invariant to transformations . Considering the inexplicability in practical time series data , it is challenging to visually check the fidelity of augmentations . Thus , we assume that the semantic identity of a time series instance is presented by its label in the downstream task , which might be either available or unavailable during the training period . Here , we start our analysis from the supervised case and will extend it to the unsupervised case later . Based on the information theory ( Tishby et al. , 2000 ) , we define the objective that keeps high fidelity as the large mutual information ( MI ) between augmentation v and the label y , i.e. , MI ( v ; y ) . We consider augmentation v as a probabilistic function of x and a random variable , that v = g ( x ; ) . From the definition of mutual information , we have MI ( v ; y ) = H ( y ) −H ( y|v ) , where H ( y ) is the ( Shannon ) entropy of y and H ( y|v ) is the entropy of y conditioned on augmentation v. Since H ( y ) is irrelevant to data augmentations , the objective is equivalent to minimizing the conditional entropy H ( y|v ) . Considering the efficient optimization , we follow Ying et al . ( 2019 ) and Luo et al . ( 2020 ) to approximate it with cross-entropy between y and ŷ , where ŷ is the prediction with augmentation v as the input and calculated via v = g ( x ; ) z = fθ ( v ) ŷ = hw ( z ) , ( 1 ) where z is the representation and hw ( · ) is a prediction projector parameterized by w. The prediction projector is optimized by the classification objective . Then , the objective of high fidelity for supervised or semi-supervised cases is to minimize CE ( y ; ŷ ) = − C∑ c=1 P ( y = c ) logP ( ŷ = c ) , ( 2 ) where C is the number of labels . In the unsupervised settings where y is unavailable , one-hot encoding ys ∈ R|X| is utilized as the pseudo label to replace y in Eq . ( 2 ) . The motivation is that augmented instances are still distinguishable from other instances with the classifier . We theoretically show that augmentations that preserving pseudo labels have the following properties . Property 1 ( Preserving Fidelity ) . If augmentation v preserves the one-hot encoding pseudo label , the mutual information between v and the downstream task label y ( although not visible to training ) is equivalent to that between raw input x and y , i.e. , MI ( v ; y ) = MI ( x ; y ) . Property 2 ( Adding New Information ) . By preserving the one-hot encoding pseudo label , augmentation v contains new information comparing to the raw input x , i.e. , H ( v ) ≥ H ( x ) . Detailed proofs are shown in the Appendix A . These properties show that in the unsupervised setting , preserving the one-hot encoding pseudo label guarantees that the generated augmentations will not decrease the fidelity , regardless of the downstream tasks and variances inherent in the augmentations . Concurrently , it may introduce new information for contrastive learning . Since the number of labels is equal to the number of instances in dataset X in an unsupervised case , direct optimization of Eq . ( 2 ) is inefficient and unscalable . Thus , we further relax it by approximating y with the batch-wise one-hot encoding yB , which decreases the number of labels C from the dataset size to the batch size . High Variety . Sufficient variances in augmentations improve the generalization capacity of contrastive learning models . In the information theory , the uncertainty inherent in the random variable ’ s possible outcomes is described by its entropy . Considering that augmented instances are generated based on the raw input x , we maximize the entropy of v conditioned on x , H ( v|x ) , to maintain a high variety of augmentations . From the definition of conditional entropy , we have H ( v|x ) = H ( v ) −MI ( v ; x ) . ( 3 ) We dismiss the first part since the unconstrained entropy of v can be dominated by meaningless noise . Considering the continuity of both v and x , we adopt a mutual information neural estimator , InfoNCE ( Oord et al. , 2018 ; Tian et al. , 2020 ) to approximately compute the mutual information for its practical effectiveness . Other MI estimators , such as Jensen-Shannon ( JSD ) estimator ( Nowozin et al. , 2016 ) , normalized temperature-scaled cross-entropy ( NT-Xent ) ( Chen et al. , 2020 ) , and leaveone-out ( Poole et al. , 2019 ) can also conveniently be the plug-and-play component in our framework . Then , the objective to encourage high variety is to maximize the InfoNCE between v and x : InfoNCE ( v ; x ) = −Ex [ log exp ( sim ( zx , zv ) ) ∑ x′∈X exp ( sim ( zx , zv′ ) ) ] , ( 4 ) where v′ is an augmented instance of input instance x′ . zx , zv , and zv′ are representations of instance x , v , and v′ respectively . sim ( z1 , z2 ) = zT1 z2 is the inner product of vectors z1 and z2 . Criteria . Combining the information aware definition of both high fidelity and variety , we propose the criteria for selecting good augmentations without prior knowledge , min v −InfoNCE ( v ; x ) + βCE ( y ; hw ( fθ ( v ) ) ) , ( 5 ) where β is a hyper-parameter to achieve the trade-off between fidelity and variety . Note that in the unsupervised settings , y is replaced by one-hot encoding pseudo label .. Relation to Information Bottleneck . Although the formation is similar to information bottleneck in data compression , minp ( e|x ) MI ( x ; e ) − βMI ( e ; y ) , our criteria are different in the following aspects . First , e in the information bottleneck is a representation of input x , while v in Eq . ( 5 ) represents the augmented instances . Second , information bottleneck aims to keep minimal and sufficient information for data compression , while our criteria are designed for data augmentations in contrastive learning . Third , in information bottleneck , the compressed representation e is a deterministic function of input x with no variances . MI ( e ; y ) and MI ( e ; x ) are constraint by MI ( x ; y ) and H ( x ) that MI ( e ; y ) ≤ MI ( x ; y ) and MI ( e ; x ) = H ( e ) , where H ( e ) is the entropy of e. In our criteria , v is a probabilistic function of input x . As a result , the variances of v makes the augmentation space much larger than the compression representation space in information bottleneck . Relation to InfoMin . InfoMin is designed based on the information bottleneck that good views should keep minimal and sufficient information from the original input ( Tian et al. , 2020 ) . Similar to the information bottleneck , InfoMin assumes that augmented views are functions of the input , which heavily constrains the variance of data augmentations . Besides , high fidelity property is dismissed in the unsupervised setting . It works for image datasets due to the availability of human knowledge . However , it may fail to generate reasonable augmentations for time series data .

Summary: This is a very good and solid paper that has both empirical elements and theoretical basis. As mentioned in the paper data augmentation in time series is a notoriously difficult problem for the simple reason that it can distort the time series completely - which in contrast to images - humans cannot verify this as it can happen with images. So the contribution of this paper is a new data augmentation approach based on information theory, a meta learning approach and an approach to select optimal data augmentation for contrastive learning.