Question: The advantage of adaptive selection: So if I understand correctly with adaptive selection the authors refer to the concrete sample from a trained categorical over possible augmentations?
Evidence: Time series data in the real world is high dimensional, unstructured, and complex with unique properties, leading to challenges for data modeling. In addition, without human recognizable patterns, it is much harder to label time series data than images and languages in realworld applications. These labeling limitations hinder deep learning methods, which typically require a huge amount of labeled data for training, been applied on time series data. Representation learning learns a fixed-dimension embedding from the original time series that keeps their inherent features. Comparing to the raw time series data, these representations are with better transferability and generalization capacity. To deal with labeling limitations, contrastive learning methods have been widely adopted in various domains for their soaring performance on representation learning, including vision, language, and graph-structured data. In a nutshell, contrastive learning methods typically train an encoder to map instances to an embedding space where dissimilar (negative) instances are easily distinguishable from similar (positive) ones and model predictions to be invariant to small noise applied to either input examples or hidden states.
Despite being effective and prevalent, contrastive learning has been less explored in the time series domain. Existing contrastive learning approaches often involve a specific data augmentation strategy that creates novel and realistic-looking training data without changing its label to construct positive alternatives for any input sample. Their success relies on carefully designed rules of thumb guided by domain expertise. Routinely used data augmentations for contrastive learning are mainly designed for image and language data, such as color distortion, flip, word replacement, and back-translation. These augmentation techniques generally do not apply to time series data.
Figure: InfoTS is composed of three parts: (1) candidate transformation that generates different augmentations of the original inputs, (2) a meta-network that selects the optimal augmentations, (3) an encoder that learns representations of time series instances. The meta-network is learned in tandem with contrastive encoder learning.
Recently, some researchers propose augmentations for time series to enhance the size and quality of the training data. For example, and propose to adopt jittering, scaling, and permutation strategies to generate augmented instances. extracts subsequences for data augmentation. In spite of the current progress, existing methods have two main limitations. First, unlike images with human recognizable features, time series data are often associated with inexplicable underlying patterns. Strong augmentation such as permutation may ruin such patterns and consequently, the model will mistake the negative handcrafts for positive ones. While weak augmentation methods such as jittering may generate augmented instances that are too similar to the raw inputs to be informative enough for contrastive learning. On the other hand, time series datasets from different domains may have diverse nature. Adapting a universal data augmentation method, such as subsequence in, for all datasets and tasks leads to sub-optimal performances. Other works follow empirical rules to select suitable augmentations from expensive trial-and-error. Akin to hand-crafting features, hand-picking choices of data augmentations are undesirable from the learning perspective. The diversity and heterogeneity of real-life time series data further hinder these methods away from wide applicability.
To address the challenges, we first introduce the criteria for selecting good data augmentations in contrastive learning. Data augmentation benefits generalizable, transferable, and robust representation learning by correctly extrapolating the input training space to a larger region. The positive instances enclose a discriminative zone in which all the data points should be similar to the original instance. The desired data augmentations for contrastive representation learning should have both high fidelity and high variety. High fidelity encourages the augmented data to maintain the semantic identity that is invariant to transformations. For example, if the downstream task is classification, then the generated augmentations of inputs should be class-preserving. Meanwhile, generating augmented samples with high variety benefits representation learning by increasing the generalization capacity. From the motivation, we theoretically analyze the information flows in data augmentations based upon information theory and derive the criteria for selecting desired time series augmentations. Due to the inexplicability in practical time series data, we assume that the semantic identity is presented by the target in the downstream task. Thus, high fidelity can be achieved by maximizing the mutual information between the downstream label and the augmented data. A one-hot pseudo label is assigned to each instance in the unsupervised setting when downstream labels are unavailable. These pseudo labels encourage augmentations of different instances to be distinguishable from each other. We show that data augmentations preserving these pseudo labels can add new information without decreasing the fidelity. Concurrently, we maximize the entropy of augmented data conditional on the original instances to increase the variety of data augmentations.
Based on the derived criteria, we propose an adaptive data augmentation method, InfoTS (as shown in Figure), by employing a meta-learning mechanism to avoid ad-hoc choices or painstakingly trial-and-error tuning. Specifically, we utilize a meta-network to learn the augmentation prior in tandem with contrastive learning. The meta-learner automatically selects optimal augmentations from candidate augmentations to generate feasible positive samples. Along with random sampled negative instances, augmented instances are then fed into a time series encoder to learn representations in a contrastive manner. With a reparameterization trick, the meta-network can be efficiently optimized with back-propagation based upon the proposed criteria. Therefore, the meta-network can automatically select data augmentations in a per dataset and per learning task manner without resorting to expert knowledge or tedious downstream validation. Our main contributions include:
• We propose criteria to guide the selection of data augmentations for contrastive time series representation learning without prefabricated knowledge. • We propose a meta-learning based method to automatically select feasible data augmentations for different time series datasets, which can be efficiently optimized with backpropagation. • We empirically verify the effectiveness of the proposed criteria to find optimal data augmentations. Extensive experiments demonstrate that InfoTS can achieve highly competitive performance with up to 11.4% reduction in MSE on forecasting task and up to 2.8% relative improvement in accuracy on classification task over the leading baselines.
To present deep insights into the proposed method, we verify key components in InfoTS with multiple ablation studies on the Electricity dataset. Results are shown in Table. 1) We verify the effectiveness of the adopted contrastive learning objective, Eq. (), by comparing InfoTS with its two variants. "w/o Local" removes the local contrastive loss between subsequences and "w/o Global" removes the instance-level contrastive loss. The comparison between these variants and InfoTS demonstrates that both instance-level and intra-temporal objectives are important in contrastive learning for time series data. 2) To demonstrate the advantage of adaptive selection of augmentations, we compare InfoTS with variants "Random" and "All". "Random" randomly selects an augmentation from candidate transformation functions each time and "All" sequentially applies transformations to generate augmented instances. Performance decreases are observed in these variances, verifying the key role of adaptive selection in our method. 3) To show the effects of variety and fidelity objectives in meta-network training, we include two variants, "w/o Fidelity" and "w/o Variety", which dismiss the fidelity or variety objective, respectively. The comparison between InfoTS and the variants empirically confirms both variety and fidelity are important for data augmentation in contrastive learning. Note that the adopted basic augmentations are manually tuned in a previous paper with high qualities for contrastive learning. Including low quality basic augmentations, we conduct more ablation studies are shown in Appendix D.3.
Based on the derived criteria, we propose an adaptive data augmentation method, InfoTS (as shown in Figure 1), by employing a meta-learning mechanism to avoid ad-hoc choices or painstakingly trial-and-error tuning. Specifically, we utilize a meta-network to learn the augmentation prior in tandem with contrastive learning. The meta-learner automatically selects optimal augmentations from candidate augmentations to generate feasible positive samples.****To demonstrate the advantage of adaptive selection of augmentations, we compare InfoTS with variants “Random” and “All”.****To show that our InfoTS can adaptively detect the most effective augmentation based on the data distribution, we follow the setting in Section 4.4 and conduct more ablation studies to investigate comprehensively into the proposed model. We compare performances of variants that each applies a single transformation to generate augmented instances in Table 5. From the table, we know that augmentation with subsequence benefits the most for the Electricity dataset. We visualize the weight weight updating process of InfoTS in Figure 8, with each line representing the normalized importance score of the corresponding transformation. The weight for subsequence increase with the epoch, showing that InfoTS tends to adopt subsequence as the optimal transformation. Consistency between accuracy performance and weight updating process demonstrates the effectiveness of InfoTS to adaptively select feasible transformations. Besides, as shown in Table 5, InfoTS outperforms the variant that uses subsequence only. This comparison shows that the meta-network learns to consider the combinations, which is better than any (single) candidate augmentation.
Answer: Yes, our meta-network learns a weight for each candidate augmentation, which is considered as adaptive selection.